From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 955C4E92FFD for ; Fri, 6 Oct 2023 18:46:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2DA988D00C9; Fri, 6 Oct 2023 14:46:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 28A4280008; Fri, 6 Oct 2023 14:46:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 12B4C8D00D5; Fri, 6 Oct 2023 14:46:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 02FC78D00C9 for ; Fri, 6 Oct 2023 14:46:36 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D785AB43F4 for ; Fri, 6 Oct 2023 18:46:35 +0000 (UTC) X-FDA: 81315917550.04.AB4AE3D Received: from mail-pg1-f171.google.com (mail-pg1-f171.google.com [209.85.215.171]) by imf29.hostedemail.com (Postfix) with ESMTP id 06F83120023 for ; Fri, 6 Oct 2023 18:46:32 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=OsnGWhfK; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=nphamcs@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1696617993; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IQ3fDKoP0HxAKgFx5HjhrMjKnZYWAS/nwLzGqRn4tZU=; b=H+PvsZvwmQF9x70aLAHYhjx5sKBhqEDARAFDGvcZUkUSH0G/UXR+PEhHrfOOBXRO/4bDh+ 51Ko/ZgoF+4R+4sATUxqSii7SHJ0V0YiTIZRnhu9UnIT8/+b8Bn21BFHHXudrE/QlR+AMk HLd7VaX171i+vW/z1j5UZ3edSZ7A7R8= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=OsnGWhfK; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.215.171 as permitted sender) smtp.mailfrom=nphamcs@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1696617993; a=rsa-sha256; cv=none; b=W1yJnABA/1KLdO5ksW0TJSIq/JdEm5d9jsrUBg1CbPLw+4n0pBLR2OdbXsrMOzxXKZtScK NbmXFT5YvqlWhm4dY8C78RP4UbZuIRJbULghngj8OjGNdxzQNcZDFS2Z/GGssBv+9XenoP 5pGRkNos5PbHH8Em5Zdoaf1UOJM2np4= Received: by mail-pg1-f171.google.com with SMTP id 41be03b00d2f7-51b4ef5378bso1894269a12.1 for ; Fri, 06 Oct 2023 11:46:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696617992; x=1697222792; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IQ3fDKoP0HxAKgFx5HjhrMjKnZYWAS/nwLzGqRn4tZU=; b=OsnGWhfKCClSmqIIz7thhZJBxZgKWx5WTMrUzxylzgOsoX+HDVhrx5K2WBVo5asnrE 3rHZ7g06/fbWIxsSRdth135yCCFy2fXe+rxuPjMafbsgaabGMigJDFOfkPglkSwL71Y5 xG/M61kDhMagtYoSzkEWOyU0ciZGstrsG0Sc+/qNFFZdDGqMQBVghpJ/6Hh/RwNVyMGE veWOSyIRUpfyU17EGUanyQx0u/IUNYyc3TEsModbyXakTE5j/Y3hIQFGzDCcziJMeXQ4 lYsM9TLVOKuwHtehuaFflPtOzgdppPjRC/aoQYF7qXgb7U8ITBxj6ZCC0VaTiRZFO48S WkRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696617992; x=1697222792; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IQ3fDKoP0HxAKgFx5HjhrMjKnZYWAS/nwLzGqRn4tZU=; b=PVQl4/y3ryFNxSBNzYU8cVzg/Zo/dvJS5U9jh2fyc/0ELzUT21/BlOdcfN88maFNB5 61bb+jAq3QBefx7LT7SZZbAhRMiJQAbtPHOJKrWdeok2v7/5IyCzpnKd4hAfvmX6QRkO 6BJr+5LbuIsXNF4+dQuKPgnYC9k+lcDlZjFg65kCwRCv2skLc4YjXffjWb+WbVbkvnaE M0xnT6rLKpB5VmlxugYBJGxJzoDaJneExVsn3zy5PTA7N89NFJem4O1fVm0d6RKoN7FG Vx17KPYLsfuGXbDzdrHbW1Ull94wk38XxyX1YztE7DCAW2qyXbWBqz5poXQyeJpiHffk CHyg== X-Gm-Message-State: AOJu0YzFO0dg8D1IBua3fKFTsliKlZmstfKo0XQSvSrFCsRLTOhvtJDc 1N4KJH51korSWc164f42It8= X-Google-Smtp-Source: AGHT+IH+P+iy9kpiRfFqEt50ag8YuOg/nc+rwk5xrE7+ZTfX7UsJHZlKqKdNGkM0xrSHZ2CUdYxkDg== X-Received: by 2002:a17:90a:b891:b0:273:441a:dae6 with SMTP id o17-20020a17090ab89100b00273441adae6mr8808492pjr.19.1696617991566; Fri, 06 Oct 2023 11:46:31 -0700 (PDT) Received: from localhost (fwdproxy-prn-000.fbsv.net. [2a03:2880:ff::face:b00c]) by smtp.gmail.com with ESMTPSA id s20-20020a170902989400b001c5b8087fe5sm4250869plp.94.2023.10.06.11.46.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Oct 2023 11:46:31 -0700 (PDT) From: Nhat Pham To: akpm@linux-foundation.org Cc: riel@surriel.com, hannes@cmpxchg.org, mhocko@kernel.org, roman.gushchin@linux.dev, shakeelb@google.com, muchun.song@linux.dev, tj@kernel.org, lizefan.x@bytedance.com, shuah@kernel.org, mike.kravetz@oracle.com, yosryahmed@google.com, fvdl@google.com, linux-mm@kvack.org, kernel-team@meta.com, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org Subject: [PATCH v4 1/4] memcontrol: add helpers for hugetlb memcg accounting Date: Fri, 6 Oct 2023 11:46:26 -0700 Message-Id: <20231006184629.155543-2-nphamcs@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231006184629.155543-1-nphamcs@gmail.com> References: <20231006184629.155543-1-nphamcs@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 06F83120023 X-Stat-Signature: bwo7b5d4wgf669uaf7ab3qfrfjbts96j X-Rspam-User: X-HE-Tag: 1696617992-528499 X-HE-Meta: U2FsdGVkX1/j996K6XfZqdG5cgLPp/IOhfyT/YW2pC8IzsIcvNzdzBMv8ZnxEHhM6q4/5+Jt14C29JBut8cAnaEirtQSGRAhSJ0QDsByd8v9/xS4aQpbPyVUoYFluijGlgNJPTljy5uq3dbw99FkJjXm8+pF496/aWYYTk3AsPPCfLSQkt5u5oJxbbzQHVk9dSpftvWMmr78QShzWSHaMXt8WbNfpj4XMZY25fMzXTUIzBFfY3ez/QQKNd70NTgnwk9GdnLbJn4nkLyaoOzrg+emyf7CUvG4ND3IaZtnLWu6XcODJgeFDXwn91lXkjiWYBvSTeUkEaR74M4PvYYKdYRvumfYDVDx681WnRlT4S0Q27j4GyXrhmc/ueVyhEdWoTOahNBZinG3+O8kPZn1WVUHjNTGUCjdIfbvFMKvvchvpnmr3VBNZ3HHkgykcX+44bc4P9vH0bhnkg8bvJviKpkd7vdu5mbFlRZrt2CZxcQU1UGz+t56rzGQg0WJVduJ/VJ1QJ4cVBbi1g8961kgf2IJ1nGoqQ2bOxy2I5ybKKm7t6l4rxyBMWPWPZLd3u3OLvpih3Pmpk/Y2XlSxdYLb8L4Jfbb32pVhXYPE7eFDGcfj5/3ilcSxVegz0GrxGGnX1zeCjA0ueU8Fp2JQuEgAwKJ3Do6amCveRUrAiQZp/MnDG0dpq5ARMqEMGsdaS9xpb/BRoFZvnRK6rg5Wcv57dP4yuL44RfaxH5DByQQ9IfusEgLYuPugSkC0+j1qAuN9VVrSXuMPOhKkwnLqx7hTknbPKmeXTUiigZDVtV8pLm422C1DXlryl+cGQ6lhYwvpMnJDUz1sbv3g50qfHA09O7kSljDxMnBe63Tz4oylGmtNatXD3DsisaRGGNCPwt4mIKx5iWNoFHrnsz0izdN8v8ZhN8GVkOy7Y55p9MhInCiTgd2+n5+JK6yr7OPQ3XQ/a9ThW6GVoNf8oNhd26 xr0Gl3py CnrIUFVLG5vU1F8JnRBqNk9T5LMq23Ukzz8Oe55pue9bssY9Nbcy+IjviUrGrZb7bNQgI1AKE8BMN5Sjv8B5X+tfRnEQBwmwvarLAjvCS3n8gFwfwyPwViwluslGE7AQPeqBWlOIf2cGEUx4jjCBib5eA2LuSd2L2zz/QWXPC3Nqwrf9O96mM8vdhKV7J3GVmO4I6ojQLxouqnAD3S60N0sayhx+AFD19GP9mZDni8o8pcSXYeX4LvK4oq5jIJegW5AfsPslHs8bvWJF8U85Y5/jG+INuyCkqPNIYKxPpF6So4VEku/z9UzcH4hSYOf5sBJxWOA+rhFabfcucpO6FzVQExrgVqJDloGbdr6jaB/Hq4IlIiBQKa3q0Rx584Bz2F7xzMHZ9J8cPNKnI+zv8bUB5TbegU58jqMUbdUhHdOP7cy4Z1lKld0o1htyexJss6e+5E4Bi4C1k0ZizpmMfjSK1bGAACv15ZmPdLSmuQAzrALARMd7YVV/eu/JT/QF+vzQV30qUM+JB6MvIH7J9PX1CeZpQrwraCviJrIpYO+jdjLo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This patch exposes charge committing and cancelling as parts of the memory controller interface. These functionalities are useful when the try_charge() and commit_charge() stages have to be separated by other actions in between (which can fail). One such example is the new hugetlb accounting behavior in the following patch. The patch also adds a helper function to obtain a reference to the current task's memcg. Acked-by: Michal Hocko Acked-by: Johannes Weiner Signed-off-by: Nhat Pham --- include/linux/memcontrol.h | 21 ++++++++++++++ mm/memcontrol.c | 59 ++++++++++++++++++++++++++++++-------- 2 files changed, 68 insertions(+), 12 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index e0cfab58ab71..42bf7e9b1a2f 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -653,6 +653,8 @@ static inline bool mem_cgroup_below_min(struct mem_cgroup *target, page_counter_read(&memcg->memory); } +void mem_cgroup_commit_charge(struct folio *folio, struct mem_cgroup *memcg); + int __mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, gfp_t gfp); /** @@ -704,6 +706,8 @@ static inline void mem_cgroup_uncharge_list(struct list_head *page_list) __mem_cgroup_uncharge_list(page_list); } +void mem_cgroup_cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages); + void mem_cgroup_migrate(struct folio *old, struct folio *new); /** @@ -760,6 +764,8 @@ struct mem_cgroup *mem_cgroup_from_task(struct task_struct *p); struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm); +struct mem_cgroup *get_mem_cgroup_from_current(void); + struct lruvec *folio_lruvec_lock(struct folio *folio); struct lruvec *folio_lruvec_lock_irq(struct folio *folio); struct lruvec *folio_lruvec_lock_irqsave(struct folio *folio, @@ -1245,6 +1251,11 @@ static inline bool mem_cgroup_below_min(struct mem_cgroup *target, return false; } +static inline void mem_cgroup_commit_charge(struct folio *folio, + struct mem_cgroup *memcg) +{ +} + static inline int mem_cgroup_charge(struct folio *folio, struct mm_struct *mm, gfp_t gfp) { @@ -1269,6 +1280,11 @@ static inline void mem_cgroup_uncharge_list(struct list_head *page_list) { } +static inline void mem_cgroup_cancel_charge(struct mem_cgroup *memcg, + unsigned int nr_pages) +{ +} + static inline void mem_cgroup_migrate(struct folio *old, struct folio *new) { } @@ -1306,6 +1322,11 @@ static inline struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) return NULL; } +static inline struct mem_cgroup *get_mem_cgroup_from_current(void) +{ + return NULL; +} + static inline struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css) { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index d1a322a75172..0219befeae38 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1086,6 +1086,27 @@ struct mem_cgroup *get_mem_cgroup_from_mm(struct mm_struct *mm) } EXPORT_SYMBOL(get_mem_cgroup_from_mm); +/** + * get_mem_cgroup_from_current - Obtain a reference on current task's memcg. + */ +struct mem_cgroup *get_mem_cgroup_from_current(void) +{ + struct mem_cgroup *memcg; + + if (mem_cgroup_disabled()) + return NULL; + +again: + rcu_read_lock(); + memcg = mem_cgroup_from_task(current); + if (!css_tryget(&memcg->css)) { + rcu_read_unlock(); + goto again; + } + rcu_read_unlock(); + return memcg; +} + static __always_inline bool memcg_kmem_bypass(void) { /* Allow remote memcg charging from any context. */ @@ -2873,7 +2894,12 @@ static inline int try_charge(struct mem_cgroup *memcg, gfp_t gfp_mask, return try_charge_memcg(memcg, gfp_mask, nr_pages); } -static inline void cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages) +/** + * mem_cgroup_cancel_charge() - cancel an uncommitted try_charge() call. + * @memcg: memcg previously charged. + * @nr_pages: number of pages previously charged. + */ +void mem_cgroup_cancel_charge(struct mem_cgroup *memcg, unsigned int nr_pages) { if (mem_cgroup_is_root(memcg)) return; @@ -2898,6 +2924,22 @@ static void commit_charge(struct folio *folio, struct mem_cgroup *memcg) folio->memcg_data = (unsigned long)memcg; } +/** + * mem_cgroup_commit_charge - commit a previously successful try_charge(). + * @folio: folio to commit the charge to. + * @memcg: memcg previously charged. + */ +void mem_cgroup_commit_charge(struct folio *folio, struct mem_cgroup *memcg) +{ + css_get(&memcg->css); + commit_charge(folio, memcg); + + local_irq_disable(); + mem_cgroup_charge_statistics(memcg, folio_nr_pages(folio)); + memcg_check_events(memcg, folio_nid(folio)); + local_irq_enable(); +} + #ifdef CONFIG_MEMCG_KMEM /* * The allocated objcg pointers array is not accounted directly. @@ -6105,7 +6147,7 @@ static void __mem_cgroup_clear_mc(void) /* we must uncharge all the leftover precharges from mc.to */ if (mc.precharge) { - cancel_charge(mc.to, mc.precharge); + mem_cgroup_cancel_charge(mc.to, mc.precharge); mc.precharge = 0; } /* @@ -6113,7 +6155,7 @@ static void __mem_cgroup_clear_mc(void) * we must uncharge here. */ if (mc.moved_charge) { - cancel_charge(mc.from, mc.moved_charge); + mem_cgroup_cancel_charge(mc.from, mc.moved_charge); mc.moved_charge = 0; } /* we must fixup refcnts and charges */ @@ -7020,20 +7062,13 @@ void mem_cgroup_calculate_protection(struct mem_cgroup *root, static int charge_memcg(struct folio *folio, struct mem_cgroup *memcg, gfp_t gfp) { - long nr_pages = folio_nr_pages(folio); int ret; - ret = try_charge(memcg, gfp, nr_pages); + ret = try_charge(memcg, gfp, folio_nr_pages(folio)); if (ret) goto out; - css_get(&memcg->css); - commit_charge(folio, memcg); - - local_irq_disable(); - mem_cgroup_charge_statistics(memcg, nr_pages); - memcg_check_events(memcg, folio_nid(folio)); - local_irq_enable(); + mem_cgroup_commit_charge(folio, memcg); out: return ret; } -- 2.34.1