From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4DE9EC433DB for ; Tue, 9 Mar 2021 10:09:10 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AD50D65257 for ; Tue, 9 Mar 2021 10:09:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AD50D65257 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=bytedance.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4D4258D00D8; Tue, 9 Mar 2021 05:09:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4AA958D007F; Tue, 9 Mar 2021 05:09:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 324068D00D8; Tue, 9 Mar 2021 05:09:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0067.hostedemail.com [216.40.44.67]) by kanga.kvack.org (Postfix) with ESMTP id 1660D8D007F for ; Tue, 9 Mar 2021 05:09:09 -0500 (EST) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id BFF1C180AD81D for ; Tue, 9 Mar 2021 10:09:08 +0000 (UTC) X-FDA: 77899912776.14.63660D3 Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) by imf26.hostedemail.com (Postfix) with ESMTP id 2A9404080F50 for ; Tue, 9 Mar 2021 10:09:06 +0000 (UTC) Received: by mail-pf1-f173.google.com with SMTP id a188so9165330pfb.4 for ; Tue, 09 Mar 2021 02:09:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=5/zxOOlFck4GlBFK2+ckmxR9mJWNjZboWgW6z95uHA0=; b=Ry1qrRc+5lI1iGnqqWVM4o17Gn43+ZLX4Fjh/ZhiXcHTUhGi2F8ZTOgb9Sd+veN5a5 L2OWCkrr0L1E8ufr1WICKVuju21XlWWG2a9vqYSdH8unRX2h6JYDR8RbcuovmMkWng4N PRRbTCYooxzYrVJ3YmFpAXrIEdBC7SLhlbKaxUTtabygFdRPE1CjhCKvU8VAO+a5JyAg mxG4PlbHrhfhyXBvh5JOqFVBvsgCibil77LD09iPtq83BrI4HPNWTzMggFd0hgyAYxgA dOquiJ+yiFcZI5nDV/xrwvbjPJ3En7Oh7I2ANROqFF4BBF/kaOeKMHQ6IBmzIRgwWIQr uTxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=5/zxOOlFck4GlBFK2+ckmxR9mJWNjZboWgW6z95uHA0=; b=mspzlUS82IL4Wj5gq6uicnmR//h1PN9ozAmHcQHGS9ml5/MabbRUT5vIjOgi+cO09l nhPzSXlxRIftOPFQoajaW+MdyC8eMV5REcURa7oSahz+3BZMlEXqayNyVf5Qsugk0o7A phrpdq/LF36AbrPhiIkZ6Vuhx9HrCGA2/ONRk0pJsbPaa9C6hNpR6lARhLv0is5SLqQG JFbiJ1EVdthaUixmwRqckLLpICWCm8jC60T2/UtLaOv1sc41LokYsLcMakn0tt/ZuO/X NWd8QUCz9vSkhMiHggOzI8mp16B2F5ieY/2CMO7uPcLsm5BgyT5bauV+3d31Re4ZyzN8 ha6w== X-Gm-Message-State: AOAM532QeB2RBB/aLfccmsbFBzdrms5bNdJ0idT/0R6yP6zddz/LF0Zy 0EHiZAwD6+da/TlQmHYaUSjZ0w== X-Google-Smtp-Source: ABdhPJwZ6+mzvwKJVap7GfMIaRdnbYNCKvspqSspeZGIQo0L9r5nG6rhOHbzHqOBBynBPD6y/NczBg== X-Received: by 2002:a62:5a43:0:b029:1ed:263a:b05c with SMTP id o64-20020a625a430000b02901ed263ab05cmr24839949pfb.16.1615284547311; Tue, 09 Mar 2021 02:09:07 -0800 (PST) Received: from localhost.localdomain ([139.177.225.255]) by smtp.gmail.com with ESMTPSA id a70sm9258424pfa.202.2021.03.09.02.09.03 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Mar 2021 02:09:06 -0800 (PST) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, Muchun Song Subject: [PATCH v3 3/4] mm: memcontrol: use obj_cgroup APIs to charge kmem pages Date: Tue, 9 Mar 2021 18:07:16 +0800 Message-Id: <20210309100717.253-4-songmuchun@bytedance.com> X-Mailer: git-send-email 2.21.0 (Apple Git-122) In-Reply-To: <20210309100717.253-1-songmuchun@bytedance.com> References: <20210309100717.253-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 2A9404080F50 X-Stat-Signature: 1dy67dxrbwf1p8wtrmbdt753qrhf9gy4 Received-SPF: none (bytedance.com>: No applicable sender policy available) receiver=imf26; identity=mailfrom; envelope-from=""; helo=mail-pf1-f173.google.com; client-ip=209.85.210.173 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615284546-146044 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since Roman series "The new cgroup slab memory controller" applied. All slab objects are charged via the new APIs of obj_cgroup. The new APIs introduce a struct obj_cgroup to charge slab objects. It prevents long-living objects from pinning the original memory cgroup in the memory= . But there are still some corner objects (e.g. allocations larger than order-1 page on SLUB) which are not charged via the new APIs. Those objects (include the pages which are allocated from buddy allocator directly) are charged as kmem pages which still hold a reference to the memory cgroup. This patch aims to charge the kmem pages by using the new APIs of obj_cgroup. Finally, the page->memcg_data of the kmem page points to an object cgroup. We can use the page_objcg() to get the object cgroup associated with a kmem page. Or we can use page_memcg_check() to get the memory cgroup associated with a kmem page, but caller must ensure that the returned memcg won't be released (e.g. acquire the rcu_read_lock or css_set_lock). Signed-off-by: Muchun Song --- include/linux/memcontrol.h | 63 ++++++++++++++++++------ mm/memcontrol.c | 119 ++++++++++++++++++++++++++++++---------= ------ 2 files changed, 128 insertions(+), 54 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 83cbcdcfcc92..07c449af9c0f 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -370,6 +370,18 @@ static inline bool page_memcg_charged(struct page *p= age) } =20 /* + * After the initialization objcg->memcg is always pointing at + * a valid memcg, but can be atomically swapped to the parent memcg. + * + * The caller must ensure that the returned memcg won't be released: + * e.g. acquire the rcu_read_lock or css_set_lock. + */ +static inline struct mem_cgroup *obj_cgroup_memcg(struct obj_cgroup *obj= cg) +{ + return READ_ONCE(objcg->memcg); +} + +/* * page_memcg - get the memory cgroup associated with a non-kmem page * @page: a pointer to the page struct * @@ -422,15 +434,19 @@ static inline struct mem_cgroup *page_memcg_rcu(str= uct page *page) * @page: a pointer to the page struct * * Returns a pointer to the memory cgroup associated with the page, - * or NULL. This function unlike page_memcg() can take any page + * or NULL. This function unlike page_memcg() can take any page * as an argument. It has to be used in cases when it's not known if a p= age - * has an associated memory cgroup pointer or an object cgroups vector. + * has an associated memory cgroup pointer or an object cgroups vector o= r + * an object cgroup. * * Any of the following ensures page and memcg binding stability: * - the page lock * - LRU isolation * - lock_page_memcg() * - exclusive reference + * + * Should be called under rcu lock which can protect memcg associated wi= th a + * kmem page from being released. */ static inline struct mem_cgroup *page_memcg_check(struct page *page) { @@ -443,6 +459,13 @@ static inline struct mem_cgroup *page_memcg_check(st= ruct page *page) if (memcg_data & MEMCG_DATA_OBJCGS) return NULL; =20 + if (memcg_data & MEMCG_DATA_KMEM) { + struct obj_cgroup *objcg; + + objcg =3D (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + return obj_cgroup_memcg(objcg); + } + return (struct mem_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); } =20 @@ -501,6 +524,25 @@ static inline struct obj_cgroup **page_objcgs_check(= struct page *page) return (struct obj_cgroup **)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); } =20 +/* + * page_objcg - get the object cgroup associated with a kmem page + * @page: a pointer to the page struct + * + * Returns a pointer to the object cgroup associated with the kmem page, + * or NULL. This function assumes that the page is known to have an + * associated object cgroup. It's only safe to call this function + * against kmem pages (PageMemcgKmem() returns true). + */ +static inline struct obj_cgroup *page_objcg(struct page *page) +{ + unsigned long memcg_data =3D page->memcg_data; + + VM_BUG_ON_PAGE(PageSlab(page), page); + VM_BUG_ON_PAGE(memcg_data & MEMCG_DATA_OBJCGS, page); + VM_BUG_ON_PAGE(!(memcg_data & MEMCG_DATA_KMEM), page); + + return (struct obj_cgroup *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); +} #else static inline struct obj_cgroup **page_objcgs(struct page *page) { @@ -511,6 +553,11 @@ static inline struct obj_cgroup **page_objcgs_check(= struct page *page) { return NULL; } + +static inline struct obj_cgroup *page_objcg(struct page *page) +{ + return NULL; +} #endif =20 static __always_inline bool memcg_stat_item_in_bytes(int idx) @@ -729,18 +776,6 @@ static inline void obj_cgroup_put(struct obj_cgroup = *objcg) percpu_ref_put(&objcg->refcnt); } =20 -/* - * After the initialization objcg->memcg is always pointing at - * a valid memcg, but can be atomically swapped to the parent memcg. - * - * The caller must ensure that the returned memcg won't be released: - * e.g. acquire the rcu_read_lock or css_set_lock. - */ -static inline struct mem_cgroup *obj_cgroup_memcg(struct obj_cgroup *obj= cg) -{ - return READ_ONCE(objcg->memcg); -} - static inline void mem_cgroup_put(struct mem_cgroup *memcg) { if (memcg) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e1dc73ceb98a..38376f9d6659 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -859,15 +859,26 @@ void __mod_lruvec_page_state(struct page *page, enu= m node_stat_item idx, pg_data_t *pgdat =3D page_pgdat(page); struct lruvec *lruvec; =20 - memcg =3D page_memcg_check(head); - /* Untracked pages have no memcg, no lruvec. Update only the node */ - if (!memcg) { - __mod_node_page_state(pgdat, idx, val); - return; + if (PageMemcgKmem(head)) { + rcu_read_lock(); + memcg =3D obj_cgroup_memcg(page_objcg(page)); + } else { + memcg =3D page_memcg(head); + /* + * Untracked pages have no memcg, no lruvec. Update only the + * node. + */ + if (!memcg) { + __mod_node_page_state(pgdat, idx, val); + return; + } } =20 lruvec =3D mem_cgroup_lruvec(memcg, pgdat); __mod_lruvec_state(lruvec, idx, val); + + if (PageMemcgKmem(head)) + rcu_read_unlock(); } EXPORT_SYMBOL(__mod_lruvec_page_state); =20 @@ -2906,6 +2917,20 @@ static void commit_charge(struct page *page, struc= t mem_cgroup *memcg) page->memcg_data =3D (unsigned long)memcg; } =20 +static inline struct mem_cgroup *obj_cgroup_memcg_get(struct obj_cgroup = *objcg) +{ + struct mem_cgroup *memcg; + + rcu_read_lock(); +retry: + memcg =3D obj_cgroup_memcg(objcg); + if (unlikely(!css_tryget(&memcg->css))) + goto retry; + rcu_read_unlock(); + + return memcg; +} + #ifdef CONFIG_MEMCG_KMEM int memcg_alloc_page_obj_cgroups(struct page *page, struct kmem_cache *s= , gfp_t gfp, bool new_page) @@ -3071,15 +3096,8 @@ static int obj_cgroup_charge_pages(struct obj_cgro= up *objcg, gfp_t gfp, struct mem_cgroup *memcg; int ret; =20 - rcu_read_lock(); -retry: - memcg =3D obj_cgroup_memcg(objcg); - if (unlikely(!css_tryget(&memcg->css))) - goto retry; - rcu_read_unlock(); - + memcg =3D obj_cgroup_memcg_get(objcg); ret =3D __memcg_kmem_charge(memcg, gfp, nr_pages); - css_put(&memcg->css); =20 return ret; @@ -3144,18 +3162,18 @@ static void __memcg_kmem_uncharge(struct mem_cgro= up *memcg, unsigned int nr_page */ int __memcg_kmem_charge_page(struct page *page, gfp_t gfp, int order) { - struct mem_cgroup *memcg; + struct obj_cgroup *objcg; int ret =3D 0; =20 - memcg =3D get_mem_cgroup_from_current(); - if (memcg && !mem_cgroup_is_root(memcg)) { - ret =3D __memcg_kmem_charge(memcg, gfp, 1 << order); + objcg =3D get_obj_cgroup_from_current(); + if (objcg) { + ret =3D obj_cgroup_charge_pages(objcg, gfp, 1 << order); if (!ret) { - page->memcg_data =3D (unsigned long)memcg | + page->memcg_data =3D (unsigned long)objcg | MEMCG_DATA_KMEM; return 0; } - css_put(&memcg->css); + obj_cgroup_put(objcg); } return ret; } @@ -3167,17 +3185,16 @@ int __memcg_kmem_charge_page(struct page *page, g= fp_t gfp, int order) */ void __memcg_kmem_uncharge_page(struct page *page, int order) { - struct mem_cgroup *memcg; + struct obj_cgroup *objcg; unsigned int nr_pages =3D 1 << order; =20 if (!page_memcg_charged(page)) return; =20 - memcg =3D page_memcg_check(page); - VM_BUG_ON_PAGE(mem_cgroup_is_root(memcg), page); - __memcg_kmem_uncharge(memcg, nr_pages); + objcg =3D page_objcg(page); + obj_cgroup_uncharge_pages(objcg, nr_pages); page->memcg_data =3D 0; - css_put(&memcg->css); + obj_cgroup_put(objcg); } =20 static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_= bytes) @@ -6806,11 +6823,23 @@ static inline void uncharge_gather_clear(struct u= ncharge_gather *ug) static void uncharge_batch(const struct uncharge_gather *ug) { unsigned long flags; + unsigned long nr_pages; =20 - if (!mem_cgroup_is_root(ug->memcg)) { - page_counter_uncharge(&ug->memcg->memory, ug->nr_pages); + /* + * The kmem pages can be reparented to the root memcg, in + * order to prevent the memory counter of root memcg from + * increasing indefinitely. We should decrease the memory + * counter when unchange. + */ + if (mem_cgroup_is_root(ug->memcg)) + nr_pages =3D ug->nr_kmem; + else + nr_pages =3D ug->nr_pages; + + if (nr_pages) { + page_counter_uncharge(&ug->memcg->memory, nr_pages); if (do_memsw_account()) - page_counter_uncharge(&ug->memcg->memsw, ug->nr_pages); + page_counter_uncharge(&ug->memcg->memsw, nr_pages); if (!cgroup_subsys_on_dfl(memory_cgrp_subsys) && ug->nr_kmem) page_counter_uncharge(&ug->memcg->kmem, ug->nr_kmem); memcg_oom_recover(ug->memcg); @@ -6828,7 +6857,7 @@ static void uncharge_batch(const struct uncharge_ga= ther *ug) =20 static void uncharge_page(struct page *page, struct uncharge_gather *ug) { - unsigned long nr_pages; + unsigned long nr_pages, nr_kmem; struct mem_cgroup *memcg; =20 VM_BUG_ON_PAGE(PageLRU(page), page); @@ -6836,34 +6865,44 @@ static void uncharge_page(struct page *page, stru= ct uncharge_gather *ug) if (!page_memcg_charged(page)) return; =20 + nr_pages =3D compound_nr(page); /* * Nobody should be changing or seriously looking at - * page memcg at this point, we have fully exclusive - * access to the page. + * page memcg or objcg at this point, we have fully + * exclusive access to the page. */ - memcg =3D page_memcg_check(page); + if (PageMemcgKmem(page)) { + struct obj_cgroup *objcg; + + objcg =3D page_objcg(page); + memcg =3D obj_cgroup_memcg_get(objcg); + + page->memcg_data =3D 0; + obj_cgroup_put(objcg); + nr_kmem =3D nr_pages; + } else { + memcg =3D page_memcg(page); + page->memcg_data =3D 0; + nr_kmem =3D 0; + } + if (ug->memcg !=3D memcg) { if (ug->memcg) { uncharge_batch(ug); uncharge_gather_clear(ug); } ug->memcg =3D memcg; + ug->dummy_page =3D page; =20 /* pairs with css_put in uncharge_batch */ css_get(&ug->memcg->css); } =20 - nr_pages =3D compound_nr(page); ug->nr_pages +=3D nr_pages; + ug->nr_kmem +=3D nr_kmem; + ug->pgpgout +=3D !nr_kmem; =20 - if (PageMemcgKmem(page)) - ug->nr_kmem +=3D nr_pages; - else - ug->pgpgout++; - - ug->dummy_page =3D page; - page->memcg_data =3D 0; - css_put(&ug->memcg->css); + css_put(&memcg->css); } =20 /** --=20 2.11.0