From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7550CC3ABA9 for ; Fri, 2 May 2025 00:18:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 881BB6B00A6; Thu, 1 May 2025 20:18:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 830F56B00A9; Thu, 1 May 2025 20:18:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F9936B00AC; Thu, 1 May 2025 20:18:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4FD386B00A6 for ; Thu, 1 May 2025 20:18:11 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 406EEC16DD for ; Fri, 2 May 2025 00:18:12 +0000 (UTC) X-FDA: 83396055624.20.98D1391 Received: from out-189.mta0.migadu.com (out-189.mta0.migadu.com [91.218.175.189]) by imf03.hostedemail.com (Postfix) with ESMTP id 8A90820005 for ; Fri, 2 May 2025 00:18:10 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=GNaYOfNm; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf03.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.189 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1746145090; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=z8ptQwJSFfLo2cDiqIGklIojumEAEh3QOUi7xbHY5m8=; b=u5v/zOaQ2seVifFcJsL/wu3wt7o7CvXi3gn/773TgupCW9KLbqOb5JtU0pA0RURvnuQwcC FpFqlSpIyp3Gb47vlAZuMgkWGZhSN6Nz+z7oM8kf2HePiNSy1HXSEWajwot0zq6mhRGiW5 RCrIPBCW/zhrjFuTJmdjHlZg81ubLiM= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=GNaYOfNm; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf03.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.189 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1746145090; a=rsa-sha256; cv=none; b=Qgkxhtlq1F5YcZJZb+4w1uQOHIu3T49+7hAf0uC21EMnJfV+dTJlYqA7QrbZZhIFJRLJff LTcC7tIiXlFnubcgvPdw5NNrOInMcmWxEpgGauq8rz/R0fV0bN0faUFjf1SPWkrtG/J0Dm ELvfJyxBlHzTjLuC4OjArWGavdj15Gg= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1746145088; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=z8ptQwJSFfLo2cDiqIGklIojumEAEh3QOUi7xbHY5m8=; b=GNaYOfNmpJl8BUzBF6IJvETURlGcX7+mT1FmFkX5jN+8dtI6ZDkgvV4isSNPLKcFIn8J3Z mOF846iWFKWTtdFEtu+vqTfGLylxg97fRQwXvP3QwlZ5p6lE7YONP7RLjUb1f613dMdYYK Z5pvu04FsO8rvWt3jdEsOZ2S8KWaJ24= From: Shakeel Butt To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Alexei Starovoitov , linux-mm@kvack.org, cgroups@vger.kernel.org, bpf@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH v2 1/3] memcg: separate local_trylock for memcg and obj Date: Thu, 1 May 2025 17:17:40 -0700 Message-ID: <20250502001742.3087558-2-shakeel.butt@linux.dev> In-Reply-To: <20250502001742.3087558-1-shakeel.butt@linux.dev> References: <20250502001742.3087558-1-shakeel.butt@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Stat-Signature: 7omkaztm938m11hqpz7jmwc5h5piham7 X-Rspamd-Queue-Id: 8A90820005 X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1746145090-881437 X-HE-Meta: U2FsdGVkX1+hSuEGty/EmeVIqcxBpYSmE5I0InJiZbEGaHilAz5OdDPvNCQE+iH/Va96PynS761nPNyo4KwFRKEPJxKgp9ZCZMxFwsMQ3vL8a0PHn4DUWAEN/UyvSJByOpUAqmhIloMmTQ9JliNj+xNr+S0Zb96FpSqbDTGqlsUlkqLVzSQnbYSY8lz+5Zf5QEXmQfkY/L3YAgq+ir8GcDLoLqFHVoUThDlIGpLcTgtGN4gzieYjvs8UexB9EXMF44ru9g8GHBv6oBVnkU62KJ2YgKk3qAAeZp7fYDWXBznOoZ0uAmrZ6tehsagBDgq+yLIpK/8Eu0xKQxIi/50UVhuMpvf9GkiynO1ndynd8bOvrTDFT3h7ZuXX/zOPMvjHca7ebItTiaqGTKIEQQE2PdOfjkd8mpgw6VlQELtoukYu18D4ZzhfXnXrLCNoWSkRjEhJU0n+vcePpWZFUvTRXRQ46cdWWHG455Hh0zfQGHXlNA0AVndUOQY2E1ZnwXXJ6PQgM3hkXOb5ZD2B24qU5vigZ381HgONjd+E9QupOC+Moy3kwoCjVSwx3UV3OtJi2KOJmybV1PE5fJOF9EGrSEJBeWXBHmeSaNZjUqP/++AqplEeMCr3p9x+xTM2dMXXKYV1FyUukqEqKYaPrGbyYZKop0osKxVg5PCXcfztr8tdYHsfmVLZ4aREOk/sYXzTCtTw/RwnFmkKGbRPILrjgfIcp6tGsehCpqH+D5hV8JCFrUNRzvzPu3pl/AzIq7zjNL/k8pcX17/+C/GyCtnhU7Gf20UqtL61HAPb/3PJn7BIv/oSXToQIcGf86Kgj+C3taprnnyXMsd83EmUf1LSS5e1u+11ThWZ8x3ENlLmMoTUXHoH1Bdb4qrdS1yuPJHYzGRZe/YqJJXHtdbFRwhSR+n136jf4fJVhu/uzm+PMQ2fUcCKdntRbVv0OmO0XigPjxKzmLbQCcmPCeVSUCs JZw8SJwx Nv9ZizHVfGgvw9XxePtOlm1ALdlhXfmshislOw7IibpPPM82awmd04za+pJstN22YeguZYcGJMYuzfMZA6lDeYNHDCzhDZ+kJbu0d1dYHOp+5u1kPDPVSQld2FGsfItSWBYtCFwDI/BnRvHFjDjCUCEGE3BW/6ZqYEZa20/EUSYHAlUiF29v5szurIsI1h4A6fzePMkJ+qG8F7/w+TURUAsoM9fLR0SMKq7oL X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The per-cpu stock_lock protects cached memcg and cached objcg and their respective fields. However there is no dependency between these fields and it is better to have fine grained separate locks for cached memcg and cached objcg. This decoupling of locks allows us to make the memcg charge cache and objcg charge cache to be nmi safe independently. At the moment, memcg charge cache is already nmi safe and this decoupling will allow to make memcg charge cache work without disabling irqs. Signed-off-by: Shakeel Butt --- Changes since v1: - Drop usage of preempt_disable() as suggested by Vlastimil. mm/memcontrol.c | 51 ++++++++++++++++++++++++++----------------------- 1 file changed, 27 insertions(+), 24 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 0d42699bb564..14714e1d36e9 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1806,13 +1806,14 @@ void mem_cgroup_print_oom_group(struct mem_cgroup *memcg) */ #define NR_MEMCG_STOCK 7 struct memcg_stock_pcp { - local_trylock_t stock_lock; + local_trylock_t memcg_lock; uint8_t nr_pages[NR_MEMCG_STOCK]; struct mem_cgroup *cached[NR_MEMCG_STOCK]; + local_trylock_t obj_lock; + unsigned int nr_bytes; struct obj_cgroup *cached_objcg; struct pglist_data *cached_pgdat; - unsigned int nr_bytes; int nr_slab_reclaimable_b; int nr_slab_unreclaimable_b; @@ -1821,7 +1822,8 @@ struct memcg_stock_pcp { #define FLUSHING_CACHED_CHARGE 0 }; static DEFINE_PER_CPU_ALIGNED(struct memcg_stock_pcp, memcg_stock) = { - .stock_lock = INIT_LOCAL_TRYLOCK(stock_lock), + .memcg_lock = INIT_LOCAL_TRYLOCK(memcg_lock), + .obj_lock = INIT_LOCAL_TRYLOCK(obj_lock), }; static DEFINE_MUTEX(percpu_charge_mutex); @@ -1854,8 +1856,8 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages, return ret; if (gfpflags_allow_spinning(gfp_mask)) - local_lock_irqsave(&memcg_stock.stock_lock, flags); - else if (!local_trylock_irqsave(&memcg_stock.stock_lock, flags)) + local_lock_irqsave(&memcg_stock.memcg_lock, flags); + else if (!local_trylock_irqsave(&memcg_stock.memcg_lock, flags)) return ret; stock = this_cpu_ptr(&memcg_stock); @@ -1872,7 +1874,7 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages, break; } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + local_unlock_irqrestore(&memcg_stock.memcg_lock, flags); return ret; } @@ -1918,19 +1920,19 @@ static void drain_local_stock(struct work_struct *dummy) struct memcg_stock_pcp *stock; unsigned long flags; - /* - * The only protection from cpu hotplug (memcg_hotplug_cpu_dead) vs. - * drain_stock races is that we always operate on local CPU stock - * here with IRQ disabled - */ - local_lock_irqsave(&memcg_stock.stock_lock, flags); + if (WARN_ONCE(!in_task(), "drain in non-task context")) + return; + local_lock_irqsave(&memcg_stock.obj_lock, flags); stock = this_cpu_ptr(&memcg_stock); drain_obj_stock(stock); + local_unlock_irqrestore(&memcg_stock.obj_lock, flags); + + local_lock_irqsave(&memcg_stock.memcg_lock, flags); + stock = this_cpu_ptr(&memcg_stock); drain_stock_fully(stock); clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); - - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + local_unlock_irqrestore(&memcg_stock.memcg_lock, flags); } static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) @@ -1953,10 +1955,10 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) VM_WARN_ON_ONCE(mem_cgroup_is_root(memcg)); if (nr_pages > MEMCG_CHARGE_BATCH || - !local_trylock_irqsave(&memcg_stock.stock_lock, flags)) { + !local_trylock_irqsave(&memcg_stock.memcg_lock, flags)) { /* * In case of larger than batch refill or unlikely failure to - * lock the percpu stock_lock, uncharge memcg directly. + * lock the percpu memcg_lock, uncharge memcg directly. */ memcg_uncharge(memcg, nr_pages); return; @@ -1988,7 +1990,7 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) WRITE_ONCE(stock->nr_pages[i], nr_pages); } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + local_unlock_irqrestore(&memcg_stock.memcg_lock, flags); } static bool is_drain_needed(struct memcg_stock_pcp *stock, @@ -2063,11 +2065,12 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) stock = &per_cpu(memcg_stock, cpu); - /* drain_obj_stock requires stock_lock */ - local_lock_irqsave(&memcg_stock.stock_lock, flags); + /* drain_obj_stock requires obj_lock */ + local_lock_irqsave(&memcg_stock.obj_lock, flags); drain_obj_stock(stock); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + local_unlock_irqrestore(&memcg_stock.obj_lock, flags); + /* no need for the local lock */ drain_stock_fully(stock); return 0; @@ -2920,7 +2923,7 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, unsigned long flags; bool ret = false; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + local_lock_irqsave(&memcg_stock.obj_lock, flags); stock = this_cpu_ptr(&memcg_stock); if (objcg == READ_ONCE(stock->cached_objcg) && stock->nr_bytes >= nr_bytes) { @@ -2931,7 +2934,7 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, __account_obj_stock(objcg, stock, nr_bytes, pgdat, idx); } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + local_unlock_irqrestore(&memcg_stock.obj_lock, flags); return ret; } @@ -3020,7 +3023,7 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, unsigned long flags; unsigned int nr_pages = 0; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + local_lock_irqsave(&memcg_stock.obj_lock, flags); stock = this_cpu_ptr(&memcg_stock); if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */ @@ -3042,7 +3045,7 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, stock->nr_bytes &= (PAGE_SIZE - 1); } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + local_unlock_irqrestore(&memcg_stock.obj_lock, flags); if (nr_pages) obj_cgroup_uncharge_pages(objcg, nr_pages); -- 2.47.1