From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B59B6C3ABDD for ; Fri, 16 May 2025 06:49:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A2E416B0092; Fri, 16 May 2025 02:49:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9DE766B00BF; Fri, 16 May 2025 02:49:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 87EB36B00C0; Fri, 16 May 2025 02:49:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 67D296B00BF for ; Fri, 16 May 2025 02:49:45 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 9A6891217A7 for ; Fri, 16 May 2025 06:49:46 +0000 (UTC) X-FDA: 83447845572.05.9BA9A93 Received: from out-185.mta0.migadu.com (out-185.mta0.migadu.com [91.218.175.185]) by imf28.hostedemail.com (Postfix) with ESMTP id D4676C0004 for ; Fri, 16 May 2025 06:49:44 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="WSR4/oQs"; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf28.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.185 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1747378185; a=rsa-sha256; cv=none; b=w/IiB9ngrl+1RHvNfJLCL/dn7xRb3kuRkFJzcjn7HewbpT03HtFCQwR4sH93zIvPFZrxFD dzqjnwhhPv00RI9E3BhKmrlr2h241x+2LXxHMIZHxfAvriPgB0cp+ix2ZhArgmoGHOkfPC Fesk7qEwYd2ODCnzfIEe65Fv5A2Br1w= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="WSR4/oQs"; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf28.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.185 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1747378185; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wjgeN2qjXaFT2yhGcnf6ojnEDqDgQ6jiTAUoDbWljT0=; b=KPhxhZrZDju2CVvaamZFtnVIiahp4L6rUf/WWKsZhNNelnZQHbb/kA+cPq08gFi1aBrPGE Lyb7AGLDe9krHtAjkgLr8RfcMLoYC1AIgaJcLxDVHQakhZdxaFh1fOsZhNmOw/0rQs7Tw/ WkZS2sGGriNraIcqPhUShuGNP03nA48= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1747378183; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=wjgeN2qjXaFT2yhGcnf6ojnEDqDgQ6jiTAUoDbWljT0=; b=WSR4/oQsdzkHgky5kgsfe7Q3covs1cTkeEfb8OaN46IWjcemjgSUOJOSDHT4tf9q8JeusI f3f8hayqZo9C32j3hvK/L+zPzmYInvz8GgshCBLLJP5qKhCIT7a0GiQI/HtJW0fGIIkigw dj+8morZoI+7sBShj5rXkhBOH0jZGtk= From: Shakeel Butt To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Vlastimil Babka , Alexei Starovoitov , Sebastian Andrzej Siewior , Harry Yoo , Yosry Ahmed , Peter Zijlstra , Mathieu Desnoyers , bpf@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team Subject: [PATCH 2/5] memcg: nmi safe memcg stats for specific archs Date: Thu, 15 May 2025 23:49:09 -0700 Message-ID: <20250516064912.1515065-3-shakeel.butt@linux.dev> In-Reply-To: <20250516064912.1515065-1-shakeel.butt@linux.dev> References: <20250516064912.1515065-1-shakeel.butt@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Stat-Signature: ha8kf154xigkp3cq98szfc1yut33sygn X-Rspam-User: X-Rspamd-Queue-Id: D4676C0004 X-Rspamd-Server: rspam06 X-HE-Tag: 1747378184-9060 X-HE-Meta: U2FsdGVkX1893DEZwZvmKdRDv3/bxGxZrnQwX4akOI6I/sfeCkcT9xq5MC99u7pe7SU4esiMKNul+kMQ4JHj++7nj5HpV0X1U8mbmx+xUJZeJriC4Agnivd0hb2Omhyu4NsM+D0m87Hx36iUHRVPbMX3CnxFfz5ys7cOdTXffAC9aPyQPQ6deTGUHaPpZfHwZhTW29xVLLWTiw2p6z1qbcRIhOyocEH6N/jvqKhEOe561A7P4EKl4olDIN5vtjBdCp2EbU0SsT0M5P1Ilw7bLzqgdgPqUs8xOX0XrZn3Y9DQFGcOdgiWkXWfptPH5IPxWjRe9QZcoQK6scnnXZrRA0yzGnDcmaapNxlr4Y4m/cHSlVU8IZCW9k+J10YxHquVDt6xUPAOD121b7wjCivxAU69b3nFi10Oy5srOGVUqnIQ4xH+7qpzz1kaSlHlPEbBHwr5esz/UMd7/w03jsMBfT6zRLFHisfn442kziG9nU8aYJuWhlUuaaoCkOPFfX7YO+/S2grYW+DldmO4ra57Fmma1rEkbG/4rPGU97ISv4pCLUUkOk6K6qXK9LgpfzCKqKSPeIEDulL//zQDXz0JQRUXKrP9X6Eknt4hGlc/vGu16efcbZYbBh0HUTASO7rqR4R5iVpBfY9mC9Fhf3sC+RvzaScq4XNoo+Dja4T2zrg9WhXyJ7GGCklYBxF4ro3ehtBKUjtc5lRHxyKno4XoO1ZvUQMLBXYktoetmnMMkO4ExDTgGh2QSun3e83GMiYox45TOjsSjuO2H6Fp0sW8NgEYCQsDXSQxzwp9T+gqubneDZVg2sr6K4sJQO57qIU9Thq+hpGiv7nPkB5sF3DIKKSlJtVLliuOQ93s5TpaiJErBO4rmZMCRWSzUsczPwza+l7gHU+JEiK0gOetHK97M29RUcbRUXzjsfedKtxpOkYDspCjj5i5udMOB0iT91uRS8MuDQkHTCVvHzFzRWz kFFS03L5 uUTSzIE/L5ZoKrQ1/EitruUv+Yxuwnb5fpCnxCS1JhogwvW+jiJpnhPfUPxq4/tAylZkDN9oBeFTX6P85bSCY/9bos5UuqIEHIxXBHGlUScsLmu4fQIG27V/kgJCdHpJ19sg+st4PDUfXI8Muxf+u1j06dHL/vxgPQxPJxGpMUvP2P+k= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There are archs which have NMI but does not support this_cpu_* ops safely in the nmi context but they support safe atomic ops in nmi context. For such archs, let's add infra to use atomic ops for the memcg stats which can be updated in nmi. At the moment, the memcg stats which get updated in the objcg charging path are MEMCG_KMEM, NR_SLAB_RECLAIMABLE_B & NR_SLAB_UNRECLAIMABLE_B. Rather than adding support for all memcg stats to be nmi safe, let's just add infra to make these three stats nmi safe which this patch is doing. Signed-off-by: Shakeel Butt --- include/linux/memcontrol.h | 20 ++++++++++++++-- mm/memcontrol.c | 49 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 67 insertions(+), 2 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 53920528821f..b10ae2388c27 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -62,9 +62,15 @@ struct mem_cgroup_reclaim_cookie { #ifdef CONFIG_MEMCG -#if defined(CONFIG_ARCH_HAS_NMI_SAFE_THIS_CPU_OPS) || \ - !defined(CONFIG_HAVE_NMI) || defined(ARCH_HAVE_NMI_SAFE_CMPXCHG) +#if defined(CONFIG_ARCH_HAS_NMI_SAFE_THIS_CPU_OPS) || !defined(CONFIG_HAVE_NMI) + +#define MEMCG_SUPPORTS_NMI_CHARGING + +#elif defined(ARCH_HAVE_NMI_SAFE_CMPXCHG) + #define MEMCG_SUPPORTS_NMI_CHARGING +#define MEMCG_NMI_NEED_ATOMIC + #endif #define MEM_CGROUP_ID_SHIFT 16 @@ -118,6 +124,12 @@ struct mem_cgroup_per_node { CACHELINE_PADDING(_pad2_); unsigned long lru_zone_size[MAX_NR_ZONES][NR_LRU_LISTS]; struct mem_cgroup_reclaim_iter iter; + +#ifdef MEMCG_NMI_NEED_ATOMIC + /* slab stats for nmi context */ + atomic_t slab_reclaimable; + atomic_t slab_unreclaimable; +#endif }; struct mem_cgroup_threshold { @@ -241,6 +253,10 @@ struct mem_cgroup { atomic_long_t memory_events[MEMCG_NR_MEMORY_EVENTS]; atomic_long_t memory_events_local[MEMCG_NR_MEMORY_EVENTS]; +#ifdef MEMCG_NMI_NEED_ATOMIC + /* MEMCG_KMEM for nmi context */ + atomic_t kmem_stat; +#endif /* * Hint of reclaim pressure for socket memroy management. Note * that this indicator should NOT be used in legacy cgroup mode diff --git a/mm/memcontrol.c b/mm/memcontrol.c index dface07f69bb..102fdec3f49e 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3979,6 +3979,53 @@ static void mem_cgroup_stat_aggregate(struct aggregate_control *ac) } } +#ifdef MEMCG_NMI_NEED_ATOMIC +static void flush_nmi_stats(struct mem_cgroup *memcg, struct mem_cgroup *parent, + int cpu) +{ + int nid; + + if (atomic_read(&memcg->kmem_stat)) { + int kmem = atomic_xchg(&memcg->kmem_stat, 0); + int index = memcg_stats_index(MEMCG_KMEM); + + memcg->vmstats->state[index] += kmem; + if (parent) + parent->vmstats->state_pending[index] += kmem; + } + + for_each_node_state(nid, N_MEMORY) { + struct mem_cgroup_per_node *pn = memcg->nodeinfo[nid]; + struct lruvec_stats *lstats = pn->lruvec_stats; + struct lruvec_stats *plstats = NULL; + + if (parent) + plstats = parent->nodeinfo[nid]->lruvec_stats; + + if (atomic_read(&pn->slab_reclaimable)) { + int slab = atomic_xchg(&pn->slab_reclaimable, 0); + int index = memcg_stats_index(NR_SLAB_RECLAIMABLE_B); + + lstats->state[index] += slab; + if (plstats) + plstats->state_pending[index] += slab; + } + if (atomic_read(&pn->slab_unreclaimable)) { + int slab = atomic_xchg(&pn->slab_unreclaimable, 0); + int index = memcg_stats_index(NR_SLAB_UNRECLAIMABLE_B); + + lstats->state[index] += slab; + if (plstats) + plstats->state_pending[index] += slab; + } + } +} +#else +static void flush_nmi_stats(struct mem_cgroup *memcg, struct mem_cgroup *parent, + int cpu) +{} +#endif + static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) { struct mem_cgroup *memcg = mem_cgroup_from_css(css); @@ -3987,6 +4034,8 @@ static void mem_cgroup_css_rstat_flush(struct cgroup_subsys_state *css, int cpu) struct aggregate_control ac; int nid; + flush_nmi_stats(memcg, parent, cpu); + statc = per_cpu_ptr(memcg->vmstats_percpu, cpu); ac = (struct aggregate_control) { -- 2.47.1