From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A0017C8303D for ; Fri, 4 Jul 2025 15:02:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 665D76B805A; Fri, 4 Jul 2025 11:02:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5EF1B6B8059; Fri, 4 Jul 2025 11:02:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4DECF6B805A; Fri, 4 Jul 2025 11:02:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3A6396B8059 for ; Fri, 4 Jul 2025 11:02:35 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 062B11249F2 for ; Fri, 4 Jul 2025 15:02:35 +0000 (UTC) X-FDA: 83626898670.12.230109C Received: from smtpout.efficios.com (smtpout.efficios.com [158.69.130.18]) by imf28.hostedemail.com (Postfix) with ESMTP id 62DCDC0004 for ; Fri, 4 Jul 2025 15:02:33 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=efficios.com header.s=smtpout1 header.b=q1GsSnwT; spf=pass (imf28.hostedemail.com: domain of mathieu.desnoyers@efficios.com designates 158.69.130.18 as permitted sender) smtp.mailfrom=mathieu.desnoyers@efficios.com; dmarc=pass (policy=none) header.from=efficios.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1751641353; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DIVG3dUfifCBnHlsCE0WEw4YgAUJ85fPZgs4Sd2dXrQ=; b=csNqH77l71aIDzfnQzvbUDiWfhk/uwbREupFwTpi3LOEjO0J+BG5Y4q5gZT4EiYu81umwD +ZQ5rFHm618ayPNaTwcdZQPSkma2xZ3vEwF5ndZmS+oUFMJ6uBM4cuEyYdmGD/H5zm5HTe pN1TnfiIVD3VgtA/PrLA2aBGeQ2IJcc= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=efficios.com header.s=smtpout1 header.b=q1GsSnwT; spf=pass (imf28.hostedemail.com: domain of mathieu.desnoyers@efficios.com designates 158.69.130.18 as permitted sender) smtp.mailfrom=mathieu.desnoyers@efficios.com; dmarc=pass (policy=none) header.from=efficios.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1751641353; a=rsa-sha256; cv=none; b=7hJMLHWMyX6J+inAiOMaDM4rDzWtL9fVM/CIDPjQuOn95Ut4BdemCm8VbgrCFqQaSMtd50 L3lA45ftDw5lysBsoKtHp/l1pnxUPte3anFyOg3t0abn51YTkEiEAW9qPDzIpaLbXZXShV Xp/ie9vao2H8RWB4+SO57+ufjvkaAQA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=efficios.com; s=smtpout1; t=1751641352; bh=j/PIZtj3C7WN2ujCAC9jz57UIbbgUtT4qWo/g2lwLIw=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=q1GsSnwTD6t/QRJRya0hIgysmTrfGFOu39CAJoDDORll1BppOw7yA02yHwIh4t0lS 419GI7OPGlP8mGGE7GWAut5tFF4JddOUMssP/crun7XRFBAMxDLG/2V7FNv4zXy5xE 1DrxE+OGYCvdo375mA7pY3kPFe7VxxNKGIz/UcB7mZQI7IYZ8nCabGt1ObW0RJsgJh 6orEYxsccKXodyWoB5SBrCQ+aIzSO5nBLv2L60sEiDGnTRBgn6dmFq3m2HXCOyPxyz b/AmKpLmJHJVjislUse/5j4TqMwgEYm8UxTEYe53llRPw4u3auN3ugZVA5+Wjsq+U8 fzuA5tLc1n/Ow== Received: from thinkos.internal.efficios.com (192-222-132-26.qc.cable.ebox.net [192.222.132.26]) by smtpout.efficios.com (Postfix) with ESMTPSA id 4bYcLc1KMlz7B8; Fri, 4 Jul 2025 11:02:32 -0400 (EDT) From: Mathieu Desnoyers To: Andrew Morton Cc: linux-kernel@vger.kernel.org, Mathieu Desnoyers , "Paul E. McKenney" , Steven Rostedt , Masami Hiramatsu , Dennis Zhou , Tejun Heo , Christoph Lameter , Martin Liu , David Rientjes , christian.koenig@amd.com, Shakeel Butt , Johannes Weiner , Sweet Tea Dorminy , Lorenzo Stoakes , "Liam R . Howlett" , Suren Baghdasaryan , Vlastimil Babka , Christian Brauner , Wei Yang , David Hildenbrand , Miaohe Lin , Al Viro , linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, Yu Zhao , Roman Gushchin , Mateusz Guzik , Matthew Wilcox Subject: [RFC PATCH v6 2/2] mm: Fix OOM killer inaccuracy on large many-core systems Date: Fri, 4 Jul 2025 11:02:26 -0400 Message-Id: <20250704150226.47980-3-mathieu.desnoyers@efficios.com> X-Mailer: git-send-email 2.39.5 In-Reply-To: <20250704150226.47980-1-mathieu.desnoyers@efficios.com> References: <20250704150226.47980-1-mathieu.desnoyers@efficios.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Stat-Signature: 3ye497m7x9k67sg3s3gbhb69monecfdp X-Rspamd-Queue-Id: 62DCDC0004 X-Rspamd-Server: rspam11 X-Rspam-User: X-HE-Tag: 1751641353-387076 X-HE-Meta: U2FsdGVkX18jgUAVEHf+ZCXuhInDbRbBgzD8g/xUKOX7vDKUf6+EWO2VFGGyAVPx940cLGXO18XGaPKU8h3L4GF36iCW7PjG4ATM3emcPSP+BtqX93yLTHraqsDinh1S4R6SgmpSmEjiIM1ulq35irF8C7gTuV0SdQVXbmB62c6Iz3y3WfniUXQMHFysDiTLOG2XsF4CnU/7/HzMXGtZCtB7h/x4TMG53I+lYEtU7sKZ4G1T5EVjtElNtWYRWC2LNayFPHVshOSoVmqzJ+PzQ5by6Q/R5lkStxIKy5FMttHrh1kDWSVkwY+Q+i3QRa3WMEp9hFbUbcJWkTg78OndZY6AaxRSbtCnXq3RbWZJhDrqphvgvDXw2hYNzkwIN1KlTOHLnPx1rOw8f/JYrKp66tuGBU/oLZeo1YvXbFw/TQcjgC0N53N31YBSz4pZtB6cjXoNRtuHgXR1Q6G1lPand7XplZi/QQlinQK60Mi3O/B3E1Qq5ve/ohQyn0gKS1L89NRhqnfJhhoze7ubCR/PDRVizT5GDbM8g+Bf2hbh2NTUSpW/sYqpA6fNM8FfC0Lc5oJNQUiV/8RjKXvvrMJvKGPWKrHymIkXqEmy6rLNNDuOYBQMYt8p5CXDD9Ki0NZB4wCBdZCLS7ontZsUhgB49jX9gi2oYkS0/jpyXm3rTUSM+BET21Blc+wxwyODq+T5RTn22oqlFIGvvRAj9JPnNlqPqcDx3TAVE3fFTka6R6rT6y8JfN8a/Qj7PAzum8BQP9PDcA8cALH80IOICp1TAUX8hTXJSyTPxqFkFL6TpbS/tUyRquLMFGs7cfg2kfOLmTNv3lxKxegif1PgFi2zKB3BENY0SHYaHndDUf9S8R6Y+OsQWJYPKlJa4C3nZdFIr1xL+X3GYV5DuVmU5DXKtDVqe++zlU5ZOwsaFsxS5WZOXnTBg5ls/Gfd5hZMyEKk2xIDoP43PTSzE5FEyIg NiyX7XDk 4gdOkp7igRjZKYO3ygB9bIUnPi6krh3oWMf65ZwEYPP5fHMjTJkMbh+BZY2ZMghv3C+MnNMW/NSnYIuZJYsysB+y3NIV9CiXSwVvBKuIZNm/ujvXKLmcD08yOClDsB7QNCSYiQDgKSYOFRJE4FTPmT10YJxUhyn8n3QinHpAe8/61RJnzv/0IzG8ndEG0iuV5rzq8Xg0lDQv264smYZ/F5JNpjlfWmmHzzoNBDMoFQ95m8aW/sKV+RT/pRDjorxb0UbAngMd2NCvsVh4e8o15EqReYLCpXQyXQLoIv/B68a1p1+Ss6yGmg+UiJkeQfk2mg9FypS58uLeBweOPvqB9cWwVkJmlxI967u+AsI5lPGfogirZVb/rXNe431B8ga0SR+ztfCQd1lbgSp2IACyV6JD2wS+V96y6fS9+228MLu14o0pPiT5bZfUvMcJJyAqobwQQdo/3O6/eXbvo5DGo5g+zkVDbbz4FMUUjNB6eTUkJ7ZKigN9Vf+m4+s4ZXLpsToAqLexmBFH6PoMJpEk8R2yMzDPlm/P07cgSuurtgcJrJIDb3+2TTkMSE0hfYATToy/vMSloKlqStuFoJ7/wKsfrSRAiuJwaymU36fDz7g5W+5Uzssb8XeNrRKXPV6ovZlEW6CGHs5lAorW3QDzMN1WmhxjX2KLTa8OOBsQSVjOc/xHMPMI5c6CBor1f3zc2j51cfSkLMdl/nBSUntQ/J6p895zePLVrS9FQ7aydIFsVdkZslKrIuZzoU1f6reZw16wvyFsCzojnZ+DN/UYh7YQQYtvwfWfYdOXtZtUxMbzTEQlRhYlZvckyyQ3M93Nf4LU/z+na/5FeRUfrPZGfQLyFTdBKLkAsr0TznO3GdFFGgpg1nFMKNYsGfCY41Z5WFAYrLVhIr78vLZBiclOBtOzyiTkHR0IExnCbmGWAJaSPVZKG5T86oj/GooxBmIQSPKepZ2NCqcFlAclqPzq7Ghni0cLA fQP7w5aA NsK6ja6M3nY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use hierarchical per-cpu counters for rss tracking to fix the per-mm RSS tracking which has become too inaccurate for OOM killer purposes on large many-core systems. The following rss tracking issues were noted by Sweet Tea Dorminy [1], which lead to picking wrong tasks as OOM kill target: Recently, several internal services had an RSS usage regression as part of a kernel upgrade. Previously, they were on a pre-6.2 kernel and were able to read RSS statistics in a backup watchdog process to monitor and decide if they'd overrun their memory budget. Now, however, a representative service with five threads, expected to use about a hundred MB of memory, on a 250-cpu machine had memory usage tens of megabytes different from the expected amount -- this constituted a significant percentage of inaccuracy, causing the watchdog to act. This was a result of f1a7941243c1 ("mm: convert mm's rss stats into percpu_counter") [1]. Previously, the memory error was bounded by 64*nr_threads pages, a very livable megabyte. Now, however, as a result of scheduler decisions moving the threads around the CPUs, the memory error could be as large as a gigabyte. This is a really tremendous inaccuracy for any few-threaded program on a large machine and impedes monitoring significantly. These stat counters are also used to make OOM killing decisions, so this additional inaccuracy could make a big difference in OOM situations -- either resulting in the wrong process being killed, or in less memory being returned from an OOM-kill than expected. Here is a (possibly incomplete) list of the prior approaches that were used or proposed, along with their downside: 1) Per-thread rss tracking: large error on many-thread processes. 2) Per-CPU counters: up to 12% slower for short-lived processes and 9% increased system time in make test workloads [1]. Moreover, the inaccuracy increases with O(n^2) with the number of CPUs. 3) Per-NUMA-node counters: requires atomics on fast-path (overhead), error is high with systems that have lots of NUMA nodes (32 times the number of NUMA nodes). The approach proposed here is to replace this by the hierarchical per-cpu counters, which bounds the inaccuracy based on the system topology with O(N*logN). Link: https://lore.kernel.org/lkml/20250331223516.7810-2-sweettea-kernel@dorminy.me/ # [1] Signed-off-by: Mathieu Desnoyers Cc: Andrew Morton Cc: "Paul E. McKenney" Cc: Steven Rostedt Cc: Masami Hiramatsu Cc: Mathieu Desnoyers Cc: Dennis Zhou Cc: Tejun Heo Cc: Christoph Lameter Cc: Martin Liu Cc: David Rientjes Cc: christian.koenig@amd.com Cc: Shakeel Butt Cc: Johannes Weiner Cc: Sweet Tea Dorminy Cc: Lorenzo Stoakes Cc: "Liam R . Howlett" Cc: Suren Baghdasaryan Cc: Vlastimil Babka Cc: Christian Brauner Cc: Wei Yang Cc: David Hildenbrand Cc: Miaohe Lin Cc: Al Viro Cc: linux-mm@kvack.org Cc: linux-trace-kernel@vger.kernel.org Cc: Yu Zhao Cc: Roman Gushchin Cc: Mateusz Guzik Cc: Matthew Wilcox --- Changes since v5: - Use percpu_counter_tree_approximate_sum_positive. Change since v4: - get_mm_counter needs to return 0 or a positive value. --- include/linux/mm.h | 8 ++++---- include/linux/mm_types.h | 4 ++-- include/trace/events/kmem.h | 2 +- kernel/fork.c | 31 +++++++++++++++++++++---------- 4 files changed, 28 insertions(+), 17 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index e51dba8398f7..934a296e5e33 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2705,28 +2705,28 @@ static inline bool get_user_page_fast_only(unsigned long addr, */ static inline unsigned long get_mm_counter(struct mm_struct *mm, int member) { - return percpu_counter_read_positive(&mm->rss_stat[member]); + return percpu_counter_tree_approximate_sum_positive(&mm->rss_stat[member]); } void mm_trace_rss_stat(struct mm_struct *mm, int member); static inline void add_mm_counter(struct mm_struct *mm, int member, long value) { - percpu_counter_add(&mm->rss_stat[member], value); + percpu_counter_tree_add(&mm->rss_stat[member], value); mm_trace_rss_stat(mm, member); } static inline void inc_mm_counter(struct mm_struct *mm, int member) { - percpu_counter_inc(&mm->rss_stat[member]); + percpu_counter_tree_add(&mm->rss_stat[member], 1); mm_trace_rss_stat(mm, member); } static inline void dec_mm_counter(struct mm_struct *mm, int member) { - percpu_counter_dec(&mm->rss_stat[member]); + percpu_counter_tree_add(&mm->rss_stat[member], -1); mm_trace_rss_stat(mm, member); } diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 56d07edd01f9..85b15109106a 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -18,7 +18,7 @@ #include #include #include -#include +#include #include #include @@ -1059,7 +1059,7 @@ struct mm_struct { unsigned long saved_auxv[AT_VECTOR_SIZE]; /* for /proc/PID/auxv */ - struct percpu_counter rss_stat[NR_MM_COUNTERS]; + struct percpu_counter_tree rss_stat[NR_MM_COUNTERS]; struct linux_binfmt *binfmt; diff --git a/include/trace/events/kmem.h b/include/trace/events/kmem.h index f74925a6cf69..f4558d705dbd 100644 --- a/include/trace/events/kmem.h +++ b/include/trace/events/kmem.h @@ -477,7 +477,7 @@ TRACE_EVENT(rss_stat, __entry->mm_id = mm_ptr_to_hash(mm); __entry->curr = !!(current->mm == mm); __entry->member = member; - __entry->size = (percpu_counter_sum_positive(&mm->rss_stat[member]) + __entry->size = (percpu_counter_tree_approximate_sum_positive(&mm->rss_stat[member]) << PAGE_SHIFT); ), diff --git a/kernel/fork.c b/kernel/fork.c index 168681fc4b25..dd458adc5543 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -129,6 +129,11 @@ */ #define MAX_THREADS FUTEX_TID_MASK +/* + * Batch size of rss stat approximation + */ +#define RSS_STAT_BATCH_SIZE 32 + /* * Protected counters by write_lock_irq(&tasklist_lock) */ @@ -843,11 +848,10 @@ static void check_mm(struct mm_struct *mm) "Please make sure 'struct resident_page_types[]' is updated as well"); for (i = 0; i < NR_MM_COUNTERS; i++) { - long x = percpu_counter_sum(&mm->rss_stat[i]); - - if (unlikely(x)) - pr_alert("BUG: Bad rss-counter state mm:%p type:%s val:%ld\n", - mm, resident_page_types[i], x); + if (unlikely(percpu_counter_tree_precise_compare_value(&mm->rss_stat[i], 0) != 0)) + pr_alert("BUG: Bad rss-counter state mm:%p type:%s val:%d\n", + mm, resident_page_types[i], + percpu_counter_tree_precise_sum(&mm->rss_stat[i])); } if (mm_pgtables_bytes(mm)) @@ -930,6 +934,8 @@ static void cleanup_lazy_tlbs(struct mm_struct *mm) */ void __mmdrop(struct mm_struct *mm) { + int i; + BUG_ON(mm == &init_mm); WARN_ON_ONCE(mm == current->mm); @@ -945,8 +951,8 @@ void __mmdrop(struct mm_struct *mm) put_user_ns(mm->user_ns); mm_pasid_drop(mm); mm_destroy_cid(mm); - percpu_counter_destroy_many(mm->rss_stat, NR_MM_COUNTERS); - + for (i = 0; i < NR_MM_COUNTERS; i++) + percpu_counter_tree_destroy(&mm->rss_stat[i]); free_mm(mm); } EXPORT_SYMBOL_GPL(__mmdrop); @@ -1285,6 +1291,8 @@ static void mmap_init_lock(struct mm_struct *mm) static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, struct user_namespace *user_ns) { + int i; + mt_init_flags(&mm->mm_mt, MM_MT_FLAGS); mt_set_external_lock(&mm->mm_mt, &mm->mmap_lock); atomic_set(&mm->mm_users, 1); @@ -1332,15 +1340,18 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, if (mm_alloc_cid(mm, p)) goto fail_cid; - if (percpu_counter_init_many(mm->rss_stat, 0, GFP_KERNEL_ACCOUNT, - NR_MM_COUNTERS)) - goto fail_pcpu; + for (i = 0; i < NR_MM_COUNTERS; i++) { + if (percpu_counter_tree_init(&mm->rss_stat[i], RSS_STAT_BATCH_SIZE, GFP_KERNEL_ACCOUNT)) + goto fail_pcpu; + } mm->user_ns = get_user_ns(user_ns); lru_gen_init_mm(mm); return mm; fail_pcpu: + for (i--; i >= 0; i--) + percpu_counter_tree_destroy(&mm->rss_stat[i]); mm_destroy_cid(mm); fail_cid: destroy_context(mm); -- 2.39.5