From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55E6CC48BC3 for ; Mon, 19 Feb 2024 08:30:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CE4278D0005; Mon, 19 Feb 2024 03:30:18 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C948D8D0001; Mon, 19 Feb 2024 03:30:18 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B0DCB8D0005; Mon, 19 Feb 2024 03:30:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A05D28D0001 for ; Mon, 19 Feb 2024 03:30:18 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 48D1F1C01F7 for ; Mon, 19 Feb 2024 08:30:18 +0000 (UTC) X-FDA: 81807881316.22.7520CEA Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.223.130]) by imf03.hostedemail.com (Postfix) with ESMTP id C82522001C for ; Mon, 19 Feb 2024 08:30:15 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=ZNalOCFR; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=u9F1A7Q+; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=ZNalOCFR; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=u9F1A7Q+; spf=pass (imf03.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708331416; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/nmFrGbKHKgGYmJjLYoolA8lLxxnfPyDdiArr554cgk=; b=UBSp7IBrgEPFrlRjUarUPgdWDaDfcEjtLEvBRNa0dafhWjwUG8E70ntfvttdp8Ml56PNhN nxuwjt4WuPCV6+WNpGaxmy1pVzaVU3wa9h2Nlo0VoMHFexx4jqVp8lLKPyOS1WYQ0iYFpM pF1vXTd7A1t1kI9+ylG57IVxBo2X4aU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708331416; a=rsa-sha256; cv=none; b=PVXZk0KmmBYMJz+Dg+vxyDyfFDuNKisQbvevDL7S+CYxjgrZawfEwXvmyVBVYRHOVAF2jW wzgNdWqHnt9mgh4fs/7p2J48YUyONIxLKIjMqwPTUFV7vPUT+7SpMvsOvE0+5tjtvjZjK0 2t/bsE1SSUESfxeQLmDa2DoU8u7qW7E= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=ZNalOCFR; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=u9F1A7Q+; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=ZNalOCFR; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=u9F1A7Q+; spf=pass (imf03.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.130 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id 6DAA0222FC; Mon, 19 Feb 2024 08:30:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1708331412; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/nmFrGbKHKgGYmJjLYoolA8lLxxnfPyDdiArr554cgk=; b=ZNalOCFRO6AI3F8f9uAAmKb+uNRopyNXwXONZ9Z9dkOh3I6AGGxQKAARrWZlQTQXFT1RAU Ykq+n0hBsVU3iEx8Ga/rd3EodCUV5zfCA0gVXteAR/u3zuHg9kuDtO6QP2DxIcrM10xCM1 ZBMv8GqOmdcvZqta28VyvBE6NrrPy1M= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1708331412; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/nmFrGbKHKgGYmJjLYoolA8lLxxnfPyDdiArr554cgk=; b=u9F1A7Q+BcCO6J2MU930pVReTp1WgEXuPqbRBQESidkfCa5+hfUbmAFNc10BAe3MJdPOeC XtCsMmNHKs2WVjBg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1708331412; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/nmFrGbKHKgGYmJjLYoolA8lLxxnfPyDdiArr554cgk=; b=ZNalOCFRO6AI3F8f9uAAmKb+uNRopyNXwXONZ9Z9dkOh3I6AGGxQKAARrWZlQTQXFT1RAU Ykq+n0hBsVU3iEx8Ga/rd3EodCUV5zfCA0gVXteAR/u3zuHg9kuDtO6QP2DxIcrM10xCM1 ZBMv8GqOmdcvZqta28VyvBE6NrrPy1M= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1708331412; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/nmFrGbKHKgGYmJjLYoolA8lLxxnfPyDdiArr554cgk=; b=u9F1A7Q+BcCO6J2MU930pVReTp1WgEXuPqbRBQESidkfCa5+hfUbmAFNc10BAe3MJdPOeC XtCsMmNHKs2WVjBg== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 5260513647; Mon, 19 Feb 2024 08:30:12 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id /LzNE5QR02WNBgAAD6G6ig (envelope-from ); Mon, 19 Feb 2024 08:30:12 +0000 Message-ID: Date: Mon, 19 Feb 2024 09:30:11 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH] slub: avoid scanning all partial slabs in get_slabinfo() Content-Language: en-US To: David Rientjes , Jianfeng Wang Cc: cl@linux.com, penberg@kernel.org, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Chengming Zhou References: <20240215211457.32172-1-jianfeng.w.wang@oracle.com> <6b58d81f-8e8f-3732-a5d4-40eece75013b@google.com> From: Vlastimil Babka In-Reply-To: <6b58d81f-8e8f-3732-a5d4-40eece75013b@google.com> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: C82522001C X-Rspam-User: X-Stat-Signature: 94zdx98rgy574zknj63es4yrydzzeutd X-Rspamd-Server: rspam03 X-HE-Tag: 1708331415-793785 X-HE-Meta: U2FsdGVkX18pK+SoOru1hosuEoLcRqO7VqZM/YYeo7gtctLvd+RHoX3dXjoipg2qOkt123oFYOsTECeUDfpzeS5fcauDI46WvPeHwId68meKFledQxN3rR4wijpeVjwTh+6RKroQ3vwMKcw9q5Fmav7Ks9fme3jrACtg3oj8FeGj2tX7zOpM2dLp6E7giarsZnz+acehrQ+18W0DKpYViKsmTneNv+7VOD/4A7t0Z3eXOZv31jyEmWeMNgX5FJCny9jHP4oKgSoX57kwbZT7EbrT6NDYzezMp+eivTAUXdpPfxGbvkbYs7iIUeUM+IdwdOTqw9cF2eydhCp/UgB/pBlYm9ZIJ2J70jLmH+C+XeD5YkWjo6B8OoixAiRbfV7P3Df69EcVgpIQtW84gg/TP+2kNUK07xiw/tefctndGWUf66bOeYM4VAYy2QM/szcE9qrsqlOeMfDP6ZKZcjU5SMIgF/X/pBBDLssB+5Nvezw2tW31qoKDP5jxV6kKyLUjJ99RCmunsUnPcwvugi7C/qsxCRwIeaC23oQNSoZm/F9XMAf9sGfAZYPVyfK24sjmoX1CDX9ABDqu756T714sXte5qvSKwe9yJKy4eFhQmc4nX8BafZSI5/d2HtePPfvc/YFTCDoH9Zn452wOA0nyx8Qa1CeDqhg1I53JIfn14BP4ceHQJz7qIiOBqapkbxU+9fzyDmhVq64E1NUtzciYt3rJa5X89DPTnZ5j+L9cemj/cG15W32DnDcMFIe2uLFwkiKsqTyg6C1BAbdajdZTum5nB4Tgac1U6eboUbJuVSlt9E08WsG/rOEvrWt2CJn56dTzoCYrsW3L4vVQUR+If7NfcP6R6skVlEq7BH6eG54Qh+j4ZPUbuGz8hsKvOFpyTEIKYuwZSUDTK7/mvajd/SpNxhV08Ki2kZWzU4WxfZrKb+TfCcV75lo3qmlXkUiR6lyEOWFkiJ0XzKKTphu eP14thQJ 3uLVuwnCk7XyugXpEgWElDkHVh4Qy0f2TZ6kLaDjwrU0+1edEybxYuKpFYHVZNZxEOpIKWjOFNBfTGP5KxdrJIbMtlrfN2Oig18El8CCwoNu/Sk9zrYTYEX8H05VUPRlhZAVBfjtWTfj8wNyHNj9SObrgsR6jiS0l3je6wiyI8DnWxf+S5MltsjwiWiKhsrhF7leirW8PmxUjBq/6wiP0ITJEYVGdFMjRgRmiTMrl/qEEy99WvfWhzCrV/6Y1FWuE+6Ai+Mwb9VZcDYQWvStB0KATAZ5mkxNlkJsv1gXYrUx2Q40T/gbUUXusq0PA36Kx4WH/BsGKqrMT/0OuXJtCDBKDjPS0u1J7RLk/R7Bj66gqkRrmwh5IfJXnZMW/psemkPXUXtCrGodBkfUd0TRdI+PS+I81rZfb8FfRWzagfHOR/V1px73FddDUkLmDR5NT9Z6yhil4NR6x2Yx0fysfP/29AT0RVWEcwnFIpZJ9XfTMX4qNR0ks5kG+RnLZQHmoROC28nY1ZhQx65Q= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2/18/24 20:25, David Rientjes wrote: > On Thu, 15 Feb 2024, Jianfeng Wang wrote: > >> When reading "/proc/slabinfo", the kernel needs to report the number of >> free objects for each kmem_cache. The current implementation relies on >> count_partial() that counts the number of free objects by scanning each >> kmem_cache_node's partial slab list and summing free objects from all >> partial slabs in the list. This process must hold per kmem_cache_node >> spinlock and disable IRQ. Consequently, it can block slab allocation >> requests on other CPU cores and cause timeouts for network devices etc., >> if the partial slab list is long. In production, even NMI watchdog can >> be triggered because some slab caches have a long partial list: e.g., >> for "buffer_head", the number of partial slabs was observed to be ~1M >> in one kmem_cache_node. This problem was also observed by several >> others [1-2] in the past. >> >> The fix is to maintain a counter of free objects for each kmem_cache. >> Then, in get_slabinfo(), use the counter rather than count_partial() >> when reporting the number of free objects for a slab cache. per-cpu >> counter is used to minimize atomic or lock operation. >> >> Benchmark: run hackbench on a dual-socket 72-CPU bare metal machine >> with 256 GB memory and Intel(R) Xeon(R) CPU E5-2699 v3 @ 2.3 GHz. >> The command is "hackbench 18 thread 20000". Each group gets 10 runs. >> > > This seems particularly intrusive for the common path to optimize for > reading of /proc/slabinfo, and that's shown in the benchmark result. > > Could you discuss the /proc/slabinfo usage model a bit? It's not clear if > this is being continuously read, or whether even a single read in > isolation is problematic. > > That said, optimizing for reading /proc/slabinfo at the cost of runtime > performance degradation doesn't sound like the right trade-off. It should be possible to make this overhead smaller by restricting the counter only to partial list slabs, as [2] did. This would keep it out of the fast paths, where it's really not acceptable. Note [2] used atomic_long_t and the percpu counters used here should be lower overhead. So basically try to get the best of both attemps. >> Results: >> - Mainline: >> 21.0381 +- 0.0325 seconds time elapsed ( +- 0.15% ) >> - Mainline w/ this patch: >> 21.1878 +- 0.0239 seconds time elapsed ( +- 0.11% ) >> >> [1] https://lore.kernel.org/linux-mm/ >> alpine.DEB.2.21.2003031602460.1537@www.lameter.com/T/ >> [2] https://lore.kernel.org/lkml/ >> alpine.DEB.2.22.394.2008071258020.55871@www.lameter.com/T/ >> >> Signed-off-by: Jianfeng Wang >> --- >> mm/slab.h | 4 ++++ >> mm/slub.c | 31 +++++++++++++++++++++++++++++-- >> 2 files changed, 33 insertions(+), 2 deletions(-) >> >> diff --git a/mm/slab.h b/mm/slab.h >> index 54deeb0428c6..a0e7672ba648 100644 >> --- a/mm/slab.h >> +++ b/mm/slab.h >> @@ -11,6 +11,7 @@ >> #include >> #include >> #include >> +#include >> >> /* >> * Internal slab definitions >> @@ -277,6 +278,9 @@ struct kmem_cache { >> unsigned int red_left_pad; /* Left redzone padding size */ >> const char *name; /* Name (only for display!) */ >> struct list_head list; /* List of slab caches */ >> +#ifdef CONFIG_SLUB_DEBUG >> + struct percpu_counter free_objects; >> +#endif >> #ifdef CONFIG_SYSFS >> struct kobject kobj; /* For sysfs */ >> #endif >> diff --git a/mm/slub.c b/mm/slub.c >> index 2ef88bbf56a3..44f8ded96574 100644 >> --- a/mm/slub.c >> +++ b/mm/slub.c >> @@ -736,6 +736,12 @@ static inline bool slab_update_freelist(struct kmem_cache *s, struct slab *slab, >> static unsigned long object_map[BITS_TO_LONGS(MAX_OBJS_PER_PAGE)]; >> static DEFINE_SPINLOCK(object_map_lock); >> >> +static inline void >> +__update_kmem_cache_free_objs(struct kmem_cache *s, s64 delta) >> +{ >> + percpu_counter_add_batch(&s->free_objects, delta, INT_MAX); >> +} >> + >> static void __fill_map(unsigned long *obj_map, struct kmem_cache *s, >> struct slab *slab) >> { >> @@ -1829,6 +1835,9 @@ slab_flags_t kmem_cache_flags(unsigned int object_size, >> return flags | slub_debug_local; >> } >> #else /* !CONFIG_SLUB_DEBUG */ >> +static inline void >> +__update_kmem_cache_free_objs(struct kmem_cache *s, s64 delta) {} >> + >> static inline void setup_object_debug(struct kmem_cache *s, void *object) {} >> static inline >> void setup_slab_debug(struct kmem_cache *s, struct slab *slab, void *addr) {} >> @@ -2369,6 +2378,7 @@ static struct slab *allocate_slab(struct kmem_cache *s, gfp_t flags, int node) >> slab->inuse = 0; >> slab->frozen = 0; >> >> + __update_kmem_cache_free_objs(s, slab->objects); >> account_slab(slab, oo_order(oo), s, flags); >> >> slab->slab_cache = s; >> @@ -2445,6 +2455,7 @@ static void free_slab(struct kmem_cache *s, struct slab *slab) >> call_rcu(&slab->rcu_head, rcu_free_slab); >> else >> __free_slab(s, slab); >> + __update_kmem_cache_free_objs(s, -slab->objects); >> } >> >> static void discard_slab(struct kmem_cache *s, struct slab *slab) >> @@ -3859,6 +3870,8 @@ static __fastpath_inline void *slab_alloc_node(struct kmem_cache *s, struct list >> */ >> slab_post_alloc_hook(s, objcg, gfpflags, 1, &object, init, orig_size); >> >> + if (object) >> + __update_kmem_cache_free_objs(s, -1); >> return object; >> } >> >> @@ -4235,6 +4248,7 @@ static __always_inline void do_slab_free(struct kmem_cache *s, >> unsigned long tid; >> void **freelist; >> >> + __update_kmem_cache_free_objs(s, cnt); >> redo: >> /* >> * Determine the currently cpus per cpu slab. >> @@ -4286,6 +4300,7 @@ static void do_slab_free(struct kmem_cache *s, >> struct slab *slab, void *head, void *tail, >> int cnt, unsigned long addr) >> { >> + __update_kmem_cache_free_objs(s, cnt); >> __slab_free(s, slab, head, tail, cnt, addr); >> } >> #endif /* CONFIG_SLUB_TINY */ >> @@ -4658,6 +4673,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, >> memcg_slab_alloc_error_hook(s, size, objcg); >> } >> >> + __update_kmem_cache_free_objs(s, -i); >> return i; >> } >> EXPORT_SYMBOL(kmem_cache_alloc_bulk); >> @@ -4899,6 +4915,9 @@ void __kmem_cache_release(struct kmem_cache *s) >> cache_random_seq_destroy(s); >> #ifndef CONFIG_SLUB_TINY >> free_percpu(s->cpu_slab); >> +#endif >> +#ifdef CONFIG_SLUB_DEBUG >> + percpu_counter_destroy(&s->free_objects); >> #endif >> free_kmem_cache_nodes(s); >> } >> @@ -5109,6 +5128,14 @@ static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) >> s->random = get_random_long(); >> #endif >> >> +#ifdef CONFIG_SLUB_DEBUG >> + int ret; >> + >> + ret = percpu_counter_init(&s->free_objects, 0, GFP_KERNEL); >> + if (ret) >> + return ret; >> +#endif >> + >> if (!calculate_sizes(s)) >> goto error; >> if (disable_higher_order_debug) { >> @@ -7100,15 +7127,15 @@ void get_slabinfo(struct kmem_cache *s, struct slabinfo *sinfo) >> { >> unsigned long nr_slabs = 0; >> unsigned long nr_objs = 0; >> - unsigned long nr_free = 0; >> + unsigned long nr_free; >> int node; >> struct kmem_cache_node *n; >> >> for_each_kmem_cache_node(s, node, n) { >> nr_slabs += node_nr_slabs(n); >> nr_objs += node_nr_objs(n); >> - nr_free += count_partial(n, count_free); >> } >> + nr_free = percpu_counter_sum_positive(&s->free_objects); >> >> sinfo->active_objs = nr_objs - nr_free; >> sinfo->num_objs = nr_objs; >> -- >> 2.42.1 >> >>