From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5798EC4167B for ; Tue, 31 Oct 2023 14:09:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D3BF96B02FD; Tue, 31 Oct 2023 10:09:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CED5C6B02FE; Tue, 31 Oct 2023 10:09:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BDBAC6B0300; Tue, 31 Oct 2023 10:09:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id AFD106B02FD for ; Tue, 31 Oct 2023 10:09:03 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 8B145120A44 for ; Tue, 31 Oct 2023 14:09:03 +0000 (UTC) X-FDA: 81405938166.09.625108E Received: from out-184.mta1.migadu.com (out-184.mta1.migadu.com [95.215.58.184]) by imf14.hostedemail.com (Postfix) with ESMTP id 608F4100018 for ; Tue, 31 Oct 2023 14:09:00 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=p+5YJNbM; spf=pass (imf14.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.184 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698761340; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=vdLIZ4HQ998mSidy99GBLdIh30G2cHpNbjGYcZ/M/84=; b=2Uou0AiZOvQly5pDO5IuY2/RmTLbfwRQfvY/gOoPq54vu2Qkh9RKFkD5tOtE2M2QZoF7J0 gdPyjHSbixc2wJp3lm/yDvhHCY8IgkdcxWq0uAxJ3WF6ZuvMOfwLh/jtHKpKB8D5w1Lubf ge/F4z+6PnM6G6PC+PTQZmlvncoM/ms= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698761340; a=rsa-sha256; cv=none; b=zCTynF0D4u3Q0XKdffkQsT/hb25Z1TDaHyn2em9X9R46ozRiHP6o2M4hg6eril8aQ5v7ue HO8PsqWduMZQ/ZGRDIjneWkOqkquS+BSSORgtv7P65zeFvhf5a8RQHh30dc1L8tqW4PTZj As86bqRGFYkVqOVM5afvzEKfC3QPMl8= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=p+5YJNbM; spf=pass (imf14.hostedemail.com: domain of chengming.zhou@linux.dev designates 95.215.58.184 as permitted sender) smtp.mailfrom=chengming.zhou@linux.dev; dmarc=pass (policy=none) header.from=linux.dev X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1698761338; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=vdLIZ4HQ998mSidy99GBLdIh30G2cHpNbjGYcZ/M/84=; b=p+5YJNbM7e/sK1WxMNjw8pKd/1/3/bovhNq+XaMIz4UveYORIbck2aiUG0YXcuvA2XLaS/ tzYIGfCQY1h9jgoNsST3gAMMrUgN0F9wyxtg+u7h6lEjqTOkL/zWbIuywgdmE8mlPWZ11g xnirByVbnC7o+g6WllDelNK24JkGeNo= From: chengming.zhou@linux.dev To: vbabka@suse.cz, cl@linux.com, penberg@kernel.org, willy@infradead.org Cc: rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, chengming.zhou@linux.dev, Chengming Zhou Subject: [RFC PATCH v4 0/9] slub: Delay freezing of CPU partial slabs Date: Tue, 31 Oct 2023 14:07:32 +0000 Message-Id: <20231031140741.79387-1-chengming.zhou@linux.dev> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 608F4100018 X-Rspam-User: X-Stat-Signature: 9ktbzs9b9kubp4dtuprbnppswbsdgpgn X-Rspamd-Server: rspam03 X-HE-Tag: 1698761340-18904 X-HE-Meta: U2FsdGVkX1+C7rXZzX+8XOMasyAoovPZ7Gjc4ZIuLHcrRLgK+AX6L8JBQiVMhCIj0c2XKDcRGBsfUDCyGsGqlPz6YxD533Ea+UybxYnPp4PQ0qRWK/eiQ50Wse9iy/BogiesqTTTAeDJ0WPP1A1DnDekgHaKbPjeOyyfpFnR+V/ei+cg+xW0b9pEbAsRhO/GPW9MdOdOac2iQvj+Vm0L2ssYVgBo26c0UjQ3bci0XSXd/eiYIfV07TJ32CdsJH0zLVpS5L3bHuGKBw93hOctH/2L8GYqbm/fWtyWL+PFE3nVgnnShJdvSW4tIn4uGhCiAUOVZgdGh+HctxfXGDQPEZnGuXqL62MFNx5LDTKaD1SZe0+R2z75/9FGooS/I4v8SQWNAUT2WT1zi3I9Vh5gWlE5P+jrK7RJbJb7y6Lbq6sz6tE7+hxE334ZuMtqcbqmz+7tI1H+cRghp6/vmN//MOyhgEqebvdIEACAQlbgsnQ2RNXvVQbJZiC9BvTjJrFLWAq/6D5rqNyps+8DDXUlW7JC3VDDsIeR2nHrlFz9R8cetyPy1T5lJUYXIftk9N272OqeKGJkiH8tGMWVMfR5utpKwLWeUt+hX1oeIE/Ini1W3B5iqNkRdG7EA+EspQJcKwFd7CQeaL+KHFXkKLkoiPjrbxC8m4jAE0cF2HyHAsd2uaeXMf1B2PuK1fQAobS0DKqM5I0tGpYlE/KoL4yoyV+XMFiSMn/4hw0YUCP+AOlVEEz2uzmNnw7A2q5cM2vCFiGMilHGkB+M4JFUxAr55zg/D/mUw4AIyVGirvD8NVA8VTIA43bnbGvx2Dq7bc0H7rITl08K8051hvj2noTcjc7SGk/ntr5FfEkuP9b/l3KaFlSlkO/1EBB0uGdSXrBY5QFuzmKYwqJPMHm5/bt2YmrKNyBbB8xSg6sD+DYcT0Oj0BhBp+3Avb6jUZlSbj/1SdZzT+UWTQUQZqde4kT LnhoTken lGdXaOK+bBc5yjJcyNMYj2oOKjCppcSAk/YgmooT926qDD6zyOdotIJYn1DpgW9z/RIvzTcZ+g5zfGEKa65+yECoXtLyN36BnzTA4SWBr/iblrm9CRhom/Blp1umgNDbbAUOZ8kLcnDk+I4ccQ3XdUOrO9gFTtUb3RugUd+KKrsHlcz4YlDOXEymQ1K13t9mrvGA0l1YlkG5nt6CgSRpn6ozRLIG0LsjkoW6BQ20mIOM7M5kJ6w2T3lc3jZ17LUd/3uYYkT/zlg3Nwd8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Chengming Zhou Changes in RFC v4: - Reorder patches to put the two cleanup patches to the front. - Move slab_node_partial flag functions to mm/slub.c. - Fix freeze_slab() by using slab_update_freelist(). - Fix build error when !CONFIG_SLUB_CPU_PARTIAL. - Add a patch to rename all *unfreeze_partials* functions. - Add a patch to update inconsistent documentations in the source. - Some comments and changelog improvements. - Add Reviewed-by and Suggested-by tags. Many thanks! - RFC v3: https://lore.kernel.org/all/20231024093345.3676493-1-chengming.zhou@linux.dev/ Changes in RFC v3: - Directly use __set_bit() and __clear_bit() for the slab_node_partial flag operations to avoid exporting non-atomic "workingset" interfaces. - Change get_partial() related functions to return a slab instead of returning the freelist or single object. - Don't freeze any slab under the node list_lock to further reduce list_lock holding times, as suggested by Vlastimil Babka. - Introduce freeze_slab() to do the delay freezing and return freelist. - Reorder patches. - RFC v2: https://lore.kernel.org/all/20231021144317.3400916-1-chengming.zhou@linux.dev/ Changes in RFC v2: - Reuse PG_workingset bit to keep track of whether slub is on the per-node partial list, as suggested by Matthew Wilcox. - Fix OOM problem on kernel without CONFIG_SLUB_CPU_PARTIAL, which is caused by leak of partial slabs when get_partial_node(). - Add a patch to simplify acquire_slab(). - Reorder patches a little. - RFC v1: https://lore.kernel.org/all/20231017154439.3036608-1-chengming.zhou@linux.dev/ 1. Problem ========== Now we have to freeze the slab when get from the node partial list, and unfreeze the slab when put to the node partial list. Because we need to rely on the node list_lock to synchronize the "frozen" bit changes. This implementation has some drawbacks: - Alloc path: twice cmpxchg_double. It has to get some partial slabs from node when the allocator has used up the CPU partial slabs. So it freeze the slab (one cmpxchg_double) with node list_lock held, put those frozen slabs on its CPU partial list. Later ___slab_alloc() will cmpxchg_double try-loop again if that slab is picked to use. - Alloc path: amplified contention on node list_lock. Since we have to synchronize the "frozen" bit changes under the node list_lock, the contention of slab (struct page) can be transferred to the node list_lock. On machine with many CPUs in one node, the contention of list_lock will be amplified by all CPUs' alloc path. The current code has to workaround this problem by avoiding using cmpxchg_double try-loop, which will just break and return when contention of page encountered and the first cmpxchg_double failed. But this workaround has its own problem. For more context, see 9b1ea29bc0d7 ("Revert "mm, slub: consider rest of partial list if acquire_slab() fails""). - Free path: redundant unfreeze. __slab_free() will freeze and cache some slabs on its partial list, and flush them to the node partial list when exceed, which has to unfreeze those slabs again under the node list_lock. Actually we don't need to freeze slab on CPU partial list, in which case we can save the unfreeze cmpxchg_double operations in flush path. 2. Solution =========== We solve these problems by leaving slabs unfrozen when moving out of the node partial list and on CPU partial list, so "frozen" bit is 0. These partial slabs won't be manipulate concurrently by alloc path, the only racer is free path, which may manipulate its list when !inuse. So we need to introduce another synchronization way to avoid it, we reuse PG_workingset to keep track of whether the slab is on node partial list or not, only in that case we can manipulate the slab list. The slab will be delay frozen when it's picked to actively use by the CPU, it becomes full at the same time, in which case we still need to rely on "frozen" bit to avoid manipulating its list. So the slab will be frozen only when activate use and be unfrozen only when deactivate. 3. Testing ========== We just did some simple testing on a server with 128 CPUs (2 nodes) to compare performance for now. - perf bench sched messaging -g 5 -t -l 100000 baseline RFC 7.042s 6.966s 7.022s 7.045s 7.054s 6.985s - stress-ng --rawpkt 128 --rawpkt-ops 100000000 baseline RFC 2.42s 2.15s 2.45s 2.16s 2.44s 2.17s It shows above there is about 10% improvement on stress-ng rawpkt testcase, although no much improvement on perf sched bench testcase. Thanks for any comment and code review! Chengming Zhou (9): slub: Reflow ___slab_alloc() slub: Change get_partial() interfaces to return slab slub: Keep track of whether slub is on the per-node partial list slub: Prepare __slab_free() for unfrozen partial slab out of node partial list slub: Introduce freeze_slab() slub: Delay freezing of partial slabs slub: Optimize deactivate_slab() slub: Rename all *unfreeze_partials* functions to *put_partials* slub: Update frozen slabs documentations in the source mm/slub.c | 381 ++++++++++++++++++++++++++---------------------------- 1 file changed, 180 insertions(+), 201 deletions(-) -- 2.20.1