From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0FA25C77B61 for ; Thu, 27 Apr 2023 08:29:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 29B926B0071; Thu, 27 Apr 2023 04:29:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 24AD66B0072; Thu, 27 Apr 2023 04:29:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1128F900002; Thu, 27 Apr 2023 04:29:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 05E1D6B0071 for ; Thu, 27 Apr 2023 04:29:49 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B3779A0241 for ; Thu, 27 Apr 2023 08:29:48 +0000 (UTC) X-FDA: 80726497656.21.501A971 Received: from smtp-out1.suse.de (smtp-out1.suse.de [195.135.220.28]) by imf19.hostedemail.com (Postfix) with ESMTP id 80ECD1A000E for ; Thu, 27 Apr 2023 08:29:45 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=2VZJLOte; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=7rg0I0z9; spf=pass (imf19.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682584186; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=I5HZTSFrzWtBYnuBYintcvIIAgF3iktWG4WhcixoNBs=; b=7ZcSTPZrMeicmcPIWYmmMDxnmMZ2ma5mWuVWwU4000OL3B9uvdaMRqhrSE9k4A0CBmj3+V If/PyI0L2mK9uVWpcymrNKA2lB2Zl2VFfp+gOS22LGSWpPk3Hii8rrciZVt7yCQl9a7eZ7 4foUnOD71E706m8RGbPt8clSFlpmyY8= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=2VZJLOte; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=7rg0I0z9; spf=pass (imf19.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.28 as permitted sender) smtp.mailfrom=vbabka@suse.cz; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682584186; a=rsa-sha256; cv=none; b=5ES2Gavtfe2jOLI0mphX4EcsXQ51ayelHC+Y1NN97ExrUsN6hzUf9cfep197W2+CUVwSPN bB37n1GPBkwpUwEavN0kMRqTMHF+R1BsKcN67BclecFPIUIWlM2E7vx71IdKAjqjPPEc71 2w3XXFYlZ5JXDDYcoUjwpX0nugi4kGs= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out1.suse.de (Postfix) with ESMTPS id BA7D821A40; Thu, 27 Apr 2023 08:29:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1682584183; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=I5HZTSFrzWtBYnuBYintcvIIAgF3iktWG4WhcixoNBs=; b=2VZJLOteCXB5x7qGpePCN15IyixuXEfP/YCclnOkg2okWD9z85poLundXOr+hfQ75Bx8z6 e2RBBk+rjEmWJL3iKpcO3OVWaPOMZIzK23QTlULfge4tUgwS0Wxgi8SVUr7Z+VgxyVYGHt xyKI6uyMt5Tck82tWJbKuExwaEvmJj0= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1682584183; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=I5HZTSFrzWtBYnuBYintcvIIAgF3iktWG4WhcixoNBs=; b=7rg0I0z9R1Q5zouAGGyLG9AkNA9quvvz57/VLVxpo722dfm6Og7kn47VZCK2qUvOAa/8xK xqxRVzSPb5K5v5AQ== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 8A0AC138F9; Thu, 27 Apr 2023 08:29:43 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id fKAIIXcySmQJVAAAMHmgww (envelope-from ); Thu, 27 Apr 2023 08:29:43 +0000 Message-ID: <19acbdbb-fc2f-e198-3d31-850ef53f544e@suse.cz> Date: Thu, 27 Apr 2023 10:29:43 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.10.0 Subject: Re: [LSF/MM/BPF TOPIC] SLOB+SLAB allocators removal and future SLUB improvements To: Binder Makin Cc: lsf-pc@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-block@vger.kernel.org, bpf@vger.kernel.org, linux-xfs@vger.kernel.org, David Rientjes , Christoph Lameter , Pekka Enberg , Joonsoo Kim , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Roman Gushchin References: <4b9fc9c6-b48c-198f-5f80-811a44737e5f@suse.cz> <951d364a-05c0-b290-8abe-7cbfcaeb2df7@suse.cz> Content-Language: en-US From: Vlastimil Babka In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 80ECD1A000E X-Stat-Signature: 8qg59pr3545xjii5kxu6r8aph41ec5n7 X-HE-Tag: 1682584185-162546 X-HE-Meta: U2FsdGVkX1/Ie7YZqJqv7JzM1eG/9vRRQo5WJrL5A+cp/MyEG07JUhPn8a2gjcOPQIYdJItSBSjGQH/XqAHZMgeCYTXymCDgCmo3EGYOLxePWFGgULHZfcjAn26sfyuMWseepifohyuPCxdNgOvK1BLi9CdKyI4kV3sWhKEIoDr4FFFoWVO4NnA3YP8z4432NaDoFW+TXIZZ7q/ce6vL7mTQzb699UhC5XStcKaXKfk9xxxqhpNLrIqEUfilGbfGyuRVTo9DBwe8kkxYhVle4bgO1lVqQRUtiLpcvnzBYbtyxuEpFsU7cu8g4OaqihwllutiRNqg8LOfHQ0qcNqXnaJ6xU3FH9maYW4m16u66sxLf9eHAniHqyb9PfD+cB4HwHBBHo2McL0Eg5iO6Hff4xlyND2yg72mbz6hg2v/lM4R3zRtX+hAH7P7WJtWrQw5rMpewsZLda89aBNwQgoj2EhKz0mvXmc+X8z4KMkVNGd0kRWTyrbEcM2tY5jcGr/UK8imz+ZbIQX6IieZ3YQnYaxE5IKBfoZFV6A0+YDeInfeIiqQwykC4aKbCJvvT0A5VuuXYV/sT4B6mlp1+sXQ4eoiZlI13b1+17IivOFASPbqmI4tHG7Kv+tImSpYNty4kAgE2YxPOPlMSHzxjRu4hSTmNs7ErtMHM77oBXhF7aEuLYch4IB4EBcbmybnVaVKaKhjFxlSv6+fCCIVKX6hXFPEdXs4prIxTTV8Hgb4BCrneLKrNJ08jYWOS/6CYbKbPNFMdl5T8K5Jgf2LzbTEF4o/sjNOV/w7bXAen2yTwB0FonxLZNcARiqCf/FC9y4hUR5f2T20g/hXz873BgkOsNQ/x2/kzUgK8oKpG+nHY1ZH2DOWt4iIksMe6zVnAB/X3kNBiqVSkHtwb8GIs8kYl0hhF8DdgLg+LnDFBDUBy9WeS5IF3cv+aCQSfJw7y+elOyT0NP87xUTpjnFmYIA HZJ9VdFy e65H6ksJ++SKydYQp8DspDdzyEC48yLeK7avADOGgFdV+P8Xy2gGL890kAblMXNkIUCYCdXEn0/m9UzhXnmUS4KRm+qwLRBxWtMEprT8t9FvLQfOlqjDffxKtFQwcFxHCOInhb3W1KhmGk8ki/HrySr7cfgKyT2qiGpzPhEFFBZdXkC/cX2vPjkPBOJ0Gn6UCOo/AmJ1HpTLWm7/XIOmtfhx2glltHJgJyMkHGP82Xaag81pt8YS4C4lFcc2jNGmzGlLsISvLU8TyNwVVMaMATIFOhRf/cJjOUvS1umzApjbvPhAFeWFSOCzIhPV4mNt4PEpiMhio4HtpIOE/OhrcvX8izrGpV77qJYe0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 4/5/23 21:54, Binder Makin wrote: > I'm still running tests to explore some of these questions. > The machines I am using are roughly as follows. > > Intel dual socket 56 total cores > 192-384GB ram > LEVEL1_ICACHE_SIZE 32768 > LEVEL1_DCACHE_SIZE 32768 > LEVEL2_CACHE_SIZE 1048576 > LEVEL3_CACHE_SIZE 40370176 > > Amd dual socket 128 total cores > 1TB ram > LEVEL1_ICACHE_SIZE 32768 > LEVEL1_DCACHE_SIZE 32768 > LEVEL2_CACHE_SIZE 524288 > LEVEL3_CACHE_SIZE 268435456 > > Arm single socket 64 total cores > 256GB rma > LEVEL1_ICACHE_SIZE 65536 > LEVEL1_DCACHE_SIZE 65536 > LEVEL2_CACHE_SIZE 1048576 > LEVEL3_CACHE_SIZE 33554432 So with "some artifact of different cache layout" I didn't mean the different cache sizes of the processors, but possible differences how objects end up placed in memory by SLAB vs SLUB causing them to collide in the cache of cause false sharing less or more. This kind of interference can make interpreting (micro)benchmark results hard. Anyway, how I'd hope to approach this topic would be that SLAB removal is proposed, and anyone who opposes that because they can't switch from SLAB to SLUB would describe why they can't. I'd hope the "why" to be based on testing with actual workloads, not just benchmarks. Benchmarks are then of course useful if they can indeed distill the reason why the actual workload regresses, as then anyone can reproduce that locally and develop/test fixes etc. My hope is that if some kind of regression is found (e.g. due to lack of percpu array in SLUB), it can be dealt with by improving SLUB. Historically I recall that we (SUSE) objected somwhat to SLAB removal as our distro kernels were using it, but we have switched since. Then networking had concerns (possibly related to the lack percpu array) but seems bulk allocations helped and they use SLUB these days [1]. And IIRC Google was also sticking to SLAB, which led to some attempts to augment SLUB for those workloads years ago, but those were never finished. So I'd be curious if we should restart those effors or can just remove SLAB now. [1] https://lore.kernel.org/all/93665604-5420-be5d-2104-17850288b955@redhat.com/