From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4D1DC7619A for ; Wed, 5 Apr 2023 19:55:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 64EE26B0071; Wed, 5 Apr 2023 15:55:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5FE556B0074; Wed, 5 Apr 2023 15:55:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 49FB76B0075; Wed, 5 Apr 2023 15:55:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 39DC36B0071 for ; Wed, 5 Apr 2023 15:55:00 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id F1C12AC256 for ; Wed, 5 Apr 2023 19:54:59 +0000 (UTC) X-FDA: 80648390718.03.53B1221 Received: from mail-io1-f46.google.com (mail-io1-f46.google.com [209.85.166.46]) by imf24.hostedemail.com (Postfix) with ESMTP id 42353180008 for ; Wed, 5 Apr 2023 19:54:58 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Q5pH8VZu; spf=pass (imf24.hostedemail.com: domain of merimus@google.com designates 209.85.166.46 as permitted sender) smtp.mailfrom=merimus@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680724498; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZcYjART6vOTXgshTS0uLbmOyompGa7/lXpaivlpY/VA=; b=wJ/pPOI3B6U2RM3dsoAjZuTWy3RPKTT4KRDwLBHHDOw19v7JguaB2nmQaVS1FozMpxd+sB nl/lbbMKUJq/LoiDcoUoIm/N4mPfT4oZetLBFao5w+wwVYHPdnHGB6JY/+To9qIsgxPcn3 sq1KGmZkMqfci3URIQZxKiWnTNbUTqs= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=Q5pH8VZu; spf=pass (imf24.hostedemail.com: domain of merimus@google.com designates 209.85.166.46 as permitted sender) smtp.mailfrom=merimus@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680724498; a=rsa-sha256; cv=none; b=gEmu5/teLwJo24x7G+FMsft8KsGRuCkcsEK3CEX5s+EYF2eFD4lVZaXGPBRrSza5rtFHZ9 1ghW1ivhTH6yEvw2eHSf4y8/uzm7vkD4nyMhVCmSXO0xukNLEhg0tiuNqJ5gK/4pGZDMCF VzcnoAm62rPww9GkDBowytoFY0KIRp8= Received: by mail-io1-f46.google.com with SMTP id o10so8208101iou.5 for ; Wed, 05 Apr 2023 12:54:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; t=1680724497; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=ZcYjART6vOTXgshTS0uLbmOyompGa7/lXpaivlpY/VA=; b=Q5pH8VZulJ+C6K7kJgwa8mvOZ9ZrOOAVLsmxUjEBWvSwXZS2eYxnn+3CNEFDsgl/FB PjfOUoveEnD9PSu/Eywba4OUlZTADegV9S/hQ9fnBWU5k74/LBMblU+lOZZKTg++slTD 03UU0BISn96pJcY7cmEh/5YQiID+BNXZbn/LtyNdhSCUTcnPySOohUPn1SFbS9xrnAho y8FwD0YOjPOgKVlGfYrC+svNxrZfTOuh21opeEKDfTzKowlJOJ6wZADntkTgf6er0ndr WzczKJPJqJUu2H5QfN0VjxK2pU1vYQG2Hc0FVxsHg0Zw13iPCjmyqdvSgOMHi62oz+x6 iUzQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680724497; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZcYjART6vOTXgshTS0uLbmOyompGa7/lXpaivlpY/VA=; b=Rgd+00B8rXK5Zddz6BrMiFq4GgheLDBXv77lBmCLkKN8Mz226F8E5c9aVjzxqRZnmc 3W0MhVx6PHgreHlnwkjFj/okBi8GrrLPFCDjR60iClURmkNLoqTGPI79GxWtmP1x6y/6 WEdesweqw4+OazB4q1+CyQ7aKkyA0mQASaJLTm5wYWdYgtu605FDWpMKePfDYI2k94ty CkOmQ7X5bOuR/BFtmMEqZgDwPEkxzRyGnaE3GFWYBrUFBk8wL0c9gdTDekLzP1cp8YoX 7M+9GjUANYxbcHjoME16flG3dZKo9/yxgBMZPvXBMSBrIbk7ka8m2i7aksh0wr+bW8F2 tfgA== X-Gm-Message-State: AAQBX9cr7/OaJpVOevk9COwOBEEe8esHe2G27y/GRhBti4huSedfaV59 2IGrgAdqimeGBV7lDTM4MzV3wSYLl/IGR3vyYLxdBw== X-Google-Smtp-Source: AKy350aQGUhRBR9nlxvbezaz/UrkSAWRI9gjznAzUIFBqGpVkijBgUKrfga8wmafgeF9lt6YkUUxmVhABC3COYaTN2I= X-Received: by 2002:a5e:df47:0:b0:74c:7ea1:4f05 with SMTP id g7-20020a5edf47000000b0074c7ea14f05mr5543603ioq.2.1680724497235; Wed, 05 Apr 2023 12:54:57 -0700 (PDT) MIME-Version: 1.0 References: <4b9fc9c6-b48c-198f-5f80-811a44737e5f@suse.cz> <951d364a-05c0-b290-8abe-7cbfcaeb2df7@suse.cz> In-Reply-To: <951d364a-05c0-b290-8abe-7cbfcaeb2df7@suse.cz> From: Binder Makin Date: Wed, 5 Apr 2023 15:54:45 -0400 Message-ID: Subject: Re: [LSF/MM/BPF TOPIC] SLOB+SLAB allocators removal and future SLUB improvements To: Vlastimil Babka Cc: lsf-pc@lists.linux-foundation.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-block@vger.kernel.org, bpf@vger.kernel.org, linux-xfs@vger.kernel.org, David Rientjes , Christoph Lameter , Pekka Enberg , Joonsoo Kim , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Roman Gushchin Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 42353180008 X-Rspam-User: X-Stat-Signature: up9dox4jiks777amwmg6xjsrjz71pxfu X-HE-Tag: 1680724498-599257 X-HE-Meta: U2FsdGVkX1/EBPi9P13uRttrjMqYiLPyMSc5gqG2SAL+7GtbUM0mrgaX8ME8r348G07v7EKyp5603UjLRr+900b1tsJ2iV4UEhnDDG6q5HrTIq5HropXrWkS8HhzatS224nO54hUWVFX1CY24ZZBwo0omGqTsKSKCeN6NvKe70BL6oHqZX2/ylwPbyxn1j85k3nasGqyGPhzH7oEH6fWZITlba/TECF0WR3HyiefAWR01ziTTg6iumVeNNcUVCYVv+t51BhBtYn+DXy5Hy+FC54CgcJom5tYb7WLf9r8Dsk+z4jNhoC4MePBUQcRLVGWsxU1kX88SElLP2nLkTsYvYC5mn8nFo74eJtLyvDm3orBiWkI6rSiSDOA8Toi0pRFyypgyCWHg6WAoKVLUtVFYPa5l5IigYfdku/JEKZfToUrTsTycnqhLqAMhk3EuNxceWPpN1jfwWb6TM2VZj6ZXE75Po3K30e3KZw48tU1jhezMge+IzmK0CKHiA4t10YmxOosO8Gh+ATYycyvLfwUgn+j/4Xobn/GS1GhZb+de9TyRfZJqT5GtUtTQmYGsOc9o+lczTSWskump+cibnorkAfsTogW0jlWzATq8LQ6S4wkjnRPwx4iwaoSM5OQeyvrhhZj07LZO6c1ntSGXVcvvFoYy9CI5/+dFUtk6+YOJw+cVhEoohoANMKzVoqmBEvPDcfimAwgVRh3SOkb2eQIsY6T54cdziTqFeZbwBMK36pNhf1/x040JmI2JCJHOpP+NwVqx1ueLNcpNh4UdlnYYvADKTCD5wt+4ZWGO+YdAYdWnUPeANtcCV/WPyQSX1hEqJKj2LEM6WvPvDRWR3odv1Y4opbCnF/+U84tHCbfVYnGHE/mxnR2qDucnjG7TNSxGMExmfOVQ31YrhGjkPZftsSCW2hk8m3ThhNNGT4pYFT0EnIPjDZrPoHA0AdmdkmIFnI90fWnAsuqAiROZJs SPNSzv71 ZDoHqc8v8siGO/vwZiUqj+kgXu7txh9/TIQSECyN7iOq9/v4nwL9N+Y6F8rxEgBbBZlHW6t1FI7hDJq0siJiDy6ATziGDqyF28LdOqjcnP/JN+asUWEaJrauC0rq4QSQW06du6TxLR/dujYFzinDNMivQAf2yKXlXC3HiRYGUyWTIRhDekJkNvlUFS0vl06zOwk7w5YJrTDBy2M52cN39083nQPj6I0eyPZDovKHosm+zSz4/GQc9V76JPDS1nOLqTVF704claS7uP6BAY6NCgLTatEUgPRUx8Le8WsLfexq/TuvKPuEpJB+a9UDC1Z6emmCB6/xr3YQwV/i255j9j2qUHKII3LX3HijVuRMXuMw9EabaB2P/wm5uH097WHWr110/69mDEGFit3/AFq4in9cgDg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: I'm still running tests to explore some of these questions. The machines I am using are roughly as follows. Intel dual socket 56 total cores 192-384GB ram LEVEL1_ICACHE_SIZE 32768 LEVEL1_DCACHE_SIZE 32768 LEVEL2_CACHE_SIZE 1048576 LEVEL3_CACHE_SIZE 40370176 Amd dual socket 128 total cores 1TB ram LEVEL1_ICACHE_SIZE 32768 LEVEL1_DCACHE_SIZE 32768 LEVEL2_CACHE_SIZE 524288 LEVEL3_CACHE_SIZE 268435456 Arm single socket 64 total cores 256GB rma LEVEL1_ICACHE_SIZE 65536 LEVEL1_DCACHE_SIZE 65536 LEVEL2_CACHE_SIZE 1048576 LEVEL3_CACHE_SIZE 33554432 On Tue, Apr 4, 2023 at 12:03=E2=80=AFPM Vlastimil Babka wr= ote: > > On 3/22/23 13:30, Binder Makin wrote: > > Was looking at SLAB removal and started by running A/B tests of SLAB > > vs SLUB. Please note these are only preliminary results. > > Thanks, that's very useful. > > > These were run using 6.1.13 configured for SLAB/SLUB. > > Machines were standard datacenter servers. > > > > Hackbench shows completion time, so smaller is better. > > On all others larger is better. > > https://docs.google.com/spreadsheets/d/e/2PACX-1vQ47Mekl8BOp3ekCefwL6wL= 8SQiv6Qvp5avkU2ssQSh41gntjivE-aKM4PkwzkC4N_s_MxUdcsokhhz/pubhtml > > > > Some notes: > > SUnreclaim and SReclaimable shows unreclaimable and reclaimable memory. > > Substantially higher with SLUB, but I believe that is to be expected. > > > > Various results showing a 5-10% degradation with SLUB. That feels > > concerning to me, but I'm not sure what others' tolerance would be. > > > > redis results on AMD show some pretty bad degredations. 10-20% range > > netpipe on Intel also has issues.. 10-17% > > I guess one question is which ones are genuine SLAB/SLUB differences and = not > e.g. some artifact of different cache layout or something. For example it > seems suspicious if results are widely different between architectures. > > E.g. will-it-scale writeseek3_scalability regresses on arm64 and amd, but > improves on intel? Or is something wrong with the data, all columns for t= hat > whole benchmark suite are identical. > > hackbench ("smaller is better") seems drastically better on arm64 (30% > median time reduction?) and amd (80% reduction?!?), but 10% slower intel? > > redis seems a bit improved on arm64, slightly worse on intel but much wor= se > on amd. > > specjbb similar story, also I thought it was a java focused benchmark, > should it really be exercising kernel slab allocators in such notable way= ? > > I guess netpipe is the least surprising as networking was always mentione= d > in SLAB vs SLUB discussions. > > > On Tue, Mar 14, 2023 at 4:05=E2=80=AFAM Vlastimil Babka wrote: > >> > >> As you're probably aware, my plan is to get rid of SLOB and SLAB, leav= ing > >> only SLUB going forward. The removal of SLOB seems to be going well, t= here > >> were no objections to the deprecation and I've posted v1 of the remova= l > >> itself [1] so it could be in -next soon. > >> > >> The immediate benefit of that is that we can allow kfree() (and kfree_= rcu()) > >> to free objects from kmem_cache_alloc() - something that IIRC at least= xfs > >> people wanted in the past, and SLOB was incompatible with that. > >> > >> For SLAB removal I haven't yet heard any objections (but also didn't > >> deprecate it yet) but if there are any users due to particular workloa= ds > >> doing better with SLAB than SLUB, we can discuss why those would regre= ss and > >> what can be done about that in SLUB. > >> > >> Once we have just one slab allocator in the kernel, we can take a clos= er > >> look at what the users are missing from it that forces them to create = own > >> allocators (e.g. BPF), and could be considered to be added as a generi= c > >> implementation to SLUB. > >> > >> Thanks, > >> Vlastimil > >> > >> [1] https://lore.kernel.org/all/20230310103210.22372-1-vbabka@suse.cz/ > >> >