From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5EB8C4332F for ; Mon, 14 Nov 2022 11:35:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 330EF8E0001; Mon, 14 Nov 2022 06:35:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2BA736B0073; Mon, 14 Nov 2022 06:35:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 10CEA8E0001; Mon, 14 Nov 2022 06:35:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id F09A26B0072 for ; Mon, 14 Nov 2022 06:35:45 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id BBB3DAB2C8 for ; Mon, 14 Nov 2022 11:35:45 +0000 (UTC) X-FDA: 80131843050.02.751C191 Received: from esa5.hgst.iphmx.com (esa5.hgst.iphmx.com [216.71.153.144]) by imf23.hostedemail.com (Postfix) with ESMTP id A8AA5140005 for ; Mon, 14 Nov 2022 11:35:44 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=wdc.com; i=@wdc.com; q=dns/txt; s=dkim.wdc.com; t=1668425744; x=1699961744; h=message-id:date:mime-version:subject:to:cc:references: from:in-reply-to:content-transfer-encoding; bh=WHeLY9a6UKMTiG2de6Vs9bsuLF7tQZ2YgeC6xZyhGac=; b=H6siUf0hH/fNzezla/961ktIWp0MyZfpQlPiya2fUr2tPa/KnZZ1SzhA OzvLHar+8JaFoUt7ZbxBmOU7VFbxFMM0Ie89hR2iT88GYs0byLs55d1Sz caEuPaNLO4joa8M9YE4erqjIpaaiDk5E/efOuHqQiSqwVIXbFsxcCWT4L SPzcFe1XxUlBUki81B8fPcxrAAVKcUSQXkwOubWwg82RUxhpCRcjb2Hk7 yOHhK4yaIaCthY+oZKbpogOuNVP/mZLu+25e4/9lnYyOYn6Q3GlotiAR7 yVqm2jZoiZMLf/bhCv4ybMSWxPXAQ/1zrH5CN8GMLL7FteDGmmuiVdTek w==; X-IronPort-AV: E=Sophos;i="5.96,161,1665417600"; d="scan'208";a="216217976" Received: from uls-op-cesaip02.wdc.com (HELO uls-op-cesaep02.wdc.com) ([199.255.45.15]) by ob1.hgst.iphmx.com with ESMTP; 14 Nov 2022 19:35:43 +0800 IronPort-SDR: 0gpIISoIetAfJcHI3t92fEO4Ufwj9A/5pB9KYlDN+MB6N5FQHjmSwhcmuAiw3wewLLBwB0sSvU ot3nWH65ng0Vex7XSlPEN+/PI5LxIM4Kl1b/UnFVviuyTlCcq5HqE6rzIjBd+D8j333xHzHXJm m4uMOKjuJehoqQoZjRT04ifGatgkcMMg0l0q54+KDJzat6A+UZi5lz33LI8n0M7MbAqyxzKVLw eyc7ype5NX3KO57fAh11JR+UUNTWbX5xUx1PUyzRhYFlax8XjAntQizvbn+atESbZ57fXg24ZL +Vg= Received: from uls-op-cesaip01.wdc.com ([10.248.3.36]) by uls-op-cesaep02.wdc.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 14 Nov 2022 02:48:56 -0800 IronPort-SDR: PnEDS5TqbiurpRor7tw2kPRK+0V4J6UCK/7PX9VzRS2tNHcA5AbX8G0AYGpl//67C+1tbQdO6M 8iiRyy52nUP6Fi2MgtpUHOaW+jmnK1F264vYhO94UyjzAAMnv+I7f/kz+B0AO6xJ3ReYeAJzMM nTteBDtkmX/3ECur1pMQbjVFLF7vCBpLQYNwi4O69LxciL7C6pYTrdWRWnMN6873BlP4aY6y4z uTP1qsFS1qve2Z2BhkROIVXf85oiOIxUa4l3GRbItfQmOrD0SFVpq59XH5Z1SWejs5gWG0MhEa ohs= WDCIronportException: Internal Received: from usg-ed-osssrv.wdc.com ([10.3.10.180]) by uls-op-cesaip01.wdc.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 14 Nov 2022 03:35:43 -0800 Received: from usg-ed-osssrv.wdc.com (usg-ed-osssrv.wdc.com [127.0.0.1]) by usg-ed-osssrv.wdc.com (Postfix) with ESMTP id 4N9nLQ48GCz1RwqL for ; Mon, 14 Nov 2022 03:35:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d= opensource.wdc.com; h=content-transfer-encoding:content-type :in-reply-to:organization:from:references:to:content-language :subject:user-agent:mime-version:date:message-id; s=dkim; t= 1668425739; x=1671017740; bh=WHeLY9a6UKMTiG2de6Vs9bsuLF7tQZ2YgeC 6xZyhGac=; b=CUuGIPlwf3Bn8/s9EFG5W98t0wfQJthG48+ahxRGQogE6ytTX7G H+g5CYe+ZVHCgL6VjsYt/0d0iUEkeqCWLh4+6FeB/akADAFVY12TlCVYhVGpI+Pi x9cvnAXhvcN5MEd3To/kddt7GRpLgG/anpHt/sI8OgGdQj5RYqCDpLFVVLZ0GB2b 3iDMkXsEtyz2foVR37V1+K7uaM8dgNoiO8osoGLrGKCC2E1ZM2tVLuVJwokC6IuH 8U4zmnmlZ3kcdtp6s9LT0xKpS2gfTE+XImzeKORyTOIFRXGVg9XTxaQw+LiA2iBW o/Vm3gfA0BoViNa0vXSqfp1BgFkTmhCdtGg== X-Virus-Scanned: amavisd-new at usg-ed-osssrv.wdc.com Received: from usg-ed-osssrv.wdc.com ([127.0.0.1]) by usg-ed-osssrv.wdc.com (usg-ed-osssrv.wdc.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id w5k-IrSFJ5EA for ; Mon, 14 Nov 2022 03:35:39 -0800 (PST) Received: from [10.225.163.46] (unknown [10.225.163.46]) by usg-ed-osssrv.wdc.com (Postfix) with ESMTPSA id 4N9nLD6GLmz1RvLy; Mon, 14 Nov 2022 03:35:32 -0800 (PST) Message-ID: <35650fd4-3152-56db-7c27-b9997e31cfc7@opensource.wdc.com> Date: Mon, 14 Nov 2022 20:35:31 +0900 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.4.1 Subject: Re: Deprecating and removing SLOB Content-Language: en-US To: Vlastimil Babka , Conor Dooley Cc: Pasha Tatashin , Christoph Lameter , David Rientjes , Joonsoo Kim , Pekka Enberg , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Matthew Wilcox , Roman Gushchin , Linus Torvalds , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" , Catalin Marinas , Rustam Kovhaev , Andrew Morton , Josh Triplett , Arnd Bergmann , Russell King , Alexander Shiyan , Aaro Koskinen , Janusz Krzysztofik , Tony Lindgren , Yoshinori Sato , Rich Felker , Jonas Bonn , Stefan Kristiansson , Stafford Horne , "linux-arm-kernel@lists.infradead.org" , openrisc@lists.librecores.org, linux-riscv@lists.infradead.org, linux-sh@vger.kernel.org, Geert Uytterhoeven , Conor.Dooley@microchip.com, Paul Cercueil References: <93079aba-362e-5d1e-e9b4-dfe3a84da750@opensource.wdc.com> <44da078c-b630-a249-bf50-67df83cd8347@suse.cz> From: Damien Le Moal Organization: Western Digital Research In-Reply-To: <44da078c-b630-a249-bf50-67df83cd8347@suse.cz> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1668425745; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yKC2C8WU9krLS12NaTRhtlgi/aZZ5mGkFhXuZn5lVi4=; b=mNGuhBzV8AzAqRqsyYbkd+FzSrdaSP5YMvmD6BCiKyYkwYU9VfKLtJIhXm7Jyn9mIYueNl XQDuInU9ZZ15WzyKrsI2RTKAiC+jZB6a7Du5oYT9i4rW54jTP+OmALtGnDOhnRbPMaE5nr 8QbrfmRigKfs/2ygqCzieoIh03ZE9Sc= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=none ("invalid DKIM record") header.d=wdc.com header.s=dkim.wdc.com header.b=H6siUf0h; dkim=pass header.d=opensource.wdc.com header.s=dkim header.b=CUuGIPlw; spf=pass (imf23.hostedemail.com: domain of "prvs=310906da8=damien.lemoal@opensource.wdc.com" designates 216.71.153.144 as permitted sender) smtp.mailfrom="prvs=310906da8=damien.lemoal@opensource.wdc.com"; dmarc=pass (policy=quarantine) header.from=opensource.wdc.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1668425745; a=rsa-sha256; cv=none; b=f7tdUNofWCQQPqJypV4egtNq8Eqc+nX//XStIIXvwZXxznpE1vp11jIgh2fh+Aa3mpMykU yctSZj3w2Q+685Kj2HwvqF9Z46Ditw7Bti2UWupz6G90kks0J9LYgwrpcVqeNHUdvacspR twMwfXHUOuqgpgUJjcZDHwuEF2FwiNo= X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: A8AA5140005 Authentication-Results: imf23.hostedemail.com; dkim=none ("invalid DKIM record") header.d=wdc.com header.s=dkim.wdc.com header.b=H6siUf0h; dkim=pass header.d=opensource.wdc.com header.s=dkim header.b=CUuGIPlw; spf=pass (imf23.hostedemail.com: domain of "prvs=310906da8=damien.lemoal@opensource.wdc.com" designates 216.71.153.144 as permitted sender) smtp.mailfrom="prvs=310906da8=damien.lemoal@opensource.wdc.com"; dmarc=pass (policy=quarantine) header.from=opensource.wdc.com X-Stat-Signature: 5jfzoh3ofqnyk3rrfiq1g17urthwobgb X-HE-Tag: 1668425744-697510 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 11/14/22 18:36, Vlastimil Babka wrote: > On 11/14/22 06:48, Damien Le Moal wrote: >> On 11/14/22 10:55, Damien Le Moal wrote: >>> On 11/12/22 05:46, Conor Dooley wrote: >>>> On Fri, Nov 11, 2022 at 11:33:30AM +0100, Vlastimil Babka wrote: >>>>> On 11/8/22 22:44, Pasha Tatashin wrote: >>>>>> On Tue, Nov 8, 2022 at 10:55 AM Vlastimil Babka wrote: >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> as we all know, we currently have three slab allocators. As we discussed >>>>>>> at LPC [1], it is my hope that one of these allocators has a future, and >>>>>>> two of them do not. >>>>>>> >>>>>>> The unsurprising reasons include code maintenance burden, other features >>>>>>> compatible with only a subset of allocators (or more effort spent on the >>>>>>> features), blocking API improvements (more on that below), and my >>>>>>> inability to pronounce SLAB and SLUB in a properly distinguishable way, >>>>>>> without resorting to spelling out the letters. >>>>>>> >>>>>>> I think (but may be proven wrong) that SLOB is the easier target of the >>>>>>> two to be removed, so I'd like to focus on it first. >>>>>>> >>>>>>> I believe SLOB can be removed because: >>>>>>> >>>>>>> - AFAIK nobody really uses it? It strives for minimal memory footprint >>>>>>> by putting all objects together, which has its CPU performance costs >>>>>>> (locking, lack of percpu caching, searching for free space...). I'm not >>>>>>> aware of any "tiny linux" deployment that opts for this. For example, >>>>>>> OpenWRT seems to use SLUB and the devices these days have e.g. 128MB >>>>>>> RAM, not up to 16 MB anymore. I've heard anecdotes that the performance >>>>>>> SLOB impact is too much for those who tried. Googling for >>>>>>> "CONFIG_SLOB=y" yielded nothing useful. >>>>>> >>>>>> I am all for removing SLOB. >>>>>> >>>>>> There are some devices with configs where SLOB is enabled by default. >>>>>> Perhaps, the owners/maintainers of those devices/configs should be >>>>>> included into this thread: >>>>>> >>>>>> tatashin@soleen:~/x/linux$ git grep SLOB=y >>>> >>>>>> arch/riscv/configs/nommu_k210_defconfig:CONFIG_SLOB=y >>>>>> arch/riscv/configs/nommu_k210_sdcard_defconfig:CONFIG_SLOB=y >>>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_SLOB=y >>>> >>>>> >>>>> Turns out that since SLOB depends on EXPERT, many of those lack it so >>>>> running make defconfig ends up with SLUB anyway, unless I miss something. >>>>> Only a subset has both SLOB and EXPERT: >>>>> >>>>>> git grep CONFIG_EXPERT `git grep -l "CONFIG_SLOB=y"` >>>> >>>>> arch/riscv/configs/nommu_virt_defconfig:CONFIG_EXPERT=y >>>> >>>> I suppose there's not really a concern with the virt defconfig, but I >>>> did check the output of `make nommu_k210_defconfig" and despite not >>>> having expert it seems to end up CONFIG_SLOB=y in the generated .config. >>>> >>>> I do have a board with a k210 so I checked with s/SLOB/SLUB and it still >>>> boots etc, but I have no workloads or w/e to run on it. >>> >>> I sent a patch to change the k210 defconfig to using SLUB. However... > > Thanks! > >>> The current default config using SLOB gives about 630 free memory pages >>> after boot (cat /proc/vmstat). Switching to SLUB, this is down to about >>> 400 free memory pages (CONFIG_SLUB_CPU_PARTIAL is off). > > Thanks for the testing! How much RAM does the system have btw? I found 8MB > somewhere, is that correct? Yep, 8MB, that's it. > So 230 pages that's a ~920 kB difference. Last time we saw less dramatic > difference [1]. But that was looking at Slab pages, not free pages. The > extra overhead could be also in percpu allocations, code etc. > >>> This is with a buildroot kernel 5.19 build including a shell and sd-card >>> boot. With SLUB, I get clean boots and a shell prompt as expected. But I >>> definitely see more errors with shell commands failing due to allocation >>> failures for the shell process fork. So as far as the K210 is concerned, >>> switching to SLUB is not ideal. >>> >>> I would not want to hold on kernel mm improvements because of this toy >>> k210 though, so I am not going to prevent SLOB deprecation. I just wish >>> SLUB itself used less memory :) >> >> Did further tests with kernel 6.0.1: >> * SLOB: 630 free pages after boot, shell working (occasional shell fork >> failure happen though) >> * SLAB: getting memory allocation for order 7 failures on boot already >> (init process). Shell barely working (high frequency of shell command fork >> failures) I forgot to add here that the system was down to about 500 free pages after boot (again from the shell with "cat /proc/vmstat"). >> * SLUB: getting memory allocation for order 7 failures on boot. I do get a >> shell prompt but cannot run any shell command that involves forking a new >> process. For both slab and slub, I had cpu partial off, debug off and slab merge on, as I suspected that would lead to less memory overhead. I suspected memory fragmentation may be an issue but doing echo 3 > /proc/sys/vm/drop_caches before trying a shell command did not help much at all (it usually does on that board with SLOB). Note that this is all with buildroot, so this echo & redirect always works as it does not cause a shell fork. >> >> So if we want to keep the k210 support functional with a shell, we need >> slob. If we reduce that board support to only one application started as >> the init process, then I guess anything is OK. > > In [1] it was possible to save some more memory with more tuning. Some of > that required boot parameters and other code changes. In another reply [2] I > considered adding something like SLUB_TINY to take care of all that, so > looks like it would make sense to proceed with that. If you want me to test something, let me know. > > [1] > https://lore.kernel.org/all/Yg9xSWEaTZLA+hYt@ip-172-31-19-208.ap-northeast-1.compute.internal/ > [2] https://lore.kernel.org/all/eebc9dc8-6a45-c099-61da-230d06cb3157@suse.cz/ -- Damien Le Moal Western Digital Research