From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F21B8C282EC for ; Wed, 5 Mar 2025 18:44:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1002628000B; Wed, 5 Mar 2025 13:44:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0B10B280022; Wed, 5 Mar 2025 13:44:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E6AA228000B; Wed, 5 Mar 2025 13:44:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C21FD280022 for ; Wed, 5 Mar 2025 13:44:10 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 3198AB15CA for ; Wed, 5 Mar 2025 10:26:25 +0000 (UTC) X-FDA: 83187117930.16.B866450 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf26.hostedemail.com (Postfix) with ESMTP id ACCAC140004 for ; Wed, 5 Mar 2025 10:26:22 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=JZne7VvX; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=I9qsyjbZ; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=JZne7VvX; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=I9qsyjbZ; dmarc=none; spf=pass (imf26.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1741170383; a=rsa-sha256; cv=none; b=zhRafYD4jMNiFLUfKCoL6Jt7RIs/7Lj5dfKP42K2xDavDnqKrs4IT4AZpT8OI3fn7HL/dT f1dUpQIaSNs+YFY9GnviUWMpFnCbbkUKFQLBKedR0yugAlyA5bp4SFKFjOibKE2yFpWI/z YS0mjJwGLaCwFzmJ5wIwNcBnBLIxFtQ= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=JZne7VvX; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=I9qsyjbZ; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=JZne7VvX; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=I9qsyjbZ; dmarc=none; spf=pass (imf26.hostedemail.com: domain of vbabka@suse.cz designates 195.135.223.131 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1741170383; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fx3Hs2ab/mDkBUFq/ZI1BouZLIEyP0T4le/acD61ke0=; b=7yndDZTaxR7fKpOclb1OANxiHPIZwjfOzOXjBSpkKWYfPwHrkDb6rx4Ajn7fkrBqHsKomy HjFc2DZ0zKVAIfGmhDz5N0SaaSnQ7Qga9TTHA8pWysmW5dhz/gqf6w4Q035pCe12Y9Dkb5 mGcItLc0JgG2jSRqP+ZUJcU6gMTvy9Y= Received: from imap1.dmz-prg2.suse.org (unknown [10.150.64.97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 036E31F393; Wed, 5 Mar 2025 10:26:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1741170381; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fx3Hs2ab/mDkBUFq/ZI1BouZLIEyP0T4le/acD61ke0=; b=JZne7VvXRZNHpUSWy/pZXn9rwx+1MlyhL4ZGRDWbW4KgEvTb1ITTMNQbjg3ju50ivyr4s8 r66gPbEDx0ysstFIjChbtYlKjAJj9dGsQO7xBxao6o1ieGBdyjEk8nElbbpeMY50GP84bF H6HS8hc6glCDqLzC4lSbhCNYJ8hQ1C8= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1741170381; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fx3Hs2ab/mDkBUFq/ZI1BouZLIEyP0T4le/acD61ke0=; b=I9qsyjbZDhnAF3q1+Wl5HGuomEfeILiImg6J33lbsTlcgV6xs6Ea/yoXmrcHyKu1fbeoLh /NOXuZfiESm9LsDA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1741170381; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fx3Hs2ab/mDkBUFq/ZI1BouZLIEyP0T4le/acD61ke0=; b=JZne7VvXRZNHpUSWy/pZXn9rwx+1MlyhL4ZGRDWbW4KgEvTb1ITTMNQbjg3ju50ivyr4s8 r66gPbEDx0ysstFIjChbtYlKjAJj9dGsQO7xBxao6o1ieGBdyjEk8nElbbpeMY50GP84bF H6HS8hc6glCDqLzC4lSbhCNYJ8hQ1C8= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1741170381; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=fx3Hs2ab/mDkBUFq/ZI1BouZLIEyP0T4le/acD61ke0=; b=I9qsyjbZDhnAF3q1+Wl5HGuomEfeILiImg6J33lbsTlcgV6xs6Ea/yoXmrcHyKu1fbeoLh /NOXuZfiESm9LsDA== Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id DFDE11366F; Wed, 5 Mar 2025 10:26:20 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id FlMkNswmyGcnfwAAD6G6ig (envelope-from ); Wed, 05 Mar 2025 10:26:20 +0000 Message-ID: Date: Wed, 5 Mar 2025 11:26:20 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [LSF/MM/BPF TOPIC] SLUB allocator, mainly the sheaves caching layer Content-Language: en-US To: "Christoph Lameter (Ampere)" Cc: lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, bpf , David Rientjes , Hyeonggon Yoo <42.hyeyoo@gmail.com>, "Uladzislau Rezki (Sony)" , Alexei Starovoitov , Kees Cook References: <14422cf1-4a63-4115-87cb-92685e7dd91b@suse.cz> From: Vlastimil Babka In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: ACCAC140004 X-Stat-Signature: i8zzikrpd31wca4u3z4d8ung4d4pusui X-Rspam-User: X-HE-Tag: 1741170382-572331 X-HE-Meta: U2FsdGVkX19JwKq/86+yP/Q3rqq1zI84/EP99aZhcevCLEVDNhJS5pVf5cEgU6bFRcNfqobV92tNwghtujhu28wdpj8/BmlL0oDabNgX8R+W+kekxw1rOLrmKbb3TX4Fegs6+GtkU9o3LP7xkEkBzxYatTq0vmF2CB+xJ8OljbT9+68yWQte3b1tBqlgDP2WUxvr8QNMjLbw3CLyYijXFf/pfQpWipucO7JI73UkY4MaF5REwjPeNReW6v9l3bG8czekGu+fxcdr+L5mMRE1piZ2K97dK2xKazcb1zGtkz5jK+ulpTIqUa2gB60L4JRk5vg6xpED3Rj8FF54DJ+XMvPiIAZ/Jn/o9ngR+NPA65s0jumZHbSeRLJgVmTxo/SEo5jDIF7gzzuaUJHuiDtYSRzsiocxxoFV1kMZS/2d1l92khMPDgVodHYD8s7U+u/71KZf088SC+cEanC8c1B7zmyTpIrRtQoDdIrBEkJx4IlcjjuC91K5L1Qc/HIOOCpmjD1K1i9LEXZd/34bDul+WvWkFAZsTCIe8pHp5+ex3Y98C2w4klCGpbhzXCJl98UBqVq+lIQeOsW1cuh521EkiyJnwVPIczCDmGnHIwvrJcQjIJrDwdI2oxnvtSidtNQzKycitigUfQ0EHTdjCPHLU7Lfg2K97OAz030/jwTyUNTUw3REmsWjY3DOrjSSE1bqqEVdSahYgZZdgSb6K7tzZLheY7ctbMQuKpgPocM9aDGrONSEmfzJrYe4p8PW4kj4U4qYIoI6ZnkM90+F4oUrUfnG4f2MTsDqFkJFu4GwAEk2xtD0uDbcEXsoR5oorMzJeEASoN4B0XgCXlP8SsWfIzr6v+L6izaD/ySy2P8FhG7Xgx+Igvn59rduCFmjhzQvJKt2ZS8zuTVMNLBoJP75IB3Fr+IL9/sFUURCB7biWTgQLIjiulMQcajnLrGgNThZM3N0UZRMJYIQfzIOcAB raqoJn11 Xzy+YZ3gau3yallfXyZNcYXlhCjTVJOVITWBHywtAZIKzQqT45rmqS86izxhdviYCdSXfUAQSxATlTY9lSE38lZYA3bgPKuxY5EekcgD0Gl2UY15jDXhmf5oG7+zP2jHQb+glOD+DNTcPoNCmnl7RsvTvt53cu+JFD87z6ZlAG4tNDWtNmShPNt36WaOulUV7y0pp9ngMJfE49frG1Sp+fMstOJGaoYcnmXqL+tpAx20+Fc1Lp085tvw8ARQqrmjRSD3K6N+MGg/iDNsMeDJHSAB0LkcO3vjxblV9/uvmxWgNnz5T98NjUNh7o0vGkIsLG4I3goPxT080oDxezvaWZfXdbd3wTbUo4IHkrKSLLCJU294= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 2/26/25 01:17, Christoph Lameter (Ampere) wrote: > > Let me just express my general concern. SLUB was written because SLAB > became a Byzantine mess with layer upon layer of debugging and queues I don't recall it having much debugging. IIRC it was behind some config that nobody enabled. SLUB's debugging that can be dynamically enabled on boot is so much better. > here and there and with "maintenance" for these queues going on every 2 > seconds staggered on all processors. This caused a degree of OS noise that > caused HPC jobs (and today we see similar issues with AI jobs) to not be > able to accomplish a deterministic rendezvous. On some large machines Yeah, I don't want to reintroduce this, hence sheaves intentionally don't support NUMA restricted allocations so none of the flushed alien arrays are necessary. > we had ~10% of the whole memory vanish into one of the other queue on boot > up with the customers being a bit upset were all the expensive memory > went. > > It seems that were have nearly recreated the old nightmare again. I don't see it that bleak. > I would suggest rewriting the whole allocator once again trying to > simplify things as much as possible and isolating specialized allocator > functionality needed for some subsystems into different APIs. Any specific suggestions? Some things are hard to isolate i.e. make them work on top of the core allocator because not interacting with the internals would not allow some useful functionality, or efficiency. > The main allocation / free path needs to be as simple and as efficient as > possible. It may not be possible to accomplish something like that given > all the special casing that we have been pushing into it. Also consider the I see some possibilities for simplification in not trying to support KASAN together with slab_debug anymore. KASAN should be superior for that purpose (of course you pay the extra cost) and it's tricky to not have it step on each other's toes with slab_debug. > runtime security measures and verification stuff that is on by default at > runtime as well. Yeah more and more hardening seems to be the current trend. But also not realistically possible to isolate away from the core. I at least tried to always compile all of it away completely when the respective CONFIG is not enabled. OTOH I'd like to see some of that to support boot parameters (via static keys etc) so it can be compiled in but not enabled. That would not completely eliminate the overhead of passing e.g. the bucket parameter or performing kmalloc random index evaluation, but would not allocate the separate caches if not enabled, so the memory overhead of that would not be imposed.