From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 513C0C83F22 for ; Fri, 18 Jul 2025 02:16:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C7C776B00A5; Thu, 17 Jul 2025 22:16:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C2C4C6B00A6; Thu, 17 Jul 2025 22:16:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B1BF06B00A7; Thu, 17 Jul 2025 22:16:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A11546B00A5 for ; Thu, 17 Jul 2025 22:16:52 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id F291B1602F4 for ; Fri, 18 Jul 2025 02:16:51 +0000 (UTC) X-FDA: 83675772222.29.C654EAD Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf05.hostedemail.com (Postfix) with ESMTP id 37090100002 for ; Fri, 18 Jul 2025 02:16:50 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=VT6oRthh; spf=pass (imf05.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1752805010; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=bjPUtTo5Q4RHjWDwA/8ArVUYFnewjb9EWWS17pFPGq0=; b=V/MxSayjnN4PMxsK4Ts/J7/gOMDTvKwH1wpjODimMmOo+wOwu4G2ezMs8UXcb5keXY/njU aTyVrHtBPpf+pLqwLc7XEytwMA0IlUNsY2ec7ywGH33z8/w2+9ZlFGLP+4hujtOUmgUnbr FbcHHNgouXTAZiIs5f5UldgGfOViZQU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1752805010; a=rsa-sha256; cv=none; b=gFnKDAPMQpa81/6tw62M97iUNzmrwj9eJBiXuJpiF9nwz72qpl7T/M4vglHBi6nFdM2oKR hsixcM6cDet/h1D3zi+iIZuXgQYrn1ZRZ3KDMJZbPfzens0pmzJue5WwQeTq4R1tNWnz3d udNuxoGfEMvTjs9WfdPSDqaM/L1F9kQ= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=VT6oRthh; spf=pass (imf05.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-23dc5bcf49eso19557555ad.2 for ; Thu, 17 Jul 2025 19:16:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1752805009; x=1753409809; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=bjPUtTo5Q4RHjWDwA/8ArVUYFnewjb9EWWS17pFPGq0=; b=VT6oRthh7XGVXqk+JNL6oZ6nngwN813yQEsdBRp1AUz6+p2zoumT21xyCVPaA+Fnf2 bmdmeBaVLEGSI/Glyish9FScqGJ9o2DTGIjRUAUtBJ7QnPI6wKNn9hP2NeZo7YR7tZxV BKUm/a2emsKtp9nE3nRmGRf1JnSB3PI5nb7tw6x8q8XfJ+Aid1Rox19zDz+NKHmhBdnj aYHXoRDO0McBoa4LbY1oXgnmyf5N14/wVojJlmAFY2pdWa60DZmneMNDlCj/DpQtQgc7 mpmkvPNB59noYX3eRbrf01SjNwi3Y3aHrDLgD09bAB3ZIkkGSZroSSSSs8zVydM9dc5E ypxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1752805009; x=1753409809; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=bjPUtTo5Q4RHjWDwA/8ArVUYFnewjb9EWWS17pFPGq0=; b=KwAjQNa2ELMdgNseIaFXirG9jUVW+ISNzCmGfDN1twMhqp+JintV5h0MKN0tdFFm+8 G+5Qyfaevb1JRmE4D9hcNec6Jc1hIw1zTztHSnmgXZUFXwp0gq3mqUhkxjpKADOs+T7S m1X2pBJbhxRiJNPpymqrJBhr1Pm2wIMiyKj4saQoX7aWZcWVNpfHZ+nki75WM3V+AT8c hQJwlpnMUDEL24ohOi/8ubuoiCVQAN1oGr58gb7qmx/H6Vf1itRI4I8gpnDTeL17kjFc 6w2b6ECQqZzhj4m5ikeDNuJZ1FV++TaiP9pYuhwmf3MSof85HS1gGX6XiaDiq4ojywuR 4EDQ== X-Forwarded-Encrypted: i=1; AJvYcCWS6jKDEYdl9U+4QUOe68b15eJoVVxfwot+3rD3pxvSCaApvwE43lzTioIKnR8cZH3OQtrAFDGySg==@kvack.org X-Gm-Message-State: AOJu0Yy2dw+OsXS429t+Lw8NPL1mw6pPGLPsOX+WYfoPzlFOyGvSaMml vy4jiah7Z1o/fHnK1tBnnpOAehJWCEIjJCeYvs1l6lCvkgvv4XLZc7p6 X-Gm-Gg: ASbGncvS9EM3p6e0xrjGG47TOYTlSdlxkLBFq/leog/YvfWgkPt5NFIqNM813cXv2x+ ziu6AYZ05tEsoTOVoE5SiKmIgADKaUMtekEbMAzl6Ib4yoQKLNgYvT9II3Wk0ZKFKh2QeEZjFBI usiPnYbn6q5xgfN8W2bMTcXKL27zsyat5B9J0sjPzhuXDdRdHStPyvS/eTHZ6UR6imFZtv5aGTT R+3rskHeWLWQDOJyWlzO2Vumt7dWs0FfEtwUhzU2/gniI0u9Sf5w8ymFi7Ci4RNCmGG0qeWnntw 442QpMq+GvEPgNFmu8xJlEOxHUGW1J34HTxHdL2ciRSaZY1HuUiVjAEhD2w4eSCvlSiMK5iu1n2 jMFoWvxHgzWzgrwE73YFpMTnhkUT0SnpBV9O2+slko5kDD5pOSMt/rWGixizrdCk= X-Google-Smtp-Source: AGHT+IFUs3XFpbvaHD7JdqhEQ2KJXQRT8Nq9OZyid4ETRatsPiNyKJKqTiWH2Vs2/DixhBEDNe8Wkg== X-Received: by 2002:a17:903:1447:b0:238:120:134a with SMTP id d9443c01a7336-23e302cad77mr81099765ad.22.1752805008892; Thu, 17 Jul 2025 19:16:48 -0700 (PDT) Received: from localhost.localdomain ([2001:558:600a:7:a83d:600f:32cc:235a]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-23e3b6d2fa7sm3154335ad.162.2025.07.17.19.16.47 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 17 Jul 2025 19:16:48 -0700 (PDT) From: Alexei Starovoitov To: bpf@vger.kernel.org, linux-mm@kvack.org Cc: vbabka@suse.cz, harry.yoo@oracle.com, shakeel.butt@linux.dev, mhocko@suse.com, bigeasy@linutronix.de, andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, rostedt@goodmis.org, hannes@cmpxchg.org Subject: [PATCH v4 0/6] slab: Re-entrant kmalloc_nolock() Date: Thu, 17 Jul 2025 19:16:40 -0700 Message-Id: <20250718021646.73353-1-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 37090100002 X-Stat-Signature: nmyims91fjumajs87nozgofgbonkyyrg X-Rspam-User: X-HE-Tag: 1752805010-202906 X-HE-Meta: U2FsdGVkX1/fhfWX7A40s9jM+e6yDQRADE2rqL9qvSl35kX4nTxkLy4CgIbGs6NW7X3vsy208Rd4ZhHEJuveAw5QESynS9izyimXCAVJtKcFTT6Oa1lMLCZt4er+Knj51N6M8gKCN3Vwk0s0LaT79f+ooghT/WDJExeyOacrsnNZjzUGobGr9ZTUNfzbZI1+FnlbbT21URifGzsBeYKOg44pTgmU8ywtzlUCa7N91xNT7MboCT9H/8gBgN+DAaJ5FFA4BJqpEPIzY5E7FV/qWBQ15vOyYiV1z/N6awsmw7IdQkqolRYtBxsIlQMVXCe+xX1+nwZU+g/m1Aqukqo4DaGzt3DzhtLYqhTBOYCexi0VQZvJtDBXBBh5VrTO5k5Hcq2/h/I141eMk22+SApRvPRm0iCbz7t0n/hnHoCwFCvkmFMPGTUs881PHH2Xi//sDhjyNUTpM0HLVKcNnL5TjJfNn5nNUezsow8tWyaTtFPXbEIT8aoQR3nf3e7UWDaPOmO0+RUwMMy8FYPVxL49iPk0kEpG3k3UsQz1Q29j4GnkdcunbyBupp3kSbrlO+TB2AHRqowV70XPwJNvGHmDzbyxrSN41F6GNdERTVpfQ9afIh9urKQVsXmpX1zUYz8aRED+NqzHdOlR2j3kbzZIyCD9Ykg/sS/wv8NgGVpGx9QdkpQc9HGMGMLWut9so8eOqv683kvXKsVXPbOYwxRf6eBX4C4OFOMbw6nCT5//0oUqAM5F1khv6yLL6sT7LvSOyp0RMf6oesXzOrECv+FQPiUSVSo+Raedhlq/sBnNWbwmyFevveYn612u2+fO2D2J6T792VAWPLdLp1sHBs9LUOFrhcHQaZSYByJ0nKGIfi911vC3/qT/Eb164gpQULkWvKOuIoE5iOISlzcShNMUTku1Drgx44uUqf+jpvygl3Z/7Cu7iS0pHoH+aHkyHaC0oHkkodb5/kl1wjr3Wp1 7K9Z+aGG jDpMTGgoMokOV5sEMQSKQsmq+maeT5Xd/IykvUb+mtxd0DikUnmdb8w4JSK9GrY/3gvZPbeMcSn31A+5sqo/eryrwOy3TviZhY6NkOTfsHNmuLlKDIPKe0U7wCLzWd6K+qI9Lyzciml572XfR1nqqjHUqzdFMY+/mD4X/LzV2ZWvNW9G44G5sNALo8hfHZSF480kgp3aeHMJMxXaYeLs/zbengmARMxKhVTa0EIEKs/HC56wfYxv3kjRjjGTaLmmezXEyBgJX7OMMePZok1NRDG4kQp//AMxaRL11DJSdylqGQ3Bf0k6VfvNY3XpZreT8PYfROHASo6JS0RQ4YZ4A7Usj/0SmdZpAy7yus959QlWNoLING0C7qkPAhCK5wVpQybv5d020WYSgeIVIQz/9C9dfB4rjT9tuG8Ho78gBPK1DfEIzqFQScDaoz0q8lPVmSxBOLX2cgXRXStSpOcS/lmvdtoXnHfzFd4tkwB3XCmsL1Rnf9e4TCCpfHJTiuFVLOgaMkLVvw2lJYBdYfLb0Qdf+er5nU0m5eazGfPCXWDrDrJjekriVi+6gqRmNwAm2nBTbn+Sdriksx6qABd7Kmhj+p14rWsrtCwaSNT29aRgkUxpWqhIFdAx0qFGb2HqJAdKcXFurRaWbMHk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov v3->v4: - Converted local_lock_cpu_slab() to macro - Reordered patches 5 and 6 - Emphasized that kfree_nolock() shouldn't be used on kmalloc()-ed objects - Addressed other comments and improved commit logs - Fixed build issues reported by bots v3: https://lore.kernel.org/bpf/20250716022950.69330-1-alexei.starovoitov@gmail.com/ v2->v3: - Adopted Sebastian's local_lock_cpu_slab(), but dropped gfpflags to avoid extra branch for performance reasons, and added local_unlock_cpu_slab() for symmetry. - Dropped local_lock_lockdep_start/end() pair and switched to per kmem_cache lockdep class on PREEMPT_RT to silence false positive when the same cpu/task acquires two local_lock-s. - Refactorred defer_free per Sebastian's suggestion - Fixed slab leak when it needs to be deactivated via irq_work and llist as Vlastimil proposed. Including defer_free_barrier(). - Use kmem_cache->offset for llist_node pointer when linking objects instead of zero offset, since whole object could be used for slabs with ctors and other cases. - Fixed "cnt = 1; goto redo;" issue. - Fixed slab leak in alloc_single_from_new_slab(). - Retested with slab_debug, RT, !RT, lockdep, kasan, slab_tiny - Added acks to patches 1-4 that should be good to go. v2: https://lore.kernel.org/bpf/20250709015303.8107-1-alexei.starovoitov@gmail.com/ v1->v2: Added more comments for this non-trivial logic and addressed earlier comments. In particular: - Introduce alloc_frozen_pages_nolock() to avoid refcnt race - alloc_pages_nolock() defaults to GFP_COMP - Support SLUB_TINY - Added more variants to stress tester to discover that kfree_nolock() can OOM, because deferred per-slab llist won't be serviced if kfree_nolock() gets unlucky long enough. Scraped previous approach and switched to global per-cpu llist with immediate irq_work_queue() to process all object sizes. - Reentrant kmalloc cannot deactivate_slab(). In v1 the node hint was downgraded to NUMA_NO_NODE before calling slab_alloc(). Realized it's not good enough. There are odd cases that can trigger deactivate. Rewrote this part. - Struggled with SLAB_NO_CMPXCHG. Thankfully Harry had a great suggestion: https://lore.kernel.org/bpf/aFvfr1KiNrLofavW@hyeyoo/ which was adopted. So slab_debug works now. - In v1 I had to s/local_lock_irqsave/local_lock_irqsave_check/ in a bunch of places in mm/slub.c to avoid lockdep false positives. Came up with much cleaner approach to silence invalid lockdep reports without sacrificing lockdep coverage. See local_lock_lockdep_start/end(). v1: https://lore.kernel.org/bpf/20250501032718.65476-1-alexei.starovoitov@gmail.com/ Alexei Starovoitov (6): locking/local_lock: Expose dep_map in local_trylock_t. locking/local_lock: Introduce local_lock_is_locked(). mm: Allow GFP_ACCOUNT to be used in alloc_pages_nolock(). mm: Introduce alloc_frozen_pages_nolock() slab: Make slub local_(try)lock more precise for LOCKDEP slab: Introduce kmalloc_nolock() and kfree_nolock(). include/linux/gfp.h | 2 +- include/linux/kasan.h | 13 +- include/linux/local_lock.h | 2 + include/linux/local_lock_internal.h | 16 +- include/linux/rtmutex.h | 10 + include/linux/slab.h | 4 + kernel/bpf/syscall.c | 2 +- kernel/locking/rtmutex_common.h | 9 - mm/Kconfig | 1 + mm/internal.h | 4 + mm/kasan/common.c | 5 +- mm/page_alloc.c | 54 ++-- mm/slab.h | 7 + mm/slab_common.c | 3 + mm/slub.c | 486 +++++++++++++++++++++++++--- 15 files changed, 528 insertions(+), 90 deletions(-) -- 2.47.1