From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EEE16C83F17 for ; Wed, 16 Jul 2025 02:29:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 274CC6B00BD; Tue, 15 Jul 2025 22:29:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 24C546B00BE; Tue, 15 Jul 2025 22:29:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 189CF6B00BF; Tue, 15 Jul 2025 22:29:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 0B95E6B00BD for ; Tue, 15 Jul 2025 22:29:57 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 8171A1D8C4E for ; Wed, 16 Jul 2025 02:29:56 +0000 (UTC) X-FDA: 83668547592.12.8D2EC28 Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) by imf20.hostedemail.com (Postfix) with ESMTP id BA9A51C0004 for ; Wed, 16 Jul 2025 02:29:54 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=J+CmTtFN; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf20.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.52 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1752632994; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=54YMk9JDQn8IHKAz5YLE0koa8eulLZZ41W4aFx7nouU=; b=jLDWNzEbciJWr0jk7ARfkqt33Yrtzqq+eDRq0PcluREV5NhMILJvbS4KmCB0DKqfa/RBqv 2AFXV8k8MiyhjpvLdZUrc83NpQBYV7LZZtlYUu5fD5iVZ8FeVkKJpWKl5eIhexxGk6aGtO 1XaHcUgbUHYk7JM/eOsRqMuag0Dh2dg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1752632994; a=rsa-sha256; cv=none; b=DI3IembqruBBP49t7YAfBYPS0QV50GLHGz/wPKoGO8RSDnvSBXGTfozhY3vSaYH8lv8n5z e+nrxCqikcMW9bJk2BjM5N8sNl9c61xAnf5nt6b1/bbhxILIfsUHTWNXRVqy6xJsjcksr/ IUOpD5zs2EPFRfiftfIdh8qYyZ0hrc8= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=J+CmTtFN; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf20.hostedemail.com: domain of alexei.starovoitov@gmail.com designates 209.85.216.52 as permitted sender) smtp.mailfrom=alexei.starovoitov@gmail.com Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-3137c20213cso5859767a91.3 for ; Tue, 15 Jul 2025 19:29:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1752632993; x=1753237793; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=54YMk9JDQn8IHKAz5YLE0koa8eulLZZ41W4aFx7nouU=; b=J+CmTtFNL72/uB7ag6Vm2ZZCjlXsL0G4j0Rw3LZYvFhBMi4bMUG1R2GEmrS6+m37xa MOJx/aOljXqm9bNLdeX5ttjtFlJVYDKX9/mydssKJXLNVi9Fc2/PiFY8NA2zhmlvzkjt eSnyuR7EkH7B6lqIKEmRtI1XzYl8AKgGvSQW0CP1X7QQd/GddrhvRgEjxMRDIbqtw2x3 TIhc3+vLCLiWE1BlNM1HfgYsueney4TskciNHVA9jDuvMDE3KLXjehXPUJCYD+0ppuj6 Td8MKtB3YlxaflnHj8yskPTFVDZ4d7qLJUFgdiowxbvlbiTsAUYlqpkwaFBnB0GM/zHQ OKjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1752632993; x=1753237793; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=54YMk9JDQn8IHKAz5YLE0koa8eulLZZ41W4aFx7nouU=; b=VY30EzVODrZx8+B/DAqI6bJSzGY4Jpy/C/JRDVOZ9LyQqj6nh9Exf5PC7BRlfQKV++ OeGQowysFM6SbtW2Jw5vPU19gzSGuevyQP+GhECoYaLWD5JhHOZET1W1+JYSYsMTO3ah r/YsaXWiAzwygq5pmhH2YpT2aCDUyfEbeaAITx3MPc5F77n0q0B7ZvTGIyZbFZErmBrw IhUHG4Z7/Urg1rajCbCFMPiKDFhjHGC+aUExkse+HB3A+YHJm/sjplGEyxujQlIBwekw UWHGPHh2ixCJl/jVA0WhixnK/o9YclNZK0eKnaKj6R9F0AL9QGENO7OG8rq90dEW3VM3 EiMQ== X-Forwarded-Encrypted: i=1; AJvYcCWfECZK9bTC+48+yoMIGfjByzhcB0i9eTWUQ0J4kz4xrLbaJL60XEA+z/ohX5NlJOYWX+wo8ts+EQ==@kvack.org X-Gm-Message-State: AOJu0YzxsnyvBelyqNm9LrBq1A4OwAgpNEuiOfCp1ty0traxk1mrKhBn kfnLqTf70wA92E5+d81J3ZaLgkMSfYkofIIeukrXXLigavevAHkuCZaS X-Gm-Gg: ASbGncuXA9/tYpjMHDUIoc1hmWKt4b7CtV8M2DV36xJEPLUcOcJl/TypQTjoX4+VEdc 2hDicWj4YTD1gqxPi17Hcwo6j1wxilYvN95FWXnerOYpg2hqZCT9VoVRZX0Itq7XLK15Lt2ZIHX VPGWhDWLdct9fL7CAmJabhvwSWKVIkG6SxOQopzBlk+ReVbGPGQP1d+l3Vn5zYCwVTaLl98l1yC D7h+DXhyeWg/Q4NeYGq6+7CojSgFccZEmcBmzwXLSEORi0AfVVaWSeaG1XQTwFQdp69NDyNgRUn dYYDXVKZlSk5LnHc0FPEpQjgZnRMggdF8ahNHMrFypbJsHs6Kw42U1jx5FpW4LpSo6hEsP/AuzF cQsgrU+4gVjVeQyaGzDY990DJy5KoYycue33vzj/zUofN0qcHxLcPxFbDhK69VMs= X-Google-Smtp-Source: AGHT+IFH8pUexJCnFJRvpl8rZhS654EBz2PixSOVjcUbuVR23X8JzXXfrQK1qEDlWeKIKXuRZG/jMA== X-Received: by 2002:a17:90b:1c82:b0:311:ffe8:20e6 with SMTP id 98e67ed59e1d1-31c9f3eeac5mr1164056a91.3.1752632993285; Tue, 15 Jul 2025 19:29:53 -0700 (PDT) Received: from localhost.localdomain ([2001:558:600a:7:a83d:600f:32cc:235a]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-31c9f2879fcsm338049a91.28.2025.07.15.19.29.52 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 15 Jul 2025 19:29:52 -0700 (PDT) From: Alexei Starovoitov To: bpf@vger.kernel.org, linux-mm@kvack.org Cc: vbabka@suse.cz, harry.yoo@oracle.com, shakeel.butt@linux.dev, mhocko@suse.com, bigeasy@linutronix.de, andrii@kernel.org, memxor@gmail.com, akpm@linux-foundation.org, peterz@infradead.org, rostedt@goodmis.org, hannes@cmpxchg.org Subject: [PATCH v3 0/6] slab: Re-entrant kmalloc_nolock() Date: Tue, 15 Jul 2025 19:29:44 -0700 Message-Id: <20250716022950.69330-1-alexei.starovoitov@gmail.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: BA9A51C0004 X-Stat-Signature: 61t9k3c6dod63osbnw5d9dhofuy79b3j X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1752632994-906849 X-HE-Meta: U2FsdGVkX19VVLwQFITgDHoj6YcsAOsxlsC6UUYqZfhPF6XohKuoyzX8lNAG+7/QUnMbtqWGnp+9cvL3vLo0O+5LZWpByxE3AEwJqERRzVLC/ue7UB5sEaXIzSukO87WFhZrSsNUL+Gv6q4QiYFur6IlvxVWjD3cf/JB0Yere4Nm2ju43ivrbAGpwXpN4Qp0je/ann17j9H1pCGw34TVFcI/A2fU6C6yvAvnlQE91c7YHsnHtBCPyX5xIg7wTpg03yYxDfBaJV1b12IpIN4G8ReNW9o5M9XCqmL+85WqyB4dFHw3wMp5Jb7opfwbYcb4KFU0AG6mdZ5gFngdqo/5JMh38kpEgl978e6FI0o8qQKOpsr8AdHTQxhkslvk9D+gWAhM/xLXSrkxfb1UY6wkAVz5E0e069cvDVe0KCGaGCQkwYW79o8kgswPondID9P4a21DmCfk0EsGHkdA2VcGFu13LZIbrm+DDIJeub8GAo/DU0vDQ+3Q1jUUUKE8heI2UiOTv1D8tbpnnSSy5krQjA6azSmqJ1rwZ7LFThw7OncK2RV/kyB0Ti3o2qqe4CmDnvjzbmH+/uciwsxUBmG8Z3MNpekIIEBC9cwfFQLRCzwS+Y3rH9Z7kNzhwNsvoFApHHFk7sUCupbgCoDw9j1IN6OlygesqLUj39DnvRZ8cY/a2W8x/DE2NVmDReRF/KpnVsBFJhGhI5SdcC+QIVNhrQBUdXMsh2blohJqx8c7iINGDNHQuPKmp4tt5qSCI1h15WFrc2jByLEfOwTnSFch3z34zZnS4QLRXaWHzrwPSam7rZWxsFdALg6DJywYYkfhcT20S5zWHoF9gzFlqG3ikzBY9eqA3d4lswBZHHMulvguJvkdVUIb39TaNhqhXsZQe3bvz3tNCRh2MROQ7nJbgd5gVt6ycLCktGbfywnbERwZTt/Vh6O0mPb5HpSe948iSrK/KSse7dAGeEDVCrw diUjboog Ku8H8YyKlIkQ/NrmiPuEryZVF8Tl3R2n22F2WiAb4jJZoBw0AfHiQ8QmkbqKnoP6R637Z4//bXBb06rg0UMFpvYCROg98HqsGrxPVXXtImdwJhFMIOfJnYr4GT71M5tD5FtKabYSdbX1OwghUTzlRgzhE3z6uMKy/xs+xILnuMpeWyQ3t4SMCAxRro2QdIjBwilUJv1xnBlLBuvkLLrIfcdnP2nLTpSXS7ncdvfybYi1z0x2Wb/sDB0C5lnXRsb3xghwjy64L0HbjK1mzmTZPNQY8e919CwOBfnt2ZiRhHO7xXTHYpBYdz7zr6nRqNA48PQHdc+qv0XSRmW7d4Y3KW7l2unifNPIl8ep9ZcxlJ+O5AugPo5UtqeIymTPRTi30DGHpHk144bnkoZ1QrHu3n6VPtvO7Qo5ddmGOPqQlNHtpc6EykeUPbHr9sFLp8oihXQzxQA4E8qg0AtojBS/RZml7IS+gZgPcm8tDFgsjjeD6hRuQrUv0xx4naxTv9VCQBMT/AnL6dbi7CxnNX/uJNhldQ3U4qFS6xJ3nzwSF5M4Np2fUHTgKOJ5rHA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Alexei Starovoitov v2->v3: - Adopted Sebastian's local_lock_cpu_slab(), but dropped gfpflags to avoid extra branch for performance reasons, and added local_unlock_cpu_slab() for symmetry. - Dropped local_lock_lockdep_start/end() pair and switched to per kmem_cache lockdep class on PREEMPT_RT to silence false positive when the same cpu/task acquires two local_lock-s. - Refactorred defer_free per Sebastian's suggestion - Fixed slab leak when it needs to be deactivated via irq_work and llist as Vlastimil proposed. Including defer_free_barrier(). - Use kmem_cache->offset for llist_node pointer when linking objects instead of zero offset, since whole object could be used for slabs with ctors and other cases. - Fixed "cnt = 1; goto redo;" issue. - Fixed slab leak in alloc_single_from_new_slab(). - Retested with slab_debug, RT, !RT, lockdep, kasan, slab_tiny - Added acks to patches 1-4 that should be good to go. v2: https://lore.kernel.org/bpf/20250709015303.8107-1-alexei.starovoitov@gmail.com/ v1->v2: Added more comments for this non-trivial logic and addressed earlier comments. In particular: - Introduce alloc_frozen_pages_nolock() to avoid refcnt race - alloc_pages_nolock() defaults to GFP_COMP - Support SLUB_TINY - Added more variants to stress tester to discover that kfree_nolock() can OOM, because deferred per-slab llist won't be serviced if kfree_nolock() gets unlucky long enough. Scraped previous approach and switched to global per-cpu llist with immediate irq_work_queue() to process all object sizes. - Reentrant kmalloc cannot deactivate_slab(). In v1 the node hint was downgraded to NUMA_NO_NODE before calling slab_alloc(). Realized it's not good enough. There are odd cases that can trigger deactivate. Rewrote this part. - Struggled with SLAB_NO_CMPXCHG. Thankfully Harry had a great suggestion: https://lore.kernel.org/bpf/aFvfr1KiNrLofavW@hyeyoo/ which was adopted. So slab_debug works now. - In v1 I had to s/local_lock_irqsave/local_lock_irqsave_check/ in a bunch of places in mm/slub.c to avoid lockdep false positives. Came up with much cleaner approach to silence invalid lockdep reports without sacrificing lockdep coverage. See local_lock_lockdep_start/end(). v1: https://lore.kernel.org/bpf/20250501032718.65476-1-alexei.starovoitov@gmail.com/ Alexei Starovoitov (6): locking/local_lock: Expose dep_map in local_trylock_t. locking/local_lock: Introduce local_lock_is_locked(). mm: Allow GFP_ACCOUNT to be used in alloc_pages_nolock(). mm: Introduce alloc_frozen_pages_nolock() slab: Introduce kmalloc_nolock() and kfree_nolock(). slab: Make slub local_trylock_t more precise for LOCKDEP include/linux/gfp.h | 2 +- include/linux/kasan.h | 13 +- include/linux/local_lock.h | 2 + include/linux/local_lock_internal.h | 16 +- include/linux/rtmutex.h | 9 + include/linux/slab.h | 4 + kernel/bpf/syscall.c | 2 +- kernel/locking/rtmutex_common.h | 9 - mm/Kconfig | 1 + mm/internal.h | 4 + mm/kasan/common.c | 5 +- mm/page_alloc.c | 54 ++-- mm/slab.h | 7 + mm/slab_common.c | 3 + mm/slub.c | 471 +++++++++++++++++++++++++--- 15 files changed, 513 insertions(+), 89 deletions(-) -- 2.47.1