From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DA926EE6455 for ; Fri, 15 Sep 2023 11:00:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7178E8D001A; Fri, 15 Sep 2023 07:00:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6C6158D0012; Fri, 15 Sep 2023 07:00:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 566178D001A; Fri, 15 Sep 2023 07:00:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 416D58D0012 for ; Fri, 15 Sep 2023 07:00:04 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 185A1A1057 for ; Fri, 15 Sep 2023 11:00:04 +0000 (UTC) X-FDA: 81238537128.15.E34D457 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf29.hostedemail.com (Postfix) with ESMTP id 2161E120015 for ; Fri, 15 Sep 2023 11:00:01 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=WUH6ZkyG; spf=pass (imf29.hostedemail.com: domain of 3MTkEZQsKCOIQEXXISVMddSKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--matteorizzo.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3MTkEZQsKCOIQEXXISVMddSKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--matteorizzo.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694775602; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Rgcw5kSlwTUZsnv8liQ/ZCTx1m6umum1AvnN+yk4ci0=; b=Jxt0NgdspwDpQjC8mlfFqnxoTw+6pDIowmruNQwS+9nrPweWmxjkL5JyPUqe4aIo4KAQCF zk65g3m2N26qaghuvcnHMVCOxywWgAhAfk35L8AFqQGBeniQSQcrP66mWoDKLkPj4qkK0m J6m000gybGSJFhFz3618p5V5yfX67dc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694775602; a=rsa-sha256; cv=none; b=4QgZxLDe8qhq4LxeIpMK1f/lqMnjICX0fAgOrsECJy8Qpkt29RozJptEqwVuEn7SxBmXZ0 Kofq4QKt2uxqN/MOaRl/QHQd2CD9619VgTvLNljc9V1bUxVJev01rOxfhwNopnA3rBA8gP O1qrpOq8clq9oFyROCeR3gNmCfII0iA= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=WUH6ZkyG; spf=pass (imf29.hostedemail.com: domain of 3MTkEZQsKCOIQEXXISVMddSKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--matteorizzo.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3MTkEZQsKCOIQEXXISVMddSKSSKPI.GSQPMRYb-QQOZEGO.SVK@flex--matteorizzo.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-59c12d31d04so4567147b3.1 for ; Fri, 15 Sep 2023 04:00:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694775601; x=1695380401; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Rgcw5kSlwTUZsnv8liQ/ZCTx1m6umum1AvnN+yk4ci0=; b=WUH6ZkyGiesMGaFC1qnIhzOe4HtaNZgdqCtE3OARU6ueGvqWrbZb2MW1CpO8Bj3O9F RbjAaRoKYGwgAuYdKoM5HxO9sQNISZIzZ6BUIARSj1exyqLGSfZXhnYzXD4mPMnRP5MJ VeFhkO/XfHM9H78s9QO4TW5wsY88I2EkADCauFNzndHHvPjJaXJeiRFHv41FHDLINd60 o3jfJYfjR7bElH3ieNxilvesJ+UtPhQV8+4kUP/xEAvi5mhZ0ueK/fmf68bM5PQUpuvP pkGnrymsDQYQ7Bl6lqJTUmzWmg0PU8T8YxydzCIB2R4yjYpOutKRiJOwI9CftYe3LyNw VaCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694775601; x=1695380401; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Rgcw5kSlwTUZsnv8liQ/ZCTx1m6umum1AvnN+yk4ci0=; b=ZUtQHT6uu5Un9WfbvU+qS9yWB91e235qxa7fvX7beCNlwzNlgN3Z3nLjvawcN0BxT2 To5ErPFxBXAy8Xx5Ows2F+TnUs5VM4sMaSqQiJfCVKFyCbpC+U8MuGOOJ70ZmPYV+4yq VA1bzzS0579vN68IAsCc2dA0tj33qQLQ093/4rM9YDNzvmIFROMiFOromDjalkdvCOW6 t8/lgYFiPNTEs8nr206bTdNt/ml3S4F+eQOclM7/rqKsjlw/QYYybxVD2X+8UtcIXnRi IsZtRW02y0EVPvN914wg4/HlQ0yXcjzfyfzAXe1iLl60Mo4fmcdqhN2AaBaAbuoniCKY W3gw== X-Gm-Message-State: AOJu0Yw0ryENSzBf9UJO6qTkrmxcysUCQeDMqGtIu8JSRfcS9JsV0mIi 6j2p8nzyz2Bb+9oEl21nraPh2+SEGsNn65zIZw== X-Google-Smtp-Source: AGHT+IEcouH8JvuJElm8N08gEPWOgDsvJMW1WGDf8e1nxWq0e0tWAe1u3fFjyWd///wS5TI4Lwp37X8nvTPZK0q/lg== X-Received: from mr-cloudtop2.c.googlers.com ([fda3:e722:ac3:cc00:31:98fb:c0a8:2a6]) (user=matteorizzo job=sendgmr) by 2002:a81:ae43:0:b0:59b:f493:813d with SMTP id g3-20020a81ae43000000b0059bf493813dmr28901ywk.1.1694775601139; Fri, 15 Sep 2023 04:00:01 -0700 (PDT) Date: Fri, 15 Sep 2023 10:59:28 +0000 In-Reply-To: <20230915105933.495735-1-matteorizzo@google.com> Mime-Version: 1.0 References: <20230915105933.495735-1-matteorizzo@google.com> X-Mailer: git-send-email 2.42.0.459.ge4e396fd5e-goog Message-ID: <20230915105933.495735-10-matteorizzo@google.com> Subject: [RFC PATCH 09/14] mm/slub: add the slab freelists to kmem_cache From: Matteo Rizzo To: cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, keescook@chromium.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-hardening@vger.kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, corbet@lwn.net, luto@kernel.org, peterz@infradead.org Cc: jannh@google.com, matteorizzo@google.com, evn@google.com, poprdi@google.com, jordyzomer@google.com Content-Type: text/plain; charset="UTF-8" X-Stat-Signature: a1mt6idjr3ru8xxo8jidz5nbz9rh1hb9 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 2161E120015 X-Rspam-User: X-HE-Tag: 1694775601-920704 X-HE-Meta: U2FsdGVkX1/Fok9jGSEEspmTmGi/SXTzyDJwAwtryKg8EThsdxWYD3kqcsVjNk2OIwyWHRK17Jb6G/zAo83jswQsmYAOjMHV8YLNDnOBA4o9224J7PRvg5Gy1OazkI1H9dcm9R5+h0vn/Gr7mZWgkURzPnM27BFn9QjtRzzoTJZLZy4ZPEWue2x10sBjgRfGp872Fo67ExcCFPNldEpbyY3Odz8HGfN57YIXFhx09UkKg3BGCJRQbK3kLQ/ROFCBYHrudTKrFBd81+QbMiVTxKfkzda9EhCslxSmIYClEvZQa01RlqHFVselEz+m6rFKCHX0A8F3sVDRT2T5meUWfULwwu2sRDyc4+WenS/F2Nt9R2kyjUo6cohvQadQI1SUgDIYGA/BdSIVfjYypIvVc2zhjM7KUHBFTTMbi9vjrESBx/60skqMyCngiFk7pFb9Q02FBq5MBvfYuCJOhcEvtsacOmKdGT8iWTVyjacw+ShNjkGgy4FsQbSBTXoVV38Lds8SRHadO1Svyipw9GETroOR2oWTdaJ4Ffo0VaqhoQcZ/QVIG0VKh7Srw4dvNM395cKv2vCXJmQiNAHL1Gtd5jBeSvUQfNc2VaDA9/bs7Aeqk+6gFq1yMGvv6zlfU9rEf7susdu7QcdJaFw2+eavJQSGGDkfu+T5Qy+dVIRMJcLreITp92xtJ7bDx4wupHGA/LRovTpkI2PNCtu9ktjNGjho02U+soaswKt1g/HrCqY1RrwOnsgxruL3pXjs85AiwwpVJTd7OejFYrjiu+ClV0woNcH34Mr/Se56gofsAgjaLczBlZ6uE13ePir9D3QSFUy6uF4jW3Fie79fJu+5YRpzc1NKNcd+xWuD4/Lp7LmIpcBQFPHPEz6afW0SuHLDmveN4Ycs9yJVjMjx7hVz+iF7t9wBfCGcAAilXwSXcNFCtgbAdhm1UnS29EKOEzby9ZUfWvYFlQEL/s7VKBq 7UF5cQlM z+dEeQxuZDj/i6DOs89oTBtoZ3yacTFKJeerUOztmfcf1wqK7Dk3G4W3GyaJyXvSY1tWAq7c2IajVqLK/ZqcGpZoEaRgKe+0Douub+syJequ+3Kb58kd2MgAl4QXeHGBTgi1Ab4zXYrB5iusBGTPsSJvmjrIxOVnPoSxk1o9AIbMLzBU4SIgeg5xWnCVqdMxk1JAppPPifnZ/nkzS5BASlSnAdWTyBdEzUfEOACEiGDzDjZSEHcljxcy9sKMo5JBkNhmaCjXrivnj0H0p+2UM3zAEpA2U7dP8dfl9HaWYNurwOVedfKG53uZaNqZnwb7ezJ7sVBz0RCiUCTo4e14hVYbNsFyk4I12pMS7ONv5lBRCa5Y1F4qf+Z8MTM68dLX3NqUblZDLpYUo/bKIkl7z1x1twCrUAtJHmVZEjXRbXHaNdY8gob2xKlpxNcF0IbGF0bOBXrcyd7E2kZgCe4FRndZZTpFMhlzXGg7aIbh/tamPGyrWBVKlH4ZFAkgXq0c/s7yRnR+KpVX0LGbhdNAFJJJT7LBayOt8welPkhVFPTEW8jfbrE5mMzBm+/yESJYK/qEt0ZVfpfMRgQDRBF8RwQVWfktYNCRqlK6gDWUnPpuR0UpgX3NTVm1I0LcTB7bsnmzUP5AOtvnPv9cz1dKGjFl9Xa7zHQ97FgiGlTjmGzVanX/2a4fKsbpWokc2mr3jlZH8hG1wqkrVN5JdByAK1wHv5ysoumm9CGplowuO5M3iF+qJ4h9r/Kumocv7lOgDtjvvR76vjTACesAgkK8j1cYP3hSKRjfnZlraBUmP/hjWGAY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Jann Horn With SLAB_VIRTUAL enabled, unused slabs which still have virtual memory allocated to them but no physical memory are kept in a per-cache list so that they can be reused later if the cache needs to grow again. Signed-off-by: Jann Horn Co-developed-by: Matteo Rizzo Signed-off-by: Matteo Rizzo --- include/linux/slub_def.h | 16 ++++++++++++++++ mm/slub.c | 23 +++++++++++++++++++++++ 2 files changed, 39 insertions(+) diff --git a/include/linux/slub_def.h b/include/linux/slub_def.h index 0adf5ba8241b..693e9bb34edc 100644 --- a/include/linux/slub_def.h +++ b/include/linux/slub_def.h @@ -86,6 +86,20 @@ struct kmem_cache_cpu { /* * Slab cache management. */ +struct kmem_cache_virtual { +#ifdef CONFIG_SLAB_VIRTUAL + /* Protects freed_slabs and freed_slabs_min */ + spinlock_t freed_slabs_lock; + /* + * Slabs on this list have virtual memory of size oo allocated to them + * but no physical memory + */ + struct list_head freed_slabs; + /* Same as freed_slabs but with memory of size min */ + struct list_head freed_slabs_min; +#endif +}; + struct kmem_cache { #ifndef CONFIG_SLUB_TINY struct kmem_cache_cpu __percpu *cpu_slab; @@ -107,6 +121,8 @@ struct kmem_cache { /* Allocation and freeing of slabs */ struct kmem_cache_order_objects min; + struct kmem_cache_virtual virtual; + gfp_t allocflags; /* gfp flags to use on each alloc */ int refcount; /* Refcount for slab cache destroy */ void (*ctor)(void *); diff --git a/mm/slub.c b/mm/slub.c index 42e7cc0b4452..4f77e5d4fe6c 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4510,8 +4510,20 @@ static int calculate_sizes(struct kmem_cache *s) return !!oo_objects(s->oo); } +static inline void slab_virtual_open(struct kmem_cache *s) +{ +#ifdef CONFIG_SLAB_VIRTUAL + /* WARNING: this stuff will be relocated in bootstrap()! */ + spin_lock_init(&s->virtual.freed_slabs_lock); + INIT_LIST_HEAD(&s->virtual.freed_slabs); + INIT_LIST_HEAD(&s->virtual.freed_slabs_min); +#endif +} + static int kmem_cache_open(struct kmem_cache *s, slab_flags_t flags) { + slab_virtual_open(s); + s->flags = kmem_cache_flags(s->size, flags, s->name); #ifdef CONFIG_SLAB_FREELIST_HARDENED s->random = get_random_long(); @@ -4994,6 +5006,16 @@ static int slab_memory_callback(struct notifier_block *self, * that may be pointing to the wrong kmem_cache structure. */ +static inline void slab_virtual_bootstrap(struct kmem_cache *s, struct kmem_cache *static_cache) +{ + slab_virtual_open(s); + +#ifdef CONFIG_SLAB_VIRTUAL + list_splice(&static_cache->virtual.freed_slabs, &s->virtual.freed_slabs); + list_splice(&static_cache->virtual.freed_slabs_min, &s->virtual.freed_slabs_min); +#endif +} + static struct kmem_cache * __init bootstrap(struct kmem_cache *static_cache) { int node; @@ -5001,6 +5023,7 @@ static struct kmem_cache * __init bootstrap(struct kmem_cache *static_cache) struct kmem_cache_node *n; memcpy(s, static_cache, kmem_cache->object_size); + slab_virtual_bootstrap(s, static_cache); /* * This runs very early, and only the boot processor is supposed to be -- 2.42.0.459.ge4e396fd5e-goog