From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09AE7C5AD4C for ; Mon, 20 Nov 2023 18:35:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EA42F6B0419; Mon, 20 Nov 2023 13:34:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DDCCB6B040B; Mon, 20 Nov 2023 13:34:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 72A866B0406; Mon, 20 Nov 2023 13:34:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 18CE86B03AC for ; Mon, 20 Nov 2023 13:34:48 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id D5251C04A3 for ; Mon, 20 Nov 2023 18:34:47 +0000 (UTC) X-FDA: 81479183814.04.8743713 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf14.hostedemail.com (Postfix) with ESMTP id AB65A10001F for ; Mon, 20 Nov 2023 18:34:45 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=puBmaHYO; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=61j42JPK; dmarc=none; spf=pass (imf14.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700505286; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HkihGKFc7KfRcWPJtAKuLkRXW3e6zJHroeb9H0FmoFU=; b=Bh3DIj9gN13xvTq4U5n+fdGafNWKBCt8Ax4NIrcLKC+Jtovtf+wG3X9EZ5HGFCRGqG+Ao+ n71AzqKExbZ+3G7GVey7YB+PDKoeXxXKRzMeFYAIh/eYYLGxEETCTFBH9kOKKXaYyHY0wJ Z3Pvntferrf4+4F44kXG6iUX9uxf8Bs= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=suse.cz header.s=susede2_rsa header.b=puBmaHYO; dkim=pass header.d=suse.cz header.s=susede2_ed25519 header.b=61j42JPK; dmarc=none; spf=pass (imf14.hostedemail.com: domain of vbabka@suse.cz designates 195.135.220.29 as permitted sender) smtp.mailfrom=vbabka@suse.cz ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700505286; a=rsa-sha256; cv=none; b=ku2tJ0ERjWnPaVNcUI9CSVi2R9a5unka6sFZHKdF08BiiuejkVsfvrKm+GvESKU27acCVY 3WQI7MgMaOj71f24G+dAPVSHbkNYz1YmDG0BpVm8Pb+KwmbOmSJkWMu4ZL0ngxcC4Pzsoc DrZ9lho8aYVUqoZsblYBV535phu6oDs= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id 7BA801F8B0; Mon, 20 Nov 2023 18:34:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_rsa; t=1700505282; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HkihGKFc7KfRcWPJtAKuLkRXW3e6zJHroeb9H0FmoFU=; b=puBmaHYOLcQhulZIDjXI09Vo8lTbTPvOPmojSUQejrrEFaTLXdCS3Jkg1iYOVQWphuVXI0 yBCLxHR7NAQOIexwX+cU6UWapVt23UObrlMqRJwWBxcBzyNgNObd7pD7PVTp4sgRJLHEFa u2Os382UirLoEEFn1tiSPCBLBPJVkJw= DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=suse.cz; s=susede2_ed25519; t=1700505282; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HkihGKFc7KfRcWPJtAKuLkRXW3e6zJHroeb9H0FmoFU=; b=61j42JPKc2OPFeYKBJ+8Rs2/oGvqgHWV0FVnaaCdFKUgEgCcW51EtfqMnyXUrtU+S1rntl kr6P9SX6Fr7TfBBA== Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id 4A47713912; Mon, 20 Nov 2023 18:34:42 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id ANuuEcKmW2UUMgAAMHmgww (envelope-from ); Mon, 20 Nov 2023 18:34:42 +0000 From: Vlastimil Babka Date: Mon, 20 Nov 2023 19:34:24 +0100 Subject: [PATCH v2 13/21] mm/slab: move pre/post-alloc hooks from slab.h to slub.c MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <20231120-slab-remove-slab-v2-13-9c9c70177183@suse.cz> References: <20231120-slab-remove-slab-v2-0-9c9c70177183@suse.cz> In-Reply-To: <20231120-slab-remove-slab-v2-0-9c9c70177183@suse.cz> To: David Rientjes , Christoph Lameter , Pekka Enberg , Joonsoo Kim Cc: Andrew Morton , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Roman Gushchin , Andrey Ryabinin , Alexander Potapenko , Andrey Konovalov , Dmitry Vyukov , Vincenzo Frascino , Marco Elver , Johannes Weiner , Michal Hocko , Shakeel Butt , Muchun Song , Kees Cook , linux-mm@kvack.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, cgroups@vger.kernel.org, linux-hardening@vger.kernel.org, Vlastimil Babka X-Mailer: b4 0.12.4 X-Rspamd-Queue-Id: AB65A10001F X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: zab6pxy8unu35ypeof7jsfm6m8nxgop1 X-HE-Tag: 1700505285-700878 X-HE-Meta: U2FsdGVkX1/dak2u6J5EhBvMiTySnnHorHU/6w/JK6vU4gOicMe1Y4SZWNULcRDeltO56VCHTXqy+JjOcQpmOT5SOPAzFwhLEqcwqkHxyxdUqpKvQv4ngFfv11BhR1K7hZzTwD3BvjihsE6PYQ4juvNuDoU1Q1L5cFBnslo9MgRoE5/IUZX8BuQEolJOWsx6cVQ8nirnHih3K61acEMgEqBR/T30Bpd/HbZPIODfsgDwZ/Oc7YPkhvyucy/+m4ShtoBQM3U7CAVoFLbtFZUMi8J9u01ffyXc/ZSK5jXurXB5CGMkBofblUiH9WISFUj3jA8XHrctl64pLJ93r1643Y9gBKDrZ3KFL89es8PJr2vCQ9RmbIgPevWuTnLgBzZG8gFAhjm3gTkK/0CIuN7D4F2VZMRVg4ySDzePv6/h8AVtpUeplm3zlF1l6NmR7L6lfYAPMjIr4VtWVe/9tAQmOGvCZSWRtvH7tOSqjpmLVGC2p5tQgWV+QKokpS9h7HsS9S2FRZKC4vjBqC3QcNOwP9vNr049uSiljmeHlZ1tY3b1GG6ukZZFHS89mxfE9bmibDFCW8wP687FxL0PgYRiPk2RC26zQ+5u+xQ8kcNh6F+4Xam0lIaBnAh4dx3pkfuGSTM2ZI7WciNZuXpD8N1WcLDWVWJEMUFmuIIn4wn2xy5ANBilKHZpEGaKVsYT9z6AOJbBSDwKhuzDozeOL2TkGmDyc/wINmldIALD8dUl+hCB8Ttvj2lUzc9Pwsejjj643Y+rOw+BKTUwMYWDOqwZ9C5xF7yV+8mgUpUr12QRtjJX09+XoPZD5OiXKgcbcWnvCnCv3FWPM973YRnd73tYeiOOC1+6qr2WK8PDuG6FwJDkHuf0KaYHvHp5WWrmoMeplwrJ+g8PKzLVunaEu0khEHgYs+3yviYlH4DAPuakltGpXgu6mx9greAbS2YuwKGX6m6TMdkeZcnYFI5r9Nt FZMAY22/ JgXsD/qummFg82TFXf6gPRHSxiBXr703Jta8HpVMegN2yEqLkpo9BJSc2mYgTdvW4mI4ryxT79zHdLhdKzfZmP+fDhYjrnkIb2Ny9eb8wBI1q3Ovi9E1d4o3IsjFfn/7a7N8IXRemEiZPqhUPN/sKaAqcjqCNDthBuEeb8wmM+zIA7vo4AJNlbynIamOgRc1O+L9dXXzvwv7VxF67w2g+9YelpXZrECSpRWEQEx1HI7C0HpS9I9B0hP1LzLPUEaAwzKrt/TidsxT3gjOmIDV08fhR/bpp2N9bjBk0daa9aTUhNsASo/Ousim6dspWdBM9/gTYoNDiD53J5DY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We don't share the hooks between two slab implementations anymore so they can be moved away from the header. As part of the move, also move should_failslab() from slab_common.c as the pre_alloc hook uses it. This means slab.h can stop including fault-inject.h and kmemleak.h. Fix up some files that were depending on the includes transitively. Reviewed-by: Kees Cook Signed-off-by: Vlastimil Babka --- mm/kasan/report.c | 1 + mm/memcontrol.c | 1 + mm/slab.h | 72 ------------------------------------------------- mm/slab_common.c | 8 +----- mm/slub.c | 81 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 5 files changed, 84 insertions(+), 79 deletions(-) diff --git a/mm/kasan/report.c b/mm/kasan/report.c index e77facb62900..011f727bfaff 100644 --- a/mm/kasan/report.c +++ b/mm/kasan/report.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 947fb50eba31..8a0603517065 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -64,6 +64,7 @@ #include #include #include +#include #include "internal.h" #include #include diff --git a/mm/slab.h b/mm/slab.h index 1ac3a2f8d4c0..65ebf86b3fe9 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -9,8 +9,6 @@ #include #include #include -#include -#include #include #include @@ -796,76 +794,6 @@ static inline size_t slab_ksize(const struct kmem_cache *s) return s->size; } -static inline struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, - struct list_lru *lru, - struct obj_cgroup **objcgp, - size_t size, gfp_t flags) -{ - flags &= gfp_allowed_mask; - - might_alloc(flags); - - if (should_failslab(s, flags)) - return NULL; - - if (!memcg_slab_pre_alloc_hook(s, lru, objcgp, size, flags)) - return NULL; - - return s; -} - -static inline void slab_post_alloc_hook(struct kmem_cache *s, - struct obj_cgroup *objcg, gfp_t flags, - size_t size, void **p, bool init, - unsigned int orig_size) -{ - unsigned int zero_size = s->object_size; - bool kasan_init = init; - size_t i; - - flags &= gfp_allowed_mask; - - /* - * For kmalloc object, the allocated memory size(object_size) is likely - * larger than the requested size(orig_size). If redzone check is - * enabled for the extra space, don't zero it, as it will be redzoned - * soon. The redzone operation for this extra space could be seen as a - * replacement of current poisoning under certain debug option, and - * won't break other sanity checks. - */ - if (kmem_cache_debug_flags(s, SLAB_STORE_USER | SLAB_RED_ZONE) && - (s->flags & SLAB_KMALLOC)) - zero_size = orig_size; - - /* - * When slub_debug is enabled, avoid memory initialization integrated - * into KASAN and instead zero out the memory via the memset below with - * the proper size. Otherwise, KASAN might overwrite SLUB redzones and - * cause false-positive reports. This does not lead to a performance - * penalty on production builds, as slub_debug is not intended to be - * enabled there. - */ - if (__slub_debug_enabled()) - kasan_init = false; - - /* - * As memory initialization might be integrated into KASAN, - * kasan_slab_alloc and initialization memset must be - * kept together to avoid discrepancies in behavior. - * - * As p[i] might get tagged, memset and kmemleak hook come after KASAN. - */ - for (i = 0; i < size; i++) { - p[i] = kasan_slab_alloc(s, p[i], flags, kasan_init); - if (p[i] && init && (!kasan_init || !kasan_has_integrated_init())) - memset(p[i], 0, zero_size); - kmemleak_alloc_recursive(p[i], s->object_size, 1, - s->flags, flags); - kmsan_slab_alloc(s, p[i], flags); - } - - memcg_slab_post_alloc_hook(s, objcg, flags, size, p); -} /* * The slab lists for all objects. diff --git a/mm/slab_common.c b/mm/slab_common.c index 63b8411db7ce..bbc2e3f061f1 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -1470,10 +1471,3 @@ EXPORT_TRACEPOINT_SYMBOL(kmem_cache_alloc); EXPORT_TRACEPOINT_SYMBOL(kfree); EXPORT_TRACEPOINT_SYMBOL(kmem_cache_free); -int should_failslab(struct kmem_cache *s, gfp_t gfpflags) -{ - if (__should_failslab(s, gfpflags)) - return -ENOMEM; - return 0; -} -ALLOW_ERROR_INJECTION(should_failslab, ERRNO); diff --git a/mm/slub.c b/mm/slub.c index 979932d046fd..9eb6508152c2 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -34,6 +34,7 @@ #include #include #include +#include #include #include #include @@ -3494,6 +3495,86 @@ static __always_inline void maybe_wipe_obj_freeptr(struct kmem_cache *s, 0, sizeof(void *)); } +noinline int should_failslab(struct kmem_cache *s, gfp_t gfpflags) +{ + if (__should_failslab(s, gfpflags)) + return -ENOMEM; + return 0; +} +ALLOW_ERROR_INJECTION(should_failslab, ERRNO); + +static inline struct kmem_cache *slab_pre_alloc_hook(struct kmem_cache *s, + struct list_lru *lru, + struct obj_cgroup **objcgp, + size_t size, gfp_t flags) +{ + flags &= gfp_allowed_mask; + + might_alloc(flags); + + if (should_failslab(s, flags)) + return NULL; + + if (!memcg_slab_pre_alloc_hook(s, lru, objcgp, size, flags)) + return NULL; + + return s; +} + +static inline void slab_post_alloc_hook(struct kmem_cache *s, + struct obj_cgroup *objcg, gfp_t flags, + size_t size, void **p, bool init, + unsigned int orig_size) +{ + unsigned int zero_size = s->object_size; + bool kasan_init = init; + size_t i; + + flags &= gfp_allowed_mask; + + /* + * For kmalloc object, the allocated memory size(object_size) is likely + * larger than the requested size(orig_size). If redzone check is + * enabled for the extra space, don't zero it, as it will be redzoned + * soon. The redzone operation for this extra space could be seen as a + * replacement of current poisoning under certain debug option, and + * won't break other sanity checks. + */ + if (kmem_cache_debug_flags(s, SLAB_STORE_USER | SLAB_RED_ZONE) && + (s->flags & SLAB_KMALLOC)) + zero_size = orig_size; + + /* + * When slub_debug is enabled, avoid memory initialization integrated + * into KASAN and instead zero out the memory via the memset below with + * the proper size. Otherwise, KASAN might overwrite SLUB redzones and + * cause false-positive reports. This does not lead to a performance + * penalty on production builds, as slub_debug is not intended to be + * enabled there. + */ + if (__slub_debug_enabled()) + kasan_init = false; + + /* + * As memory initialization might be integrated into KASAN, + * kasan_slab_alloc and initialization memset must be + * kept together to avoid discrepancies in behavior. + * + * As p[i] might get tagged, memset and kmemleak hook come after KASAN. + */ + for (i = 0; i < size; i++) { + p[i] = kasan_slab_alloc(s, p[i], flags, kasan_init); + if (p[i] && init && (!kasan_init || + !kasan_has_integrated_init())) + memset(p[i], 0, zero_size); + kmemleak_alloc_recursive(p[i], s->object_size, 1, + s->flags, flags); + kmsan_slab_alloc(s, p[i], flags); + } + + memcg_slab_post_alloc_hook(s, objcg, flags, size, p); +} + /* * Inlined fastpath so that allocation functions (kmalloc, kmem_cache_alloc) * have the fastpath folded into their functions. So no function call -- 2.42.1