From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51DBBC47258 for ; Tue, 23 Jan 2024 15:53:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 816746B007D; Tue, 23 Jan 2024 10:53:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7C74F6B0080; Tue, 23 Jan 2024 10:53:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 667636B0082; Tue, 23 Jan 2024 10:53:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 562736B007D for ; Tue, 23 Jan 2024 10:53:23 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 118F21602DB for ; Tue, 23 Jan 2024 15:53:23 +0000 (UTC) X-FDA: 81711020286.17.DD59503 Received: from mail-wr1-f51.google.com (mail-wr1-f51.google.com [209.85.221.51]) by imf17.hostedemail.com (Postfix) with ESMTP id 0D32240006 for ; Tue, 23 Jan 2024 15:53:20 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ViNZYHeT; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of elver@google.com designates 209.85.221.51 as permitted sender) smtp.mailfrom=elver@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706025201; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wN1zYbmyQqMAelG24I+F79Wy9Nx29GGurhOHrEZCBhQ=; b=jRwy1hcmofr6knNKhkOr0Z2cwvl8w3ltJFaw7OOHh0iX97fFEPrnCH0l+hCdc+AJO99PlV PUbfTEElEYl8OEO9O/xWPNDqYLZJUu9J/gd0BVnLAG6j8GokF190jYnmTZ9jb0PdPLVmXg KQigGoF4vXYHbrE3RQ2NlGPl6ST3y70= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ViNZYHeT; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of elver@google.com designates 209.85.221.51 as permitted sender) smtp.mailfrom=elver@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706025201; a=rsa-sha256; cv=none; b=ffIWGbGBNsuQVnZu+1DKkqP/2Ao+3fn+he3U404N4dlYTue1Y+4obBLjwdwFDb+xC7RxDx WEVHC+FicnRaEcR76Wb9akOxCxWWb59RFL7zR5vSH+DgtxiXIfM85BxAXO9udlx9QeoO5/ 75ehcb33uZoTdLtaWVDbTfKIaccc4go= Received: by mail-wr1-f51.google.com with SMTP id ffacd0b85a97d-337d90030bfso3901627f8f.2 for ; Tue, 23 Jan 2024 07:53:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706025199; x=1706629999; darn=kvack.org; h=user-agent:in-reply-to:content-disposition:mime-version:references :message-id:subject:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=wN1zYbmyQqMAelG24I+F79Wy9Nx29GGurhOHrEZCBhQ=; b=ViNZYHeThLEOn4VGcCzXvPJ6hr3wJFBEX1JywG2OZpenZ6JaVRD2JO1aNLpdeyggsX w/1+C8wEsDJ0gts7xc8trapqLOKLiDFtOiJ7k6wygMOLmVeWhbNKA7iSQJ25NB9slyIv f3ujZ3rCGm/9ufmOe5S8nIcKvfUGGGa0okWJG0HYjk4e4pkOrFqGSK758ZJw9nTovJrC GuboWRuNFDHqm4fSS0Ftxisxrf9IMTvnRG67eCMEt8GXF3TB603mJhbiihCH8qhomsM3 deehN1DyJO6Rb7aWSescE14Tm3udPIVQBfc6DSv6Af67v6YFybZbrb4XHCP8uARAgXLB VKLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706025199; x=1706629999; h=user-agent:in-reply-to:content-disposition:mime-version:references :message-id:subject:cc:to:from:date:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=wN1zYbmyQqMAelG24I+F79Wy9Nx29GGurhOHrEZCBhQ=; b=fNN5qTI2RAN2W1oFbcKr6MoebGECPuOClEFmzKhHRGtC17u8cNenMg/2LwSN6rEaLn dvO98c7bdZWjOND+eKYX1upPsuVisMeBddpIYd8xO/vROie6/OZr5yzpKFWEwSdSa4YP nCA9bNDXCPYELrMqYsQTQXy01BFi2X7Xs038NQVL6tCouodIKQMfkgyKjvkgDWVpy8l6 tbRv3dhiDjoEeX3pWxC3QvSi++B7CKeeiut/QANVm0WapMmIC8D83u9y7Y4GDKLUua7h 8wulSbZtt9DDqHeDDY8KtUwL8v1rdjxf0uEA9hDm9GHdJceEDdtWSfHZ9VdM4hS2zZVR JaNA== X-Gm-Message-State: AOJu0Yzu3+9ZWAXARj3qtfeRv0/v1hof7FQUZ8AvJ5u+a6OOgfW92eRn FseFHcoSuK0qS5olJl8G04iYJSYr3OalG47BEgrwEyAr3tgJAYxcWGLUYpioFw== X-Google-Smtp-Source: AGHT+IHiMnUjdhR55qpE61cdcG9uCPMMHHLfkzGTGVTJW44PH+W1KF3mAeezHklxMdSsPZqN/669fw== X-Received: by 2002:a5d:58f4:0:b0:337:d860:b260 with SMTP id f20-20020a5d58f4000000b00337d860b260mr1500203wrd.177.1706025199197; Tue, 23 Jan 2024 07:53:19 -0800 (PST) Received: from elver.google.com ([2a00:79e0:9c:201:3ba0:aeb0:d827:1aee]) by smtp.gmail.com with ESMTPSA id df10-20020a5d5b8a000000b0033947d7651asm1391689wrb.5.2024.01.23.07.53.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 Jan 2024 07:53:18 -0800 (PST) Date: Tue, 23 Jan 2024 16:53:13 +0100 From: Marco Elver To: Andrey Konovalov Cc: Andrew Morton , Alexander Potapenko , Dmitry Vyukov , Vlastimil Babka , linux-kernel@vger.kernel.org, linux-mm@kvack.org, kasan-dev@googlegroups.com Subject: Re: [RFC PATCH] stackdepot: use variable size records for non-evictable entries Message-ID: References: <20240122171215.319440-2-elver@google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240122171215.319440-2-elver@google.com> User-Agent: Mutt/2.2.12 (2023-09-09) X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 0D32240006 X-Stat-Signature: 8hj5g7xes8wx6da15q8mupmuxa3gka8h X-HE-Tag: 1706025200-134865 X-HE-Meta: U2FsdGVkX19kl6Rl7k0pnS5xmDlsUyi71KJ743u8kAEr/21A8MOcUQTNNH/UNYs4whDzo1v78JMdPkZIQwEdUnw7emmfXAaelPpIRR2Q9TBo5VnQZPBNofShd5rshghNArqHnCSFRU3p/jPg/9bhY5U5QrHjgt5nBi11VcgkIuFQECSrQXkt8jvkz7W/BNcw5u57yd6SD6ZMgmtrcqBFawbf7Ecbqefk9BR12d5hMyyd2HDvBex3n3dG9lNct43c44IMjeIM5MQMaK0dDSqXnNGgz2Cf5m4L4meooagrc6aesJmc7km2knZfy0CyyxIqKH1+Q2YKgUCUyqmDN4VhUReChsBQg07ezEop5ncjiQrGAuDdM7JgPs5KIe2Mbh5PI1nHsU/KNx8MVZle6w/sseaASP9lzpSbC4AQl3Ntar28GDQIxz01r8E80xUjd6SuNtzDiAh69dMcye9vi23q11cqR822xM32YcS5Kl0HkJdJfE0lVEUX4mCZONYCfxSjgYMR9uxbYaZwMQIqiSXUfc+gFXjf5NY0HTPONQe6n1btDLhMg2x0c19GaRiD1oAPuOLf+yho4EIrSOWHZOwcTpEr+lK5UO5mmi5iKe9RjwkabkXUKL9MH+f3Q7sWcO6ORXEf1lspvVQJmxp8xkjhbsj9WKDd8PqEsduLLQqgXh01OqVapfC3oE7Z3NKFBjnyssFCQOjUqNCMj+DbAMV+Sy0p0W7JWo1AOqcX0NdP5Y9yGewuzXT4LyII0ZVk5Ei51berABhpV52yU3qt9nvqd6zJDstc5inyg9IaSNlylySMmpocNvR+8OP+2Off5o57GOqvZoXvozTIkQ35VzXDyVkpoA0Qs9B2uqA4pQGCT4LMV8oI2/rY3b1Ou97L3FF/bG2WvcRlDA73PJgXbZ4lHaF2i8UQvqqOLdoqF5fYbxpyLKzDji5Y4/NjxlBUR7ZdlwF9zqGeKeG2KVGjcx5 TwGOQkyw 4wcjwZRgqeQoX8DM5gRKMj9fYIwYQuehYER9X/SgCWrPm7KGlywx7U0IBtT7pjNdBt+0zH/1diHQkWZ1m3jXT4lXVJ2BLYGtTP1XR2OMmshioY/RkzKDxx8YxWbRCSWy7zwLJxzlnrjTmcsO+vasmQD5EbW2q6kHrWV3vNLZMNKIM3VTSCin3+831wN7BRkW96zjxS6Mje5Xu8wWF2ePnfLpWy3sUvqxjMIAo5Mcur12iKvRLWwjkx6XHziQRyFG4MZh36jeyQv8pdV5RwIK1ygdxK63yj2GD23C0p5YRegtDkwc1dDzAgWPBqFJ20hDvhLJS3OUwSgteNmeaNyba1PGqZuFcANlgWlXuJxa3p74rC0P865xFx8K6qbXm9DUEq9MnzJ18J56O/mrdQJO6kGaQ0ASyLaspv3a+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: And on top of this we can make KASAN generic happier again: Objections? ------ >8 ------ From: Marco Elver Date: Tue, 23 Jan 2024 12:11:36 +0100 Subject: [PATCH RFC] kasan: revert eviction of stack traces in generic mode This partially reverts commits cc478e0b6bdf, 63b85ac56a64, 08d7c94d9635, a414d4286f34, and 773688a6cb24 to make use of variable-sized stack depot records, since eviction of stack entries from stack depot forces fixed- sized stack records. Care was taken to retain the code cleanups by the above commits. Eviction was added to generic KASAN as a response to alleviating the additional memory usage from fixed-sized stack records, but this still uses more memory than previously. With the re-introduction of variable-sized records for stack depot, we can just switch back to non-evictable stack records again, and return back to the previous performance and memory usage baseline. Before (observed after a KASAN kernel boot): pools: 597 allocations: 29657 frees: 6425 in_use: 23232 freelist_size: 3493 After: pools: 315 allocations: 28964 frees: 0 in_use: 28964 freelist_size: 0 As can be seen from the number of "frees", with a generic KASAN config, evictions are no longer used but due to using variable-sized records, I observe a reduction of 282 stack depot pools (saving 4512 KiB). Fixes: cc478e0b6bdf ("kasan: avoid resetting aux_lock") Fixes: 63b85ac56a64 ("kasan: stop leaking stack trace handles") Fixes: 08d7c94d9635 ("kasan: memset free track in qlink_free") Fixes: a414d4286f34 ("kasan: handle concurrent kasan_record_aux_stack calls") Fixes: 773688a6cb24 ("kasan: use stack_depot_put for Generic mode") Signed-off-by: Marco Elver Cc: Alexander Potapenko Cc: Andrey Konovalov Cc: Dmitry Vyukov --- mm/kasan/common.c | 3 +-- mm/kasan/generic.c | 54 ++++++---------------------------------------- mm/kasan/kasan.h | 8 ------- 3 files changed, 8 insertions(+), 57 deletions(-) diff --git a/mm/kasan/common.c b/mm/kasan/common.c index 610efae91220..ad32803e34e9 100644 --- a/mm/kasan/common.c +++ b/mm/kasan/common.c @@ -65,8 +65,7 @@ void kasan_save_track(struct kasan_track *track, gfp_t flags) { depot_stack_handle_t stack; - stack = kasan_save_stack(flags, - STACK_DEPOT_FLAG_CAN_ALLOC | STACK_DEPOT_FLAG_GET); + stack = kasan_save_stack(flags, STACK_DEPOT_FLAG_CAN_ALLOC); kasan_set_track(track, stack); } diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c index df6627f62402..8bfb52b28c22 100644 --- a/mm/kasan/generic.c +++ b/mm/kasan/generic.c @@ -485,16 +485,6 @@ void kasan_init_object_meta(struct kmem_cache *cache, const void *object) if (alloc_meta) { /* Zero out alloc meta to mark it as invalid. */ __memset(alloc_meta, 0, sizeof(*alloc_meta)); - - /* - * Prepare the lock for saving auxiliary stack traces. - * Temporarily disable KASAN bug reporting to allow instrumented - * raw_spin_lock_init to access aux_lock, which resides inside - * of a redzone. - */ - kasan_disable_current(); - raw_spin_lock_init(&alloc_meta->aux_lock); - kasan_enable_current(); } /* @@ -506,18 +496,8 @@ void kasan_init_object_meta(struct kmem_cache *cache, const void *object) static void release_alloc_meta(struct kasan_alloc_meta *meta) { - /* Evict the stack traces from stack depot. */ - stack_depot_put(meta->alloc_track.stack); - stack_depot_put(meta->aux_stack[0]); - stack_depot_put(meta->aux_stack[1]); - - /* - * Zero out alloc meta to mark it as invalid but keep aux_lock - * initialized to avoid having to reinitialize it when another object - * is allocated in the same slot. - */ - __memset(&meta->alloc_track, 0, sizeof(meta->alloc_track)); - __memset(meta->aux_stack, 0, sizeof(meta->aux_stack)); + /* Zero out alloc meta to mark it as invalid. */ + __memset(meta, 0, sizeof(*meta)); } static void release_free_meta(const void *object, struct kasan_free_meta *meta) @@ -526,9 +506,6 @@ static void release_free_meta(const void *object, struct kasan_free_meta *meta) if (*(u8 *)kasan_mem_to_shadow(object) != KASAN_SLAB_FREE_META) return; - /* Evict the stack trace from the stack depot. */ - stack_depot_put(meta->free_track.stack); - /* Mark free meta as invalid. */ *(u8 *)kasan_mem_to_shadow(object) = KASAN_SLAB_FREE; } @@ -571,8 +548,6 @@ static void __kasan_record_aux_stack(void *addr, depot_flags_t depot_flags) struct kmem_cache *cache; struct kasan_alloc_meta *alloc_meta; void *object; - depot_stack_handle_t new_handle, old_handle; - unsigned long flags; if (is_kfence_address(addr) || !slab) return; @@ -583,33 +558,18 @@ static void __kasan_record_aux_stack(void *addr, depot_flags_t depot_flags) if (!alloc_meta) return; - new_handle = kasan_save_stack(0, depot_flags); - - /* - * Temporarily disable KASAN bug reporting to allow instrumented - * spinlock functions to access aux_lock, which resides inside of a - * redzone. - */ - kasan_disable_current(); - raw_spin_lock_irqsave(&alloc_meta->aux_lock, flags); - old_handle = alloc_meta->aux_stack[1]; alloc_meta->aux_stack[1] = alloc_meta->aux_stack[0]; - alloc_meta->aux_stack[0] = new_handle; - raw_spin_unlock_irqrestore(&alloc_meta->aux_lock, flags); - kasan_enable_current(); - - stack_depot_put(old_handle); + alloc_meta->aux_stack[0] = kasan_save_stack(0, depot_flags); } void kasan_record_aux_stack(void *addr) { - return __kasan_record_aux_stack(addr, - STACK_DEPOT_FLAG_CAN_ALLOC | STACK_DEPOT_FLAG_GET); + return __kasan_record_aux_stack(addr, STACK_DEPOT_FLAG_CAN_ALLOC); } void kasan_record_aux_stack_noalloc(void *addr) { - return __kasan_record_aux_stack(addr, STACK_DEPOT_FLAG_GET); + return __kasan_record_aux_stack(addr, 0); } void kasan_save_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags) @@ -620,7 +580,7 @@ void kasan_save_alloc_info(struct kmem_cache *cache, void *object, gfp_t flags) if (!alloc_meta) return; - /* Evict previous stack traces (might exist for krealloc or mempool). */ + /* Invalidate previous stack traces (might exist for krealloc or mempool). */ release_alloc_meta(alloc_meta); kasan_save_track(&alloc_meta->alloc_track, flags); @@ -634,7 +594,7 @@ void kasan_save_free_info(struct kmem_cache *cache, void *object) if (!free_meta) return; - /* Evict previous stack trace (might exist for mempool). */ + /* Invalidate previous stack trace (might exist for mempool). */ release_free_meta(object, free_meta); kasan_save_track(&free_meta->free_track, 0); diff --git a/mm/kasan/kasan.h b/mm/kasan/kasan.h index d0f172f2b978..216ae0ef1e4b 100644 --- a/mm/kasan/kasan.h +++ b/mm/kasan/kasan.h @@ -6,7 +6,6 @@ #include #include #include -#include #include #if defined(CONFIG_KASAN_SW_TAGS) || defined(CONFIG_KASAN_HW_TAGS) @@ -265,13 +264,6 @@ struct kasan_global { struct kasan_alloc_meta { struct kasan_track alloc_track; /* Free track is stored in kasan_free_meta. */ - /* - * aux_lock protects aux_stack from accesses from concurrent - * kasan_record_aux_stack calls. It is a raw spinlock to avoid sleeping - * on RT kernels, as kasan_record_aux_stack_noalloc can be called from - * non-sleepable contexts. - */ - raw_spinlock_t aux_lock; depot_stack_handle_t aux_stack[2]; }; -- 2.43.0.429.g432eaa2c6b-goog