From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CD46C4167B for ; Wed, 29 Nov 2023 12:00:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B7A446B03CA; Wed, 29 Nov 2023 07:00:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B02B96B03CD; Wed, 29 Nov 2023 07:00:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A2D86B03CE; Wed, 29 Nov 2023 07:00:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 85D726B03CA for ; Wed, 29 Nov 2023 07:00:19 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 45454140505 for ; Wed, 29 Nov 2023 12:00:19 +0000 (UTC) X-FDA: 81510848958.29.18F49B5 Received: from mail-ua1-f43.google.com (mail-ua1-f43.google.com [209.85.222.43]) by imf20.hostedemail.com (Postfix) with ESMTP id 341A51C0038 for ; Wed, 29 Nov 2023 12:00:16 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=e3KMsi8X; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of elver@google.com designates 209.85.222.43 as permitted sender) smtp.mailfrom=elver@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701259216; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nPXPu1JYkKeWx1IZ2D0WOSznovDcK9NZrhDXsB39l8g=; b=GCLtAO2rGMQnGuDi6lAJk1vxP25qfkAB7TM9PFvHfkPcYLhl2knSai038HTIkRdNdBguCj Y4KImCsTbf060rfoCaS3NvjcwAzzRdnwhbozoUD9kpMt2ozRlzBwV41dlBbrHUVHrp5s3Y tchUk418AckzSahOWXwJoRXPbEnfQYc= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=e3KMsi8X; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of elver@google.com designates 209.85.222.43 as permitted sender) smtp.mailfrom=elver@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701259216; a=rsa-sha256; cv=none; b=mWTGRSAtBSx9t14FrkXUq6KCXQr+ALj90z+AgRW/ouYCp1pRxqKGF0JJN3PlLfZ48UKO/b ZHTBOwugTL6HE6SWQbrj/H1dlOx9fZD9J0CItZIXG4zxnz6jZCvjYxv9ZSBkKWzlwHZxDM dz3Q0PLCwugmpYZIBflY4QWPY1AEBcs= Received: by mail-ua1-f43.google.com with SMTP id a1e0cc1a2514c-7bb3e55c120so1953186241.0 for ; Wed, 29 Nov 2023 04:00:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1701259215; x=1701864015; darn=kvack.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=nPXPu1JYkKeWx1IZ2D0WOSznovDcK9NZrhDXsB39l8g=; b=e3KMsi8XvL3gZ569qLP7eyNKeNH9rQd2LFHyudDPiDNa4pHhjt7cA7vKCQeqEMRQr5 8sqa82y5JnaBrwjinn42RouO/x/XeeYyIzUniA4Hie1It405jaeosdLQk7J90xkEb49n HZ1N6tZuB6J/OwsJKWv4kxumzlG1inSQD3pbUDnrDDef7tOxs4MhYF1KgbGvwquvgAdJ qg2oPuhjlzBx2P5eNC/5XxNvZ8bRn3itDfmu9pauuPUSNMFnjKBFEVNFvy4a44RxsgMd uguvKkJnVe+aKRpGT56kDE8cj895NMbnbSMdputSsuBQESagb3oK5Lj0C+FlzFXEbCGI nMog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701259215; x=1701864015; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=nPXPu1JYkKeWx1IZ2D0WOSznovDcK9NZrhDXsB39l8g=; b=JYVIwoVymn3A7VU/1TCuFdSk1f03NKD++aroh35m1P+Cn3cjbs/Myrnqj94TG/WAyW sCGfGKJlHB1B8hU7okwwzbQsNjknfiQVECPvyaypj0q/oIf/hQVhceTXCRULWhlcZysJ HsP/az19w73hQXa8qp4YJ2HWi4Nr2Twwo3PFhPJcaXwQ0zRg7GAWUmSBmTToVyN8GSpC v+oLiADmQDA4FdQx53/Qz09Kuh/mlCsZvxjEWpvTw3SND8uVlZxefwGUr5JcLUp2Z/CN kG0Ll3HoN+H+niRqRhX5XQxXKHHdxXez8DoOvJs2abnwfAoisULzKBO99wuBYZDyCaQU Zixw== X-Gm-Message-State: AOJu0YwA/vC+sQR7FKOmiHXEjj5nsm5cKH3A9PiregPUjsAWFNKoH5g2 ScCKcDXWc9Q2eynmzuOnS7jeqhrPSiF96ap1AMNiNg== X-Google-Smtp-Source: AGHT+IHz+oE856HvMRcbIiiG9E1pi1IF2Nd8EHuvXGH7ld+DwtyNpIQdgPF2bLuhhYBWoK5fJanh8MVk+4DgexwHbHw= X-Received: by 2002:a05:6102:5108:b0:460:621c:d14b with SMTP id bm8-20020a056102510800b00460621cd14bmr20684204vsb.20.1701259215088; Wed, 29 Nov 2023 04:00:15 -0800 (PST) MIME-Version: 1.0 References: <20231129-slub-percpu-caches-v3-0-6bcf536772bc@suse.cz> <20231129-slub-percpu-caches-v3-4-6bcf536772bc@suse.cz> In-Reply-To: <20231129-slub-percpu-caches-v3-4-6bcf536772bc@suse.cz> From: Marco Elver Date: Wed, 29 Nov 2023 13:00:00 +0100 Message-ID: Subject: Re: [PATCH RFC v3 4/9] mm/slub: free KFENCE objects in slab_free_hook() To: Vlastimil Babka Cc: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Matthew Wilcox , "Liam R. Howlett" , Andrew Morton , Roman Gushchin , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Alexander Potapenko , Dmitry Vyukov , linux-mm@kvack.org, linux-kernel@vger.kernel.org, maple-tree@lists.infradead.org, kasan-dev@googlegroups.com Content-Type: text/plain; charset="UTF-8" X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 341A51C0038 X-Stat-Signature: tgn8jmq58jhcdwuqedw8knccw1zyqerz X-HE-Tag: 1701259216-972679 X-HE-Meta: U2FsdGVkX1/JrnfbSQhrw8p1svH8cmEIQKPCpm2K8D1XelMEbxSf/CKyOpXeMnRDiSm7ceMNMeuVPnkxelLpdA0apqU1Nu7WI3bRzpiJVGOPiDb9nGAQBFGsAO5ec1JouwoQl2tCfbBdeShx66VtsISMAN+swhH5DDeTbl4rodRL9/tPZzjcKHmJlR4Nq+9IGhipueUC4U4GnOswPbZlCUIcWSKV9OO0zsPaKMNTKNqiIuUT9CsNPYf0ohk4W3SH7fW3O/0tiUS00cCzd6JirmtVwAiAUvGInUqtGNYAa/NRlBCKmXnECmW9t3BXWoNxVWlYJYQcqjiGRIwIVOmJ82aXNlJZ0jWaWtPiGroVg7x3Wed8vA6Sq8/J1LatO8DTi2ZoACqQQOVqntT3a98Qq2YlckjU/6dTrc0hJG10fqQXaMA/SQkdL0oaTdL0K9nMM1EeOM5cmuH1OkjBYd1Xs99tSU+DQJ3xJKw9dwVWi8rdaAGFj9jiGfAreNEt2N6vUZlTanFpYK6gulf2vHDQUI2fgQdpT4gspqYLvyEsP2+3QcEqTsNXg6uss3xStE8awM3rOLkcJLfBSg2bE72+oK55Ukr9enxF8VADKao3XYb8eBmjDAu7+JjbkXlkjZu9Sk0DFWzuqBWRhZLlnyiDO3wJ0OK8BGZnk5GqDSkTz00OmMNILbLvaQXmdYNSF1KbId6bphUMy6Nf1g6NVHnmboYRlGQRE0p1Bd66Fyan9gLlSWZAMNDa2Qh1j6PiP8py3JtUZtTdIGAF2Yj4a53kZwxRJYuEZ9RfK9hUZHMvnV5RXAKMzjiNDuzOR8gwDWEJWEO2c7H75H5Rt6BiZlzZsZg1K/xy5mvjY/Ux79kVm82EgwjZsW60FrWCtPDlkldQH0U5nAifdK37kig9WFzWRfzS0RFeIpyGut6TOd2jDRb730YIYnmbdFAispYDY87HqASMNA9pKkN7VrjUKnE GqHXp+17 e9VVSES14CiXidO3ZxtbDFJtrj3lyA4jGfLslQ0APE0JeCkE9nguGUzvztS2DlERo8ypZh4R5JhECjZ5yU604FPlb1bewwzJUIefFotvXEqf6Hv1IUh+fslIXG6m4nCh60uQ6FF8yojxuvh6V/7SrOBh34eMHtVdSJ8MgHxSo6jqiiIE6j8whUFWg14cE4VFysGjen0bfo+fl5eQxYi6VYQfYWfEsk98UTV/tku4eS4thsOOnyMBI5V7pxTcxuFA3YdbQqZN2Pn7KPy4gN1HcbSEU6joSTvGvZvoQXQ/044Sky+GrhD/4b90NLECdTIyIWN54+Ki5xolVTUehDrMlW36vmxzD37AIhIOH3Hq+jGlq3NDg5GOksurGtg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, 29 Nov 2023 at 10:53, Vlastimil Babka wrote: > > When freeing an object that was allocated from KFENCE, we do that in the > slowpath __slab_free(), relying on the fact that KFENCE "slab" cannot be > the cpu slab, so the fastpath has to fallback to the slowpath. > > This optimization doesn't help much though, because is_kfence_address() > is checked earlier anyway during the free hook processing or detached > freelist building. Thus we can simplify the code by making the > slab_free_hook() free the KFENCE object immediately, similarly to KASAN > quarantine. > > In slab_free_hook() we can place kfence_free() above init processing, as > callers have been making sure to set init to false for KFENCE objects. > This simplifies slab_free(). This places it also above kasan_slab_free() > which is ok as that skips KFENCE objects anyway. > > While at it also determine the init value in slab_free_freelist_hook() > outside of the loop. > > This change will also make introducing per cpu array caches easier. > > Signed-off-by: Vlastimil Babka Tested-by: Marco Elver > --- > mm/slub.c | 21 ++++++++++----------- > 1 file changed, 10 insertions(+), 11 deletions(-) > > diff --git a/mm/slub.c b/mm/slub.c > index 7d23f10d42e6..59912a376c6d 100644 > --- a/mm/slub.c > +++ b/mm/slub.c > @@ -1772,7 +1772,7 @@ static bool freelist_corrupted(struct kmem_cache *s, struct slab *slab, > * production configuration these hooks all should produce no code at all. > * > * Returns true if freeing of the object can proceed, false if its reuse > - * was delayed by KASAN quarantine. > + * was delayed by KASAN quarantine, or it was returned to KFENCE. > */ > static __always_inline > bool slab_free_hook(struct kmem_cache *s, void *x, bool init) > @@ -1790,6 +1790,9 @@ bool slab_free_hook(struct kmem_cache *s, void *x, bool init) > __kcsan_check_access(x, s->object_size, > KCSAN_ACCESS_WRITE | KCSAN_ACCESS_ASSERT); > > + if (kfence_free(kasan_reset_tag(x))) > + return false; > + > /* > * As memory initialization might be integrated into KASAN, > * kasan_slab_free and initialization memset's must be > @@ -1819,22 +1822,25 @@ static inline bool slab_free_freelist_hook(struct kmem_cache *s, > void *object; > void *next = *head; > void *old_tail = *tail; > + bool init; > > if (is_kfence_address(next)) { > slab_free_hook(s, next, false); > - return true; > + return false; > } > > /* Head and tail of the reconstructed freelist */ > *head = NULL; > *tail = NULL; > > + init = slab_want_init_on_free(s); > + > do { > object = next; > next = get_freepointer(s, object); > > /* If object's reuse doesn't have to be delayed */ > - if (slab_free_hook(s, object, slab_want_init_on_free(s))) { > + if (slab_free_hook(s, object, init)) { > /* Move object to the new freelist */ > set_freepointer(s, object, *head); > *head = object; > @@ -3619,9 +3625,6 @@ static void __slab_free(struct kmem_cache *s, struct slab *slab, > > stat(s, FREE_SLOWPATH); > > - if (kfence_free(head)) > - return; > - > if (IS_ENABLED(CONFIG_SLUB_TINY) || kmem_cache_debug(s)) { > free_to_partial_list(s, slab, head, tail, cnt, addr); > return; > @@ -3806,13 +3809,9 @@ static __fastpath_inline > void slab_free(struct kmem_cache *s, struct slab *slab, void *object, > unsigned long addr) > { > - bool init; > - > memcg_slab_free_hook(s, slab, &object, 1); > > - init = !is_kfence_address(object) && slab_want_init_on_free(s); > - > - if (likely(slab_free_hook(s, object, init))) > + if (likely(slab_free_hook(s, object, slab_want_init_on_free(s)))) > do_slab_free(s, slab, object, object, 1, addr); > } > > > -- > 2.43.0 >