From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF492EE6457 for ; Fri, 15 Sep 2023 10:59:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4CD466B0354; Fri, 15 Sep 2023 06:59:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4547C8D001A; Fri, 15 Sep 2023 06:59:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2CF948D0012; Fri, 15 Sep 2023 06:59:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 156866B0354 for ; Fri, 15 Sep 2023 06:59:59 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id DA51512102A for ; Fri, 15 Sep 2023 10:59:58 +0000 (UTC) X-FDA: 81238536876.16.4148E77 Received: from mail-ed1-f74.google.com (mail-ed1-f74.google.com [209.85.208.74]) by imf13.hostedemail.com (Postfix) with ESMTP id 06DB120033 for ; Fri, 15 Sep 2023 10:59:56 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=btmvYX1t; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3KzkEZQsKCNwK8RRCMPGXXMEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--matteorizzo.bounces.google.com designates 209.85.208.74 as permitted sender) smtp.mailfrom=3KzkEZQsKCNwK8RRCMPGXXMEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--matteorizzo.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694775597; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rOse4MLSxLAQnwPck5JwfSnTM0/2w8+aiHpu8xQSJg0=; b=XsLiT6DLeugZnWf3I+rbAB1iHpRHwZTXgj6xCxsky8WVEM7GFZGOWvZArsoKJRtffWpPzd YpY6CWy3EtDHavuDgVOMreWEacqp6V/mfup+uh29Iv4urL8sQ9ufmwt7+kr4Ov6pKI0Oqv cRmJ2BN30pRSF0aJyIIX3ASAgwzzeJM= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=btmvYX1t; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf13.hostedemail.com: domain of 3KzkEZQsKCNwK8RRCMPGXXMEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--matteorizzo.bounces.google.com designates 209.85.208.74 as permitted sender) smtp.mailfrom=3KzkEZQsKCNwK8RRCMPGXXMEMMEJC.AMKJGLSV-KKIT8AI.MPE@flex--matteorizzo.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694775597; a=rsa-sha256; cv=none; b=UtiyXnsDpqVqoMSrnLUsQgP5gxBMRqAitj1+QblEYZAhopbXePi06Rltd1ZI0C7Ds/eD2v RCrAvvo2XQ7GRA9smV7F+qZQb5tVMo0qbAC0Th7B4CKq1slFncbn/lZ7w5g/CLVK4+yut1 VOlB958ujTQPgPWe7wmOMfjV5GUevVs= Received: by mail-ed1-f74.google.com with SMTP id 4fb4d7f45d1cf-50bf847b267so1435762a12.3 for ; Fri, 15 Sep 2023 03:59:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694775596; x=1695380396; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rOse4MLSxLAQnwPck5JwfSnTM0/2w8+aiHpu8xQSJg0=; b=btmvYX1t05T5K21FT0fCHjilWh9bRux6V0mJSCMnYdxGqvx0FqpPxHj699mcYgMvA9 w1+J/v8TeNZ/wYA7x3kQmHjyiIcTRFUpCsQptdDuYsp+orhhfGAH08AaPyfUhBvu3ipv OisS+Ky52+IEOrEFPiiRN9AD/JqQHn7FiFesdCLVKOwaaZK8qdhvo9BS+8u7cuN+cvKA V9PWXMg7CicSw3MgnFSBXECGVzVg7LuSNsfWQ19Ch12PHAF0RSGCdeLrX1j2HHZdXa1L uh1QPunWDVNcsDLI0uSzAyfVz+NQignhwH5M/iQOP8lNh4y7Vm5WALKmiRuYC/9u9eUY DL5Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694775596; x=1695380396; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rOse4MLSxLAQnwPck5JwfSnTM0/2w8+aiHpu8xQSJg0=; b=EtQqOVSEnGvJBK7Qsb9oN/OiMQ2oOBt4XhsdHtO0DZWgt6LR50edGxwhVwuIg139dt 4QFrYtdGoCr4JsmL9ERZ61s/uNAcxsOQFvxKKIjSXPwZajMUUZF0crCcyIE45Ba0bbK1 Ec+ttgBMArCgepZUDg68WYQl/TbsUYraVwyjrAydXuKJdjGuHOdeBof3xFEuIx+G9ULM 6EjvZ1cHH7656fPj7vraqtu0ZQk8IBEfmGBx603LV7zKp8oB9nz1DH8fRqLpX5DqL0T3 orbIM2VkBwS8SV0ugk2sXBYDSX1359LzgOOpllOSfNRlLY7H4yolTYu3BmMZd1KH5arX 2rTw== X-Gm-Message-State: AOJu0YxwOQPZlSRuINIMyNaT4065oBFms5VU2KWcR0FucL6xWTjppBtk d4UKuf6TUwHywvCGydPtDHEKC74H3C8tZpGWdA== X-Google-Smtp-Source: AGHT+IFLNNO7nLlngx6UR+g98LJngeXta2Ps2Exu4HntuRM5obt3GthiOTu6TJ2m43o6UrVjvgY27QRf/WNPQWcdWw== X-Received: from mr-cloudtop2.c.googlers.com ([fda3:e722:ac3:cc00:31:98fb:c0a8:2a6]) (user=matteorizzo job=sendgmr) by 2002:a50:baa4:0:b0:525:442c:2e5d with SMTP id x33-20020a50baa4000000b00525442c2e5dmr9427ede.6.1694775595758; Fri, 15 Sep 2023 03:59:55 -0700 (PDT) Date: Fri, 15 Sep 2023 10:59:26 +0000 In-Reply-To: <20230915105933.495735-1-matteorizzo@google.com> Mime-Version: 1.0 References: <20230915105933.495735-1-matteorizzo@google.com> X-Mailer: git-send-email 2.42.0.459.ge4e396fd5e-goog Message-ID: <20230915105933.495735-8-matteorizzo@google.com> Subject: [RFC PATCH 07/14] mm/slub: pass slab pointer to the freeptr decode helper From: Matteo Rizzo To: cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com, keescook@chromium.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-mm@kvack.org, linux-hardening@vger.kernel.org, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, corbet@lwn.net, luto@kernel.org, peterz@infradead.org Cc: jannh@google.com, matteorizzo@google.com, evn@google.com, poprdi@google.com, jordyzomer@google.com Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 06DB120033 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: ysfbdc7fracdtzk1z3b6pbhwaqgjcua8 X-HE-Tag: 1694775596-546248 X-HE-Meta: U2FsdGVkX1+VN2Bp/q/G9cmEyU0L3pl5VZvHQ4dhIZFuTMKdF8cWkKbIKyskx4qW8FbJQi/jcBu/E0bgKwOXmUmY+HYK52xIDcGOpePgVidqQkbSXFpA2EqsnZbrXw7HXwFyJKsPAVOSqNuSBf6LyDlvFojavm4wMyVPxttr7EsX2s143qdzrIBEj6HqXcKrGXT6mx4rIKNsxe0Dp9CZVlKUb+Mtd1ktDbPSab0MQ+2yvDW9ENdqlrczo3gGNXHR0H8y3POZ8ihg/pJE3+09EUxBfXfWw2seEwxSu9hfM/nW85MsqxQrduHemj/cKJD4Gezh7WBllDQzWDDuDA+V7B69gkFI/nRwGJlkNSKvOQbzMwck4kMNCFLnmmT+t7Iq4lwcSjhCmYZ4mg4W2qkk7FlesJorme5rHf/awDwJCV3eYmQLy0ouZRm3i+3Ldoy/6XxgeY1IjhI1Rnfldj1wsttERDDtOho5ijNTx09beNoGKuHGYybMziFB19BrcdBD1gq0+Vln5YwHWe4IwXgqtqPg+3rKZfwmkI3Ugw5JCvYZ8mPU712qI7LULwd/D0quPzGnNcxJD445BiGrKvJ3CuX+CHx46c5+3rcv2bGnng4NWDIEDRNX+6tmZiAwm0Lu3SpwT13lo6yAOfDWd5RWpM2x4L+Xz+7eKEsAZESMl7CnJHvpN4F0wH1igLceFP70+t1N+eE8XbzKGHtlx8cSYNOPjRJPRpfZ06yrNHQ+0fz93Lb6mzhJNGjoxhstxM1+eTNOXADG9AjY/04FtXYx8D+cTBy0qGRzoEfWhi5EkOjFw/ab75Psa2UtFgKLdA8UjhK8C01RJk1n9kvlkYatnc5gh97v2HvwCnuThKFxwzyBH7xN2TsHrZ3V92YXHcCAv7wad8bAGanygHTlUuVldUcW2OqKqfZYMzx2kfInJOjkGUMLUeRJy+joba0kjKBv0FlembXktFqNEe5ZkvD BypsyjmU TSJlKranBLj9568L2AE3F9m2lBaIGRN10SMLZ4bms2tieJcmrjc0XBcxrk45ThItkOnPbbXeRHPCBc7kn6wPlYOQHNIfIbVh+ldSnl7Vu33EUQSBXU28tFKdUg2M15kBm4NYJk6d0vfCeE/3ns/R04aj2BmJAqa8yqL8XGEMZ1lTEI2FuGoRC+VA82B+6KvzEfUzXzzUdQBaN+Y4QCb+pG29Tkr6ntbi8Pe+uPpeJNiQ0tmABxq7xgDxMTXDIgXXoN7h6czMCdZ7tvrldRRqADSm/qpQEszlHOTWAoCsR6axL7gXjDDgXF7doVI4US+jCxPFTHh3jLf1bcv7vHnF0qTjcP1tPy4cMCMtXiqIU/yW27cdEsUE23325tHKNgUSBVfhxV+scvL5qHgrjTfXf30crnCiOYK914WrUjEH/D+WR1byZ8HQRKfaQRZCYmZ5pT57/Bf3uK7pgg/2rJiEpIEg0FB9rApJnEFrsPwbAh6Z6Cs4ZKyaEj7NLMARKUyLyRgl9otvcKzpXB8zqsC3fOLvwGDhNgQKuZzZQgkmb9uStuEhGQVxua1qo+ggZc3qfYmbiIwtOwrrVvoLbADyEoA586OMasb9jEAnfxuGP8lqdyXK1YIdnPtK75WYUavdp64um X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Jann Horn This is refactoring in preparation for checking freeptrs for corruption inside freelist_ptr_decode(). Signed-off-by: Jann Horn Co-developed-by: Matteo Rizzo Signed-off-by: Matteo Rizzo --- mm/slub.c | 43 +++++++++++++++++++++++-------------------- 1 file changed, 23 insertions(+), 20 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index eaa1256aff89..42e7cc0b4452 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -383,7 +383,8 @@ static inline freeptr_t freelist_ptr_encode(const struct kmem_cache *s, } static inline void *freelist_ptr_decode(const struct kmem_cache *s, - freeptr_t ptr, unsigned long ptr_addr) + freeptr_t ptr, unsigned long ptr_addr, + struct slab *slab) { void *decoded; @@ -395,7 +396,8 @@ static inline void *freelist_ptr_decode(const struct kmem_cache *s, return decoded; } -static inline void *get_freepointer(struct kmem_cache *s, void *object) +static inline void *get_freepointer(struct kmem_cache *s, void *object, + struct slab *slab) { unsigned long ptr_addr; freeptr_t p; @@ -403,7 +405,7 @@ static inline void *get_freepointer(struct kmem_cache *s, void *object) object = kasan_reset_tag(object); ptr_addr = (unsigned long)object + s->offset; p = *(freeptr_t *)(ptr_addr); - return freelist_ptr_decode(s, p, ptr_addr); + return freelist_ptr_decode(s, p, ptr_addr, slab); } #ifndef CONFIG_SLUB_TINY @@ -424,18 +426,19 @@ static void prefetch_freepointer(const struct kmem_cache *s, void *object) * get_freepointer_safe() returns initialized memory. */ __no_kmsan_checks -static inline void *get_freepointer_safe(struct kmem_cache *s, void *object) +static inline void *get_freepointer_safe(struct kmem_cache *s, void *object, + struct slab *slab) { unsigned long freepointer_addr; freeptr_t p; if (!debug_pagealloc_enabled_static()) - return get_freepointer(s, object); + return get_freepointer(s, object, slab); object = kasan_reset_tag(object); freepointer_addr = (unsigned long)object + s->offset; copy_from_kernel_nofault(&p, (freeptr_t *)freepointer_addr, sizeof(p)); - return freelist_ptr_decode(s, p, freepointer_addr); + return freelist_ptr_decode(s, p, freepointer_addr, slab); } static inline void set_freepointer(struct kmem_cache *s, void *object, void *fp) @@ -627,7 +630,7 @@ static void __fill_map(unsigned long *obj_map, struct kmem_cache *s, bitmap_zero(obj_map, slab->objects); - for (p = slab->freelist; p; p = get_freepointer(s, p)) + for (p = slab->freelist; p; p = get_freepointer(s, p, slab)) set_bit(__obj_to_index(s, addr, p), obj_map); } @@ -937,7 +940,7 @@ static void print_trailer(struct kmem_cache *s, struct slab *slab, u8 *p) print_slab_info(slab); pr_err("Object 0x%p @offset=%tu fp=0x%p\n\n", - p, p - addr, get_freepointer(s, p)); + p, p - addr, get_freepointer(s, p, slab)); if (s->flags & SLAB_RED_ZONE) print_section(KERN_ERR, "Redzone ", p - s->red_left_pad, @@ -1230,7 +1233,7 @@ static int check_object(struct kmem_cache *s, struct slab *slab, return 1; /* Check free pointer validity */ - if (!check_valid_pointer(s, slab, get_freepointer(s, p))) { + if (!check_valid_pointer(s, slab, get_freepointer(s, p, slab))) { object_err(s, slab, p, "Freepointer corrupt"); /* * No choice but to zap it and thus lose the remainder @@ -1298,7 +1301,7 @@ static int on_freelist(struct kmem_cache *s, struct slab *slab, void *search) break; } object = fp; - fp = get_freepointer(s, object); + fp = get_freepointer(s, object, slab); nr++; } @@ -1810,7 +1813,7 @@ static inline bool slab_free_freelist_hook(struct kmem_cache *s, object = next; /* Single objects don't actually contain a freepointer */ if (object != old_tail) - next = get_freepointer(s, object); + next = get_freepointer(s, object, virt_to_slab(object)); /* If object's reuse doesn't have to be delayed */ if (!slab_free_hook(s, object, slab_want_init_on_free(s))) { @@ -2161,7 +2164,7 @@ static void *alloc_single_from_partial(struct kmem_cache *s, lockdep_assert_held(&n->list_lock); object = slab->freelist; - slab->freelist = get_freepointer(s, object); + slab->freelist = get_freepointer(s, object, slab); slab->inuse++; if (!alloc_debug_processing(s, slab, object, orig_size)) { @@ -2192,7 +2195,7 @@ static void *alloc_single_from_new_slab(struct kmem_cache *s, object = slab->freelist; - slab->freelist = get_freepointer(s, object); + slab->freelist = get_freepointer(s, object, slab); slab->inuse = 1; if (!alloc_debug_processing(s, slab, object, orig_size)) @@ -2517,7 +2520,7 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab, freelist_tail = NULL; freelist_iter = freelist; while (freelist_iter) { - nextfree = get_freepointer(s, freelist_iter); + nextfree = get_freepointer(s, freelist_iter, slab); /* * If 'nextfree' is invalid, it is possible that the object at @@ -2944,7 +2947,7 @@ static inline bool free_debug_processing(struct kmem_cache *s, /* Reached end of constructed freelist yet? */ if (object != tail) { - object = get_freepointer(s, object); + object = get_freepointer(s, object, slab); goto next_object; } checks_ok = true; @@ -3173,7 +3176,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, * That slab must be frozen for per cpu allocations to work. */ VM_BUG_ON(!c->slab->frozen); - c->freelist = get_freepointer(s, freelist); + c->freelist = get_freepointer(s, freelist, c->slab); c->tid = next_tid(c->tid); local_unlock_irqrestore(&s->cpu_slab->lock, flags); return freelist; @@ -3275,7 +3278,7 @@ static void *___slab_alloc(struct kmem_cache *s, gfp_t gfpflags, int node, * For !pfmemalloc_match() case we don't load freelist so that * we don't make further mismatched allocations easier. */ - deactivate_slab(s, slab, get_freepointer(s, freelist)); + deactivate_slab(s, slab, get_freepointer(s, freelist, slab)); return freelist; } @@ -3377,7 +3380,7 @@ static __always_inline void *__slab_alloc_node(struct kmem_cache *s, unlikely(!object || !slab || !node_match(slab, node))) { object = __slab_alloc(s, gfpflags, node, addr, c, orig_size); } else { - void *next_object = get_freepointer_safe(s, object); + void *next_object = get_freepointer_safe(s, object, slab); /* * The cmpxchg will only match if there was no additional @@ -3984,7 +3987,7 @@ static inline int __kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, continue; /* goto for-loop */ } - c->freelist = get_freepointer(s, object); + c->freelist = get_freepointer(s, object, c->slab); p[i] = object; maybe_wipe_obj_freeptr(s, p[i]); } @@ -4275,7 +4278,7 @@ static void early_kmem_cache_node_alloc(int node) init_tracking(kmem_cache_node, n); #endif n = kasan_slab_alloc(kmem_cache_node, n, GFP_KERNEL, false); - slab->freelist = get_freepointer(kmem_cache_node, n); + slab->freelist = get_freepointer(kmem_cache_node, n, slab); slab->inuse = 1; kmem_cache_node->node[node] = n; init_kmem_cache_node(n); -- 2.42.0.459.ge4e396fd5e-goog