From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C94DEB64D9 for ; Tue, 4 Jul 2023 13:59:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B93DD28007C; Tue, 4 Jul 2023 09:59:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B43ED280076; Tue, 4 Jul 2023 09:59:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A321F28007C; Tue, 4 Jul 2023 09:59:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 968D9280076 for ; Tue, 4 Jul 2023 09:59:12 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 5F7591609EA for ; Tue, 4 Jul 2023 13:59:12 +0000 (UTC) X-FDA: 80974086144.21.FA02B1F Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf05.hostedemail.com (Postfix) with ESMTP id 9BE92100007 for ; Tue, 4 Jul 2023 13:59:10 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=ied37usO; spf=pass (imf05.hostedemail.com: domain of 3rSWkZAsKCLQgUnnYilcttiaiiafY.Wigfchor-ggepUWe.ila@flex--matteorizzo.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3rSWkZAsKCLQgUnnYilcttiaiiafY.Wigfchor-ggepUWe.ila@flex--matteorizzo.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688479150; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=DC8Xq57VnNVy2UPnztEh7pdyCXoL2+wf6j7oIzU/k0U=; b=5wdz9/wpje+p26Gx/hPz+ccUEmsRwWYLnwdz+aHouJnE1nYqvP1YvQY8uq7v3Xxfvf49Ij NrBnq+Ax/v1c19EqDHQJqcQB2IG25kIublAY3vU+tRh4mfpRgg911Icuq7jjp3BnUPLDB1 YLh2ju0Z6r0i9piBhMKmnBl+yxuGekM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688479150; a=rsa-sha256; cv=none; b=kCGjtKIy7dN9P8HU3kcBGb+mzGK3ClS79HN6dRilBvdHhT7pLpaTH4VZIb2A1U7i3V+HWM r82mz+RMXKUMp3+sAD8c2oqKBPa6VS9qTAc8FPPUdwwP64GDuLvJVAxf6EXIMuSGaxguEu DIbCI/dHqGsbVpF1bgOx/Ebzod6F/+E= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=ied37usO; spf=pass (imf05.hostedemail.com: domain of 3rSWkZAsKCLQgUnnYilcttiaiiafY.Wigfchor-ggepUWe.ila@flex--matteorizzo.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=3rSWkZAsKCLQgUnnYilcttiaiiafY.Wigfchor-ggepUWe.ila@flex--matteorizzo.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-569e7aec37bso51200447b3.2 for ; Tue, 04 Jul 2023 06:59:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1688479149; x=1691071149; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=DC8Xq57VnNVy2UPnztEh7pdyCXoL2+wf6j7oIzU/k0U=; b=ied37usOyUZSNNUYFWB/0x9Gp2Xy3VBqOZsdjhUe4raEDydnbYF6HgEiQ3X+pvSQIR H6v+ZACeiuklWaHQuyR+PUR4X1/upWRLXxCWvZj8/qbVWoBvWqe/6sKN21wSoOfELlOc CD08aaZPDspCmCOvI+LpffHdthodD0JLKmoCGvxQZV3rodZFF55UOJFt2YJXQfDNZYV7 39xNX8E7YYFfbNKTfbuNdbxVUGDDkGZiDRmib0EsF2Mou0u4OYO/oPWvWmehwRzDFK5D Imt+Bwof1BSr+KwKcJJidytEDE5co5ZNs14Yx3ZfgsJkQv3RnObo5szD3Nqeos8pKDEj Jhbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688479149; x=1691071149; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=DC8Xq57VnNVy2UPnztEh7pdyCXoL2+wf6j7oIzU/k0U=; b=lT2Zx141D/Q4lq474oNaWK7ftVe2KXM3+EMi5Pn8uvywNch/6Yn8ZcIasgHEp48Cr4 PgyY7lX3KqEBBuSAzYZG1NNlv8yj4dyXFDb8hQGgMMoIh5HHAqM/MGC/0F9BAUThVRqR V/UFgjMwq8efPbX6crNBhoNm022FSrYh3UN2KREbJwPMU++5RJkcV6RF21cKMJmBSaGs xL7uabB5mooGPmMP5uEsrmh7hcJGYj8goUpgI2nZRez/3kefyAG/xXtIXLxQniNWVxx5 c8yepxLyAK4NwgNV73nq5yUoK/RwVJzWvpUnID83otmWcR3hq3fK/Yjh3r76IsDeEm3m Gubw== X-Gm-Message-State: ABy/qLaXW9GbWwn9bUnOa1FpUpAVXI3f7xj6igwwu6vKMgj8ihQ1WhXH cDyFlzQEa8ihpdoJBkyXpoJ12hB92VXEVidxHMxtVXCNjtLF7yi5JwwBVeHqH4Amlz/VbgUuCCJ 6i4pSp84mRH0m4IZ0oCFzDE440Gs+fgt7REvpmf+wPBZyMuQTt7+ZnWy4c7fAG2PBLbAzDw== X-Google-Smtp-Source: APBJJlHCT8YpmjLt4pwJNPMv9gOj/d3Eu5xEqnXGre+LNmaIV7PZ1W8rT72U+lTdniCwmoOaKpA4OJ+vM5tue1x5Dw== X-Received: from mr-cloudtop2.c.googlers.com ([fda3:e722:ac3:cc00:31:98fb:c0a8:fb5]) (user=matteorizzo job=sendgmr) by 2002:a81:4428:0:b0:577:4540:905d with SMTP id r40-20020a814428000000b005774540905dmr76533ywa.8.1688479149512; Tue, 04 Jul 2023 06:59:09 -0700 (PDT) Date: Tue, 4 Jul 2023 13:58:34 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.41.0.255.g8b1d071c50-goog Message-ID: <20230704135834.3884421-1-matteorizzo@google.com> Subject: [PATCH v2] mm/slub: refactor freelist to use custom type From: Matteo Rizzo To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, roman.gushchin@linux.dev, 42.hyeyoo@gmail.com Cc: jannh@google.com, matteorizzo@google.com Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 9BE92100007 X-Rspam-User: X-Stat-Signature: x8bx6aqfti3s3jdz8b7g9iyjxxkmxpni X-Rspamd-Server: rspam03 X-HE-Tag: 1688479150-420759 X-HE-Meta: U2FsdGVkX18cnzeHjdVzMtnhnMDfIQbNvbvlSHQUS0xFhmOiypDYvT39EKlYoXA2M9lDvMhnlRr/4jpCqzMdct6vdexsfJOSUU1rqWJqYAyvaVJxvGGvdTOg2iG5f0yUSddFjHhUxT6TpT74DbZeDwEDSkdxHMpSEEeHfBNDjYd+eJMF6fwnRLd7W1roSq5/VgGor6YjQBybsB+EA+T5n3er96cPQDAatCQcLsB7QRrqkNEqLAVv/noXzwv9BePSnYhimWTFwRvCpqO5rFdaJVkMrFW4v2qqsV6HnjdAZnzURPp0BO1SSNKhF5t/6A4ldiP5vwr+ytQId1zOf7mXTgkvx6mIgVrJk2EReTooYOX0kKDBmFpJoolRECpJqQ8CE5CXGmPe8FLbanWOokwSSmIMxe6OOE5zrv/ihNmDkUz5v2EmSy7t66QnvfIMaPSEJotOECEEJ1A8Gto+pwvzKsOHK6A/06JeWfDHul7rbN4jHQ4GU0J8XK8Ug5xEXpiauGWddYMynfZTqKCTO0do23xGPMi9+SUi486zcGy/D9p4SC9+imV7N3XA+V2T4qGAbNhiIYPJMyppybJVD897QsJOoM2GJo/SNNHSFkRjTulFOCFDZdjXZ0dmW75iJrCrlIh34OAFO1ahZzqYsS6hHHYQhb+UWn5rL/D0XUnJG+DUU6M/8lCfB51rIB7KYYW7wnbXLYZf1PE9JlbbkfUf6ccvWNxKGn05vmrm/cjtr/8qz/w7ABsx7gXA1PB0NJfGChMwmSgzZ3rad0CgS06r+Hek/tVf1r17Pp0Zi3d6j1RfglK4A+nLP7kvzWyYMT6W62sJfYxKtnH3OzNqfiNS/Tsqc2hU8i+36z7WDUlF+DbBtv3MoRQjEScHB74PV4ux6LDRWA6Ek9i/jDIiliFCGDvnkZu65JHKDqYZlKYLamcaM+3cWrFaQ8bcM0NwDhPmHisxRHcvEHycyNvZNEH eaQAH4sy p4Dkw4ApwiYeEZucM13I+wHouJmv8/+wnNMTeqCgWobabGcOqKOfCZXyTT9Oeck03WrhtLv2tHRVNIa+lZxB9eNIV8u00DdWbKOQ5gyto0aASY4ne+mvmNH+yGN70kFVQ5eZlz4zbEIygJm7pd5eoguBDZ6y95YvrKM4EGoKBwuRP7nw3wYIdgQtYHND3RMZBtA9dkdOwk5G7pZdbzmXjKzgO+5DoumAughzONDYEVM42R3vNrec6lrSbKWjWW667OloLjxvhz6aoga+0la1TjUhAlndxgEN77LzzAJlFesAbZVziTJHX8uEmLuPTd9rxy8oXOmF8Nm+jwnjRv5CLWgWrwfGok3nJ6L1lLaO6I+a8EjM+wAji6j6LcG/UNDpKrKadNUsH5rqGk/TM7xtOfjVfPt7Ss+3N28of7EQY8t3b+zdn4aaAawE4WdKDQ6RLbe++l/ZY1tw8kKH8A63qD1XhMLnotoTAZfr+yCWKqXHWf30ZE25j4WKnEzitJpjYoqTHHwLzLhXEsH+4iILHfYMWW26EwkCPtfe6VStvDJhwrlJC3OR4KUuTcstChn0mFm/udECuEBrmulhj0TTmFIkgYQGpk8mz0iEu/pFvG4W0DdHZMR5f9hAXZlq/j8PG7+uTXcwKgwBmBlQL00e+rCj0dOeS8cA2KTuoLfBI+dnjM4A5Hw8RcmJ9E8UnabzPE6+5Vs0pW+y946VPOs5g0SsPeg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.001369, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Jann Horn Currently the SLUB code represents encoded freelist entries as "void*". That's misleading, those things are encoded under CONFIG_SLAB_FREELIST_HARDENED so that they're not actually dereferencable. Give them their own type, and split freelist_ptr() into one function per direction (one for encoding, one for decoding). Signed-off-by: Jann Horn Co-developed-by: Matteo Rizzo Signed-off-by: Matteo Rizzo --- v2: * Fix compilation error with SLUB_TINY * Move the freeptr_t typedef to slub.c mm/slub.c | 43 ++++++++++++++++++++++++++++++++----------- 1 file changed, 32 insertions(+), 11 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index e3b5d5c0eb3a..f8cc47eff742 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -360,13 +360,19 @@ static struct workqueue_struct *flushwq; * Core slab cache functions *******************************************************************/ +/* + * freeptr_t represents a SLUB freelist pointer, which might be encoded + * and not dereferenceable if CONFIG_SLAB_FREELIST_HARDENED is enabled. + */ +typedef struct { unsigned long v; } freeptr_t; + /* * Returns freelist pointer (ptr). With hardening, this is obfuscated * with an XOR of the address where the pointer is held and a per-cache * random number. */ -static inline void *freelist_ptr(const struct kmem_cache *s, void *ptr, - unsigned long ptr_addr) +static inline freeptr_t freelist_ptr_encode(const struct kmem_cache *s, + void *ptr, unsigned long ptr_addr) { #ifdef CONFIG_SLAB_FREELIST_HARDENED /* @@ -379,25 +385,40 @@ static inline void *freelist_ptr(const struct kmem_cache *s, void *ptr, * calls get_freepointer() with an untagged pointer, which causes the * freepointer to be restored incorrectly. */ - return (void *)((unsigned long)ptr ^ s->random ^ - swab((unsigned long)kasan_reset_tag((void *)ptr_addr))); + return (freeptr_t){.v = (unsigned long)ptr ^ s->random ^ + swab((unsigned long)kasan_reset_tag((void *)ptr_addr))}; #else - return ptr; + return (freeptr_t){.v = (unsigned long)ptr}; #endif } +static inline void *freelist_ptr_decode(const struct kmem_cache *s, + freeptr_t ptr, unsigned long ptr_addr) +{ + void *decoded; + +#ifdef CONFIG_SLAB_FREELIST_HARDENED + /* See the comment in freelist_ptr_encode */ + decoded = (void *)(ptr.v ^ s->random ^ + swab((unsigned long)kasan_reset_tag((void *)ptr_addr))); +#else + decoded = (void *)ptr.v; +#endif + return decoded; +} + /* Returns the freelist pointer recorded at location ptr_addr. */ static inline void *freelist_dereference(const struct kmem_cache *s, void *ptr_addr) { - return freelist_ptr(s, (void *)*(unsigned long *)(ptr_addr), + return freelist_ptr_decode(s, *(freeptr_t *)(ptr_addr), (unsigned long)ptr_addr); } static inline void *get_freepointer(struct kmem_cache *s, void *object) { object = kasan_reset_tag(object); - return freelist_dereference(s, object + s->offset); + return freelist_dereference(s, (freeptr_t *)(object + s->offset)); } #ifndef CONFIG_SLUB_TINY @@ -421,15 +442,15 @@ __no_kmsan_checks static inline void *get_freepointer_safe(struct kmem_cache *s, void *object) { unsigned long freepointer_addr; - void *p; + freeptr_t p; if (!debug_pagealloc_enabled_static()) return get_freepointer(s, object); object = kasan_reset_tag(object); freepointer_addr = (unsigned long)object + s->offset; - copy_from_kernel_nofault(&p, (void **)freepointer_addr, sizeof(p)); - return freelist_ptr(s, p, freepointer_addr); + copy_from_kernel_nofault(&p, (freeptr_t *)freepointer_addr, sizeof(p)); + return freelist_ptr_decode(s, p, freepointer_addr); } static inline void set_freepointer(struct kmem_cache *s, void *object, void *fp) @@ -441,7 +462,7 @@ static inline void set_freepointer(struct kmem_cache *s, void *object, void *fp) #endif freeptr_addr = (unsigned long)kasan_reset_tag((void *)freeptr_addr); - *(void **)freeptr_addr = freelist_ptr(s, fp, freeptr_addr); + *(freeptr_t *)freeptr_addr = freelist_ptr_encode(s, fp, freeptr_addr); } /* Loop over all objects in a slab */ base-commit: 24be4d0b46bb0c3c1dc7bacd30957d6144a70dfc -- 2.41.0.255.g8b1d071c50-goog