From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF182EB64DA for ; Tue, 18 Jul 2023 05:46:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1FCC88D0001; Tue, 18 Jul 2023 01:46:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1AC046B0074; Tue, 18 Jul 2023 01:46:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 09D118D0001; Tue, 18 Jul 2023 01:46:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id ED2E16B0071 for ; Tue, 18 Jul 2023 01:46:22 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B1A25160125 for ; Tue, 18 Jul 2023 05:46:22 +0000 (UTC) X-FDA: 81023647404.02.30D6CCA Received: from mail-vk1-f176.google.com (mail-vk1-f176.google.com [209.85.221.176]) by imf20.hostedemail.com (Postfix) with ESMTP id C6DC91C001D for ; Tue, 18 Jul 2023 05:46:19 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=iSb10qdk; spf=pass (imf20.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.221.176 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689659179; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JtlB/+ErIBoOqa+7JJKrhH1q3NOAQ7xQbYNgpx1uRWI=; b=HF7UJyaT4URrmAwm7ngaZtUDmKscqo3t1qorOFAXXJNZjsvtqfw07xBdNLaSKJAOmgFYei OQvOGd/R7Hb3hYTIMr0mzfvBFik8G9jhFrqrjqbNlmyR52jw/2oY1DkQLRv5NtzwXzB84p T6irR7kj2tNH5C5RUG4dhRKM67d2rD4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689659179; a=rsa-sha256; cv=none; b=EjH+FDazkHFNpD/uKV1OZ0SqvvDgSLATa6inDlbXMkWF4hZiZ3kL64/B9qobWPML+vXqdA gZJudMqHNy/uqYqoaQYUJ3xkh3/z12FHOhYwnhe/FNFp8XiA0lT90tK++hBJsLOmJQPbKv ZoWpAbMhanaS2iLMrUfl2MWhwgizX/E= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=iSb10qdk; spf=pass (imf20.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.221.176 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-vk1-f176.google.com with SMTP id 71dfb90a1353d-4716e4adb14so2041038e0c.0 for ; Mon, 17 Jul 2023 22:46:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1689659179; x=1692251179; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=JtlB/+ErIBoOqa+7JJKrhH1q3NOAQ7xQbYNgpx1uRWI=; b=iSb10qdkOz9OI82YpzzRgIcgtJgvKZ91mL/Q3ixmeURZFrks871Ij9/D4Xx6hB5rYw MrovNAbXhylGQnlOTK0Ut4MlKgW3lDywWPDuXu9RM2mW7Z56OkNAZC3Qwzy8h5ydGcdB MxcUxl9f8H29er9/5aJlRiGPYYA7ONSf/o0/JS63KjuexYGhm0Ay4YKBVI+I/MUjpzzi NRxKxoxUcInW7+9W1s+TIphPCwIG1/VyXCZVO/0UKEE6HrlS5YtnReIkisdOihCTRVe6 jf/l82RJM1t9FCjqQUuTHZwgBEk2rTzY+ziW1esuCAfelyh1SpEBpkmqe2evLoaHO5uJ cNpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689659179; x=1692251179; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JtlB/+ErIBoOqa+7JJKrhH1q3NOAQ7xQbYNgpx1uRWI=; b=I3yGf18ToUjuPL0+rKcSdxEqahv3hbFvxIwhubY/nF4lByr+3UsBJwKJJNPU/yRHiM TlMkqUEIAXMSlF7xwo7UCUCaPYS/xXrBtWaNogpJBhqZLBsne6tG6j90aanb/3dl87LI +2fiE3PS9VvksWQoGJhhF2PhEavXMueaflGaGzxYC7jGOiih7F7Fa/8WT4wSJOx2vAqx neGipBHuc2OLOszgpQu7vDIuPxEwKF8PsWJprNHtyZam4aaMHwLAvzIYpaa48LdV9/gr R5bOMbaDEB8RcLP/21wP4TtgB1/aSig3FaahM88C1ySMnDYMorDvuERRkWtSs+aRsKl3 JjkQ== X-Gm-Message-State: ABy/qLYUOvl+3Tg21wc5/Hemk9uwk+1dUngmpwKZtiRqxC/XpUqIGABp qOJ5H6pWHH44gyjYsvKQOPKmZurqQTeS5fJjNpU= X-Google-Smtp-Source: APBJJlGFShVuIJgbrdtto2xxQHz1VshAGkt5H1WXbCcdY/71zVZzSkQ0IG+fEHwF+tfKBrxYc3P4bGHvHlRdqQB9a80= X-Received: by 2002:a1f:4ec1:0:b0:481:3f62:be41 with SMTP id c184-20020a1f4ec1000000b004813f62be41mr770195vkb.15.1689659178430; Mon, 17 Jul 2023 22:46:18 -0700 (PDT) MIME-Version: 1.0 References: <20230714064422.3305234-1-gongruiqi@huaweicloud.com> In-Reply-To: <20230714064422.3305234-1-gongruiqi@huaweicloud.com> From: Hyeonggon Yoo <42.hyeyoo@gmail.com> Date: Tue, 18 Jul 2023 14:46:07 +0900 Message-ID: Subject: Re: [PATCH v5] Randomized slab caches for kmalloc() To: "GONG, Ruiqi" Cc: Vlastimil Babka , Andrew Morton , Joonsoo Kim , David Rientjes , Pekka Enberg , Christoph Lameter , Tejun Heo , Dennis Zhou , Alexander Potapenko , Marco Elver , Kees Cook , Jann Horn , Roman Gushchin , Dmitry Vyukov , Alexander Lobakin , Pedro Falcato , Paul Moore , James Morris , "Serge E . Hallyn" , Wang Weiyang , Xiu Jianfeng , linux-mm@kvack.org, linux-hardening@vger.kernel.org, linux-kernel@vger.kernel.org, gongruiqi1@huawei.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: 4i9wg88ygcyrjo9njykper17qe7mamkq X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: C6DC91C001D X-Rspam-User: X-HE-Tag: 1689659179-817635 X-HE-Meta: U2FsdGVkX1/9843CKh942Y35USomnQ+BASeZHJ+OUlNebFtRiW3kmNClXP7iG5q4MRehecbpNQu4Uil91lcPm37YIjU1B+gxMho/AET0U7S350dvMRFJ0H9tIYCo2DsKi3ZzYLbeCcXPJYqRws30mZsB7MGmvryJyG3J/dNKhNXntgwGkrflVenLfyCt5slmWfi9PljB8xcew0QoSZNU8zLHY46jLCMxc43OvIlSTMoXn4zteEaOYErP0dIj2DTeMrNcRv4eN4MGc88cnyghjfT8XxBN7duaw7fF3YQQQAookOPtp4YjwaTWSyYNyp1b7UhXGvMaCRQ/rki9W7ZcxkTbaVB9ONfHXtLXW4nOIrwwsu8GMR+j2erkjEMz8IPwalJn6HfoEzmoTDuFm/lXHjUSPEsAfTX0LRkmC8Us5WYvyEMtRCXunXawPhW2zxXc0zCBWjQNDBEMvImwDgXPCqsENFLHyAW4InFXH8MIcaLhX4x5QYFAfdLwm6O2VLUYaK1k8J55gTLc1RMxMpv68lPz92TTbSr+4nkI3zw5HS5fdmhq6XbU0VswYJ9G/2UndFJQ/Me4PbOmD9qnbwh2OiBhhrcgzIsOCkr4JekGso3QPzLYYzPb6CsQ8mh2dkqxVr5CkpxzO688cw+uaucmBOkjXAC33Zk4/X1qURmuSXTz2wP6i8yEsqmENBnPgcPd4oVp4srzMuqa3RgfVeQmShTEyf+CqNOuGbEfQvQArsdU/y4Roo6Fu05BkyaOjps8NwQbxOgwjFVoIaeyrJEqYpbmIbD74jOGrNnmeieJbKJcT06N7vQADXH3x+0f32rhxLX8xbOvZ8qa0QRY80ekpzIoDljnxikFdJ1+27fJ9WLpdyM9HRZHKX1zfBtyqx275r9t8ce0lcVF4maWHLCEPCoE7dDO48QhbZiKBdITjpMPLB1qfYHPMwNTuTADde6HuVjJCr9r9HSzVYcKvhe 4BkrHNS/ AMluqiIXqm/vBX/msd5mCV4xnGlP432jYx1rjS5XFJAG3ezVjq/VAoJkGXpei46dIxf+yxGbi+ATyI+5OJ6ZchSnG2QpUcR9HLCNe5cgVXFKHMD5D60DFBOf7fwT3Zee7qmS6VddGmjUDjG9unLlKoBNU9PkBB8XZJVOvSLcnS3jjIf4KoFCBBHZpooJX+sw5WQw28VXRH+u3a/I0E9ZhDg95TkH5zjA54oF3qL0PbTOIDhIyXmB4P/2VzcOlk0iVvXd6bjZcF0vBe4mLtwjF0WnDj7rI1iSYV2HRtOsmHk975Chb2Cs8Ie+NF/TLO13QTOxif2oRHRBk/uKFOqFG9rH5voEveSxMFrZZGJA7ug2Sq/spUYAT4pcdIjnBSGz0S7Trydh+7NeGee5wS31y7DryxOs8yDZ9V7mQ7hQW6k1xVW4c5Dq/Bzmlux6uPKkHlvCb+hC9AndnEutzHfrGE3pnBsca0mchGjMSgYLHB5Am419X95mCB+xIHZzoGiGWZMwPx7Gow5gJkMrOIuELpExn70DbSg/598I25skAq4fEmJCbNut41dV+t7KMzEnrlUYUZ/1fYmPi/0aI4G9qLbltVTdrZvEl4dwIqGDI32xoFWkdpzsrGUgMKzXWfKE2ra/8tX4mM/6bp/IG/y//eWhAvQOkXdIhVeNWr9TMqO2hz8C9mrOXDbL0MRFzx8MAWimqOXh3oaKeYf7yKJ0S2w1KcT8ojzBAo0H+Q+aL2sraxURr89gH2s67rA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Jul 14, 2023 at 3:40=E2=80=AFPM GONG, Ruiqi wrote: > > When exploiting memory vulnerabilities, "heap spraying" is a common > technique targeting those related to dynamic memory allocation (i.e. the > "heap"), and it plays an important role in a successful exploitation. > Basically, it is to overwrite the memory area of vulnerable object by > triggering allocation in other subsystems or modules and therefore > getting a reference to the targeted memory location. It's usable on > various types of vulnerablity including use after free (UAF), heap out- > of-bound write and etc. > > There are (at least) two reasons why the heap can be sprayed: 1) generic > slab caches are shared among different subsystems and modules, and > 2) dedicated slab caches could be merged with the generic ones. > Currently these two factors cannot be prevented at a low cost: the first > one is a widely used memory allocation mechanism, and shutting down slab > merging completely via `slub_nomerge` would be overkill. > > To efficiently prevent heap spraying, we propose the following approach: > to create multiple copies of generic slab caches that will never be > merged, and random one of them will be used at allocation. The random > selection is based on the address of code that calls `kmalloc()`, which > means it is static at runtime (rather than dynamically determined at > each time of allocation, which could be bypassed by repeatedly spraying > in brute force). In other words, the randomness of cache selection will > be with respect to the code address rather than time, i.e. allocations > in different code paths would most likely pick different caches, > although kmalloc() at each place would use the same cache copy whenever > it is executed. In this way, the vulnerable object and memory allocated > in other subsystems and modules will (most probably) be on different > slab caches, which prevents the object from being sprayed. > > Meanwhile, the static random selection is further enhanced with a > per-boot random seed, which prevents the attacker from finding a usable > kmalloc that happens to pick the same cache with the vulnerable > subsystem/module by analyzing the open source code. In other words, with > the per-boot seed, the random selection is static during each time the > system starts and runs, but not across different system startups. > > The overhead of performance has been tested on a 40-core x86 server by > comparing the results of `perf bench all` between the kernels with and > without this patch based on the latest linux-next kernel, which shows > minor difference. A subset of benchmarks are listed below: > > sched/ sched/ syscall/ mem/ mem/ > messaging pipe basic memcpy memset > (sec) (sec) (sec) (GB/sec) (GB/sec) > > control1 0.019 5.459 0.733 15.258789 51.398026 > control2 0.019 5.439 0.730 16.009221 48.828125 > control3 0.019 5.282 0.735 16.009221 48.828125 > control_avg 0.019 5.393 0.733 15.759077 49.684759 > > experiment1 0.019 5.374 0.741 15.500992 46.502976 > experiment2 0.019 5.440 0.746 16.276042 51.398026 > experiment3 0.019 5.242 0.752 15.258789 51.398026 > experiment_avg 0.019 5.352 0.746 15.678608 49.766343 > > The overhead of memory usage was measured by executing `free` after boot > on a QEMU VM with 1GB total memory, and as expected, it's positively > correlated with # of cache copies: > > control 4 copies 8 copies 16 copies > > total 969.8M 968.2M 968.2M 968.2M > used 20.0M 21.9M 24.1M 26.7M > free 936.9M 933.6M 931.4M 928.6M > available 932.2M 928.8M 926.6M 923.9M > > Co-developed-by: Xiu Jianfeng > Signed-off-by: Xiu Jianfeng > Signed-off-by: GONG, Ruiqi > Reviewed-by: Kees Cook > --- > > v5: > - Rebase to the latest linux-next. > - Make CONFIG_RANDOM_KMALLOC_CACHES depends on !SLUB_TINY. > - Rephrase parts of CONFIG_RANDOM_KMALLOC_CACHES's help paragraph. > - Pass 0 as caller at places where the real caller is not needed. > - Restore KMALLOC_NORMAL to 0, and move KMALLOC_RANDOM_* after it. > - Change RANDOM_KMALLOC_CACHES_NR to 15, and adjust the kmalloc rnd > name accordingly. > - Replace RANDOM_KMALLOC_CACHES_BITS via ilog2(..._NR + 1). > > v4: > - Set # of caches to 16 and remove config selection. > - Shorten "kmalloc-random-" to "kmalloc-rnd-". > - Update commit log and config's help paragraph. > - Fine-tune PERCPU_DYNAMIC_SIZE_SHIFT to 12 instead of 13 (enough to > pass compilation with allmodconfig and CONFIG_SLUB_TINY=3Dn). > - Some cleanup and typo fixing. > - Link: https://lore.kernel.org/all/20230626031835.2279738-1-gongruiqi@= huaweicloud.com/ > > v3: > - Replace SLAB_RANDOMSLAB with the new existing SLAB_NO_MERGE flag. > - Shorten long code lines by wrapping and renaming. > - Update commit message with latest perf benchmark and additional > theorectical explanation. > - Remove "RFC" from patch title and make it a formal patch > - Link: https://lore.kernel.org/all/20230616111843.3677378-1-gongruiqi@= huaweicloud.com/ > > v2: > - Use hash_64() and a per-boot random seed to select kmalloc() caches. > - Change acceptable # of caches from [4,16] to {2,4,8,16}, which is > more compatible with hashing. > - Supplement results of performance and memory overhead tests. > - Link: https://lore.kernel.org/all/20230508075507.1720950-1-gongruiqi1= @huawei.com/ > > v1: > - Link: https://lore.kernel.org/all/20230315095459.186113-1-gongruiqi1@= huawei.com/ > > include/linux/percpu.h | 12 +++++++--- > include/linux/slab.h | 23 ++++++++++++++++--- > mm/Kconfig | 17 ++++++++++++++ > mm/kfence/kfence_test.c | 7 ++++-- > mm/slab.c | 2 +- > mm/slab.h | 2 +- > mm/slab_common.c | 49 ++++++++++++++++++++++++++++++++++++----- > 7 files changed, 97 insertions(+), 15 deletions(-) > > diff --git a/include/linux/percpu.h b/include/linux/percpu.h > index b3b458442330..68fac2e7cbe6 100644 > --- a/include/linux/percpu.h > +++ b/include/linux/percpu.h > @@ -35,6 +35,12 @@ > #define PCPU_BITMAP_BLOCK_BITS (PCPU_BITMAP_BLOCK_SIZE >> \ > PCPU_MIN_ALLOC_SHIFT) > > +#ifdef CONFIG_RANDOM_KMALLOC_CACHES > +#define PERCPU_DYNAMIC_SIZE_SHIFT 12 > +#else > +#define PERCPU_DYNAMIC_SIZE_SHIFT 10 > +#endif > + > /* > * Percpu allocator can serve percpu allocations before slab is > * initialized which allows slab to depend on the percpu allocator. > @@ -42,7 +48,7 @@ > * for this. Keep PERCPU_DYNAMIC_RESERVE equal to or larger than > * PERCPU_DYNAMIC_EARLY_SIZE. > */ > -#define PERCPU_DYNAMIC_EARLY_SIZE (20 << 10) > +#define PERCPU_DYNAMIC_EARLY_SIZE (20 << PERCPU_DYNAMIC_SIZE_SHIFT) > > /* > * PERCPU_DYNAMIC_RESERVE indicates the amount of free area to piggy > @@ -56,9 +62,9 @@ > * intelligent way to determine this would be nice. > */ > #if BITS_PER_LONG > 32 > -#define PERCPU_DYNAMIC_RESERVE (28 << 10) > +#define PERCPU_DYNAMIC_RESERVE (28 << PERCPU_DYNAMIC_SIZE_SHIFT) > #else > -#define PERCPU_DYNAMIC_RESERVE (20 << 10) > +#define PERCPU_DYNAMIC_RESERVE (20 << PERCPU_DYNAMIC_SIZE_SHIFT) > #endif > > extern void *pcpu_base_addr; > diff --git a/include/linux/slab.h b/include/linux/slab.h > index 848c7c82ad5a..8228d1276a2f 100644 > --- a/include/linux/slab.h > +++ b/include/linux/slab.h > @@ -19,6 +19,7 @@ > #include > #include > #include > +#include > > > /* > @@ -345,6 +346,12 @@ static inline unsigned int arch_slab_minalign(void) > #define SLAB_OBJ_MIN_SIZE (KMALLOC_MIN_SIZE < 16 ? \ > (KMALLOC_MIN_SIZE) : 16) > > +#ifdef CONFIG_RANDOM_KMALLOC_CACHES > +#define RANDOM_KMALLOC_CACHES_NR 15 // # of cache copies > +#else > +#define RANDOM_KMALLOC_CACHES_NR 0 > +#endif > + > /* > * Whenever changing this, take care of that kmalloc_type() and > * create_kmalloc_caches() still work as intended. > @@ -361,6 +368,8 @@ enum kmalloc_cache_type { > #ifndef CONFIG_MEMCG_KMEM > KMALLOC_CGROUP =3D KMALLOC_NORMAL, > #endif > + KMALLOC_RANDOM_START =3D KMALLOC_NORMAL, > + KMALLOC_RANDOM_END =3D KMALLOC_RANDOM_START + RANDOM_KMALLOC_CACH= ES_NR, > #ifdef CONFIG_SLUB_TINY > KMALLOC_RECLAIM =3D KMALLOC_NORMAL, > #else > @@ -386,14 +395,22 @@ kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH= + 1]; > (IS_ENABLED(CONFIG_ZONE_DMA) ? __GFP_DMA : 0) | \ > (IS_ENABLED(CONFIG_MEMCG_KMEM) ? __GFP_ACCOUNT : 0)) > > -static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags) > +extern unsigned long random_kmalloc_seed; > + > +static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags,= unsigned long caller) > { > /* > * The most common case is KMALLOC_NORMAL, so test for it > * with a single branch for all the relevant flags. > */ > if (likely((flags & KMALLOC_NOT_NORMAL_BITS) =3D=3D 0)) > +#ifdef CONFIG_RANDOM_KMALLOC_CACHES > + /* RANDOM_KMALLOC_CACHES_NR (=3D15) copies + the KMALLOC_= NORMAL */ > + return KMALLOC_RANDOM_START + hash_64(caller ^ random_kma= lloc_seed, > + ilog2(RANDOM_KMALLO= C_CACHES_NR + 1)); > +#else > return KMALLOC_NORMAL; > +#endif > > /* > * At least one of the flags has to be set. Their priorities in > @@ -580,7 +597,7 @@ static __always_inline __alloc_size(1) void *kmalloc(= size_t size, gfp_t flags) > > index =3D kmalloc_index(size); > return kmalloc_trace( > - kmalloc_caches[kmalloc_type(flags)][index= ], > + kmalloc_caches[kmalloc_type(flags, _RET_I= P_)][index], > flags, size); > } > return __kmalloc(size, flags); > @@ -596,7 +613,7 @@ static __always_inline __alloc_size(1) void *kmalloc_= node(size_t size, gfp_t fla > > index =3D kmalloc_index(size); > return kmalloc_node_trace( > - kmalloc_caches[kmalloc_type(flags)][index= ], > + kmalloc_caches[kmalloc_type(flags, _RET_I= P_)][index], > flags, node, size); > } > return __kmalloc_node(size, flags, node); > diff --git a/mm/Kconfig b/mm/Kconfig > index 22acffd9009d..989ab72bbecc 100644 > --- a/mm/Kconfig > +++ b/mm/Kconfig > @@ -337,6 +337,23 @@ config SLUB_CPU_PARTIAL > which requires the taking of locks that may cause latency spike= s. > Typically one would choose no for a realtime system. > > +config RANDOM_KMALLOC_CACHES > + default n > + depends on SLUB && !SLUB_TINY > + bool "Randomize slab caches for normal kmalloc" > + help > + A hardening feature that creates multiple copies of slab caches= for > + normal kmalloc allocation and makes kmalloc randomly pick one b= ased > + on code address, which makes the attackers more difficult to sp= ray > + vulnerable memory objects on the heap for the purpose of exploi= ting > + memory vulnerabilities. > + > + Currently the number of copies is set to 16, a reasonably large= value > + that effectively diverges the memory objects allocated for diff= erent > + subsystems or modules into different caches, at the expense of = a > + limited degree of memory and CPU overhead that relates to hardw= are and > + system workload. > + > endmenu # SLAB allocator options > > config SHUFFLE_PAGE_ALLOCATOR > diff --git a/mm/kfence/kfence_test.c b/mm/kfence/kfence_test.c > index 9e008a336d9f..95b2b84c296d 100644 > --- a/mm/kfence/kfence_test.c > +++ b/mm/kfence/kfence_test.c > @@ -212,7 +212,9 @@ static void test_cache_destroy(void) > > static inline size_t kmalloc_cache_alignment(size_t size) > { > - return kmalloc_caches[kmalloc_type(GFP_KERNEL)][__kmalloc_index(s= ize, false)]->align; > + /* just to get ->align so no need to pass in the real caller */ > + enum kmalloc_cache_type type =3D kmalloc_type(GFP_KERNEL, 0); > + return kmalloc_caches[type][__kmalloc_index(size, false)]->align; > } > > /* Must always inline to match stack trace against caller. */ > @@ -282,8 +284,9 @@ static void *test_alloc(struct kunit *test, size_t si= ze, gfp_t gfp, enum allocat > > if (is_kfence_address(alloc)) { > struct slab *slab =3D virt_to_slab(alloc); > + enum kmalloc_cache_type type =3D kmalloc_type(GFP= _KERNEL, _RET_IP_); > struct kmem_cache *s =3D test_cache ?: > - kmalloc_caches[kmalloc_type(GFP_K= ERNEL)][__kmalloc_index(size, false)]; > + kmalloc_caches[type][__kmalloc_in= dex(size, false)]; > > /* > * Verify that various helpers return the right v= alues > diff --git a/mm/slab.c b/mm/slab.c > index 88194391d553..9ad3d0f2d1a5 100644 > --- a/mm/slab.c > +++ b/mm/slab.c > @@ -1670,7 +1670,7 @@ static size_t calculate_slab_order(struct kmem_cach= e *cachep, > if (freelist_size > KMALLOC_MAX_CACHE_SIZE) { > freelist_cache_size =3D PAGE_SIZE << get_= order(freelist_size); > } else { > - freelist_cache =3D kmalloc_slab(freelist_= size, 0u); > + freelist_cache =3D kmalloc_slab(freelist_= size, 0u, _RET_IP_); > if (!freelist_cache) > continue; > freelist_cache_size =3D freelist_cache->s= ize; > diff --git a/mm/slab.h b/mm/slab.h > index 9c0e09d0f81f..799a315695c6 100644 > --- a/mm/slab.h > +++ b/mm/slab.h > @@ -282,7 +282,7 @@ void setup_kmalloc_cache_index_table(void); > void create_kmalloc_caches(slab_flags_t); > > /* Find the kmalloc slab corresponding for a certain size */ > -struct kmem_cache *kmalloc_slab(size_t, gfp_t); > +struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags, unsigned long = caller); > > void *__kmem_cache_alloc_node(struct kmem_cache *s, gfp_t gfpflags, > int node, size_t orig_size, > diff --git a/mm/slab_common.c b/mm/slab_common.c > index d1555ea2981a..01cdbf122463 100644 > --- a/mm/slab_common.c > +++ b/mm/slab_common.c > @@ -678,6 +678,11 @@ kmalloc_caches[NR_KMALLOC_TYPES][KMALLOC_SHIFT_HIGH = + 1] __ro_after_init =3D > { /* initialization for https://bugs.llvm.org/show_bug.cgi?id=3D42570 */= }; > EXPORT_SYMBOL(kmalloc_caches); > > +#ifdef CONFIG_RANDOM_KMALLOC_CACHES > +unsigned long random_kmalloc_seed __ro_after_init; > +EXPORT_SYMBOL(random_kmalloc_seed); > +#endif > + > /* > * Conversion table for small slabs sizes / 8 to the index in the > * kmalloc array. This is necessary for slabs < 192 since we have non po= wer > @@ -720,7 +725,7 @@ static inline unsigned int size_index_elem(unsigned i= nt bytes) > * Find the kmem_cache structure that serves a given size of > * allocation > */ > -struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags) > +struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags, unsigned long = caller) > { > unsigned int index; > > @@ -735,7 +740,7 @@ struct kmem_cache *kmalloc_slab(size_t size, gfp_t fl= ags) > index =3D fls(size - 1); > } > > - return kmalloc_caches[kmalloc_type(flags)][index]; > + return kmalloc_caches[kmalloc_type(flags, caller)][index]; > } > > size_t kmalloc_size_roundup(size_t size) > @@ -752,8 +757,11 @@ size_t kmalloc_size_roundup(size_t size) > if (size > KMALLOC_MAX_CACHE_SIZE) > return PAGE_SIZE << get_order(size); > > - /* The flags don't matter since size_index is common to all. */ > - c =3D kmalloc_slab(size, GFP_KERNEL); > + /* > + * The flags don't matter since size_index is common to all. > + * Neither does the caller for just getting ->object_size. > + */ > + c =3D kmalloc_slab(size, GFP_KERNEL, 0); > return c ? c->object_size : 0; > } > EXPORT_SYMBOL(kmalloc_size_roundup); > @@ -776,12 +784,35 @@ EXPORT_SYMBOL(kmalloc_size_roundup); > #define KMALLOC_RCL_NAME(sz) > #endif > > +#ifdef CONFIG_RANDOM_KMALLOC_CACHES > +#define __KMALLOC_RANDOM_CONCAT(a, b) a ## b > +#define KMALLOC_RANDOM_NAME(N, sz) __KMALLOC_RANDOM_CONCAT(KMA_RAND_, N)= (sz) > +#define KMA_RAND_1(sz) .name[KMALLOC_RANDOM_START + 1]= =3D "kmalloc-rnd-01-" #sz, > +#define KMA_RAND_2(sz) KMA_RAND_1(sz) .name[KMALLOC_RANDOM_START + 2]= =3D "kmalloc-rnd-02-" #sz, > +#define KMA_RAND_3(sz) KMA_RAND_2(sz) .name[KMALLOC_RANDOM_START + 3]= =3D "kmalloc-rnd-03-" #sz, > +#define KMA_RAND_4(sz) KMA_RAND_3(sz) .name[KMALLOC_RANDOM_START + 4]= =3D "kmalloc-rnd-04-" #sz, > +#define KMA_RAND_5(sz) KMA_RAND_4(sz) .name[KMALLOC_RANDOM_START + 5]= =3D "kmalloc-rnd-05-" #sz, > +#define KMA_RAND_6(sz) KMA_RAND_5(sz) .name[KMALLOC_RANDOM_START + 6]= =3D "kmalloc-rnd-06-" #sz, > +#define KMA_RAND_7(sz) KMA_RAND_6(sz) .name[KMALLOC_RANDOM_START + 7]= =3D "kmalloc-rnd-07-" #sz, > +#define KMA_RAND_8(sz) KMA_RAND_7(sz) .name[KMALLOC_RANDOM_START + 8]= =3D "kmalloc-rnd-08-" #sz, > +#define KMA_RAND_9(sz) KMA_RAND_8(sz) .name[KMALLOC_RANDOM_START + 9]= =3D "kmalloc-rnd-09-" #sz, > +#define KMA_RAND_10(sz) KMA_RAND_9(sz) .name[KMALLOC_RANDOM_START + 10]= =3D "kmalloc-rnd-10-" #sz, > +#define KMA_RAND_11(sz) KMA_RAND_10(sz) .name[KMALLOC_RANDOM_START + 11]= =3D "kmalloc-rnd-11-" #sz, > +#define KMA_RAND_12(sz) KMA_RAND_11(sz) .name[KMALLOC_RANDOM_START + 12]= =3D "kmalloc-rnd-12-" #sz, > +#define KMA_RAND_13(sz) KMA_RAND_12(sz) .name[KMALLOC_RANDOM_START + 13]= =3D "kmalloc-rnd-13-" #sz, > +#define KMA_RAND_14(sz) KMA_RAND_13(sz) .name[KMALLOC_RANDOM_START + 14]= =3D "kmalloc-rnd-14-" #sz, > +#define KMA_RAND_15(sz) KMA_RAND_14(sz) .name[KMALLOC_RANDOM_START + 15]= =3D "kmalloc-rnd-15-" #sz, > +#else // CONFIG_RANDOM_KMALLOC_CACHES > +#define KMALLOC_RANDOM_NAME(N, sz) > +#endif > + > #define INIT_KMALLOC_INFO(__size, __short_size) \ > { \ > .name[KMALLOC_NORMAL] =3D "kmalloc-" #__short_size, \ > KMALLOC_RCL_NAME(__short_size) \ > KMALLOC_CGROUP_NAME(__short_size) \ > KMALLOC_DMA_NAME(__short_size) \ > + KMALLOC_RANDOM_NAME(RANDOM_KMALLOC_CACHES_NR, __short_size) \ > .size =3D __size, \ > } > > @@ -890,6 +921,11 @@ new_kmalloc_cache(int idx, enum kmalloc_cache_type t= ype, slab_flags_t flags) > flags |=3D SLAB_CACHE_DMA; > } > > +#ifdef CONFIG_RANDOM_KMALLOC_CACHES > + if (type >=3D KMALLOC_RANDOM_START && type <=3D KMALLOC_RANDOM_EN= D) > + flags |=3D SLAB_NO_MERGE; > +#endif > + > /* > * If CONFIG_MEMCG_KMEM is enabled, disable cache merging for > * KMALLOC_NORMAL caches. > @@ -941,6 +977,9 @@ void __init create_kmalloc_caches(slab_flags_t flags) > new_kmalloc_cache(2, type, flags); > } > } > +#ifdef CONFIG_RANDOM_KMALLOC_CACHES > + random_kmalloc_seed =3D get_random_u64(); > +#endif > > /* Kmalloc array is now usable */ > slab_state =3D UP; > @@ -976,7 +1015,7 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, in= t node, unsigned long caller > return ret; > } > > - s =3D kmalloc_slab(size, flags); > + s =3D kmalloc_slab(size, flags, caller); > > if (unlikely(ZERO_OR_NULL_PTR(s))) > return s; > -- > 2.25.1 Looks good to me, Reviewed-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Thanks, Hyeonggon