From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 649DCC0015E for ; Tue, 18 Jul 2023 07:30:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C36EE6B0071; Tue, 18 Jul 2023 03:30:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BE5C88D0002; Tue, 18 Jul 2023 03:30:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD4C68D0001; Tue, 18 Jul 2023 03:30:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 9F1D06B0071 for ; Tue, 18 Jul 2023 03:30:35 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5E927C01AC for ; Tue, 18 Jul 2023 07:30:35 +0000 (UTC) X-FDA: 81023910030.11.6F64ECE Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf27.hostedemail.com (Postfix) with ESMTP id DBB6840004 for ; Tue, 18 Jul 2023 07:30:32 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=RTBhYUAt; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf27.hostedemail.com: domain of zhangpeng.00@bytedance.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=zhangpeng.00@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689665433; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=NbWcwbfvK0B2CiexlkN6if10zNcxlC4xkFU/ZNrDr6E=; b=8pNIzgBtbobVMh4KWsVZrxW7tvf9XiCv6wOL8obvrDceMMMBrJLWMzmUCaX4Yna1twKyzs d36gl3RN1mP/Hhhf04uN7S3xJwsU9KnmaOyiG88/Ee784Pwwa3hZEhIIg00xPuKtwXpeiz +skm29Q4DLW6TJqwqxx2qQtP57XwuJ4= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=RTBhYUAt; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf27.hostedemail.com: domain of zhangpeng.00@bytedance.com designates 209.85.214.169 as permitted sender) smtp.mailfrom=zhangpeng.00@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689665433; a=rsa-sha256; cv=none; b=xRzn1QjjVGXVueYjiABeqAymk3wGee1qikovKs2nuZeRksefnPg4wtmatqx+fgwpF1CNU5 USXnTv3azoLmINUYnAUwybGNWgDAqqF3W0q3XP9FO3TOUQDiwt4YWnZ4l2dvBp3a5e+f0w NLN4Vb1vO0QjDnjucR9jLD4JAw0CO+Y= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-1b9d80e33fbso32417335ad.0 for ; Tue, 18 Jul 2023 00:30:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1689665431; x=1692257431; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=NbWcwbfvK0B2CiexlkN6if10zNcxlC4xkFU/ZNrDr6E=; b=RTBhYUAtUyp1r+ibatHO3UBkkXi2vAoU4ftfUkJEdqZ5covGG2LrEdGkeRrfMoiIX/ 43mJHEVgctd1WA/TXx4AU3dUISW4JFhDW96XSiiP626EClSSIcG9LED4NPVB/S2a5Ksa bOVlGj3x3cQU5GPzvMsIenAsKRNn99M/qkfqnsGJMxBg3b/eekGh+trEZeNPpX1NU1rt LxHwc00N5Oo8Hr5rfhx6BW1k43dt+frWKDeVvJD/pPCfFUtZnJcr96m5aq9EEtgkECnm 232+Lr4wWnarcE7zN03K0xSPd9KmrSmZjDlqRzbt0CyAKYA83i34zm/p0Z1jLyYRSK15 ebWQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689665431; x=1692257431; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=NbWcwbfvK0B2CiexlkN6if10zNcxlC4xkFU/ZNrDr6E=; b=SXFT63EkmXhRR3XcoV3HJTjIoWBDEKCZqoDlS8yJVECu7xpT88Tynxroyv5E4Msugo lRrQ0qBtrbVAcr4OjZPRI33XMmlDLnP6Ub7/4X0wxITn5dmSpOfjHCLz0IOl7RqFdiaF 6Dz0xrf1uWd9v13oRPs1hMT1I97MahX5nN1rPMOxcfGtKo+ogXkXndkUk74wAoI8ADFp BMrNqL9ZW60wxIZuACAt41Wh6KU82KwYT7fCqAOMtpfeWrCxH/EppqFMm28fGZ1+Xz6b zOEJJ2Wfz9elotdfPwkph22SrhqkeYPJ4QY209Uwst1zMP2/8iZAyrtplZlvskRlm5VF 7UNw== X-Gm-Message-State: ABy/qLY6czkSEYpBe7Oxisughl6iH63Sd2egV/V06OsT/+87l4HanFQN r9A8bJ3AK2vgl1vnZ9Jij3sPFg== X-Google-Smtp-Source: APBJJlHdbX1rKsRTH2j6hYy8HdEIH2ErnTHEtAcfnkBvzSmi5oo+IMu9VNqZZekFlQsRPVqETGHKhQ== X-Received: by 2002:a17:902:ecc1:b0:1b8:9b78:df44 with SMTP id a1-20020a170902ecc100b001b89b78df44mr13733451plh.20.1689665431200; Tue, 18 Jul 2023 00:30:31 -0700 (PDT) Received: from GL4FX4PXWL.bytedance.net ([2408:8656:30f8:e020::2:d]) by smtp.gmail.com with ESMTPSA id x6-20020a1709027c0600b001b0358848b0sm1096315pll.161.2023.07.18.00.30.25 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 18 Jul 2023 00:30:30 -0700 (PDT) From: Peng Zhang To: glider@google.com, elver@google.com, dvyukov@google.com, akpm@linux-foundation.org Cc: kasan-dev@googlegroups.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, muchun.song@linux.dev, Peng Zhang Subject: [PATCH v3] mm: kfence: allocate kfence_metadata at runtime Date: Tue, 18 Jul 2023 15:30:19 +0800 Message-Id: <20230718073019.52513-1-zhangpeng.00@bytedance.com> X-Mailer: git-send-email 2.37.0 (Apple Git-136) MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Stat-Signature: 3uodoizgbwf78yb6tj4po19cyjqd7inc X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: DBB6840004 X-HE-Tag: 1689665432-57956 X-HE-Meta: U2FsdGVkX182r8aO9ITVBuuY0zXX62XfDlJKlo+1y40CkWm4ClHKCt7R3TEPkJ/AMuUsY6l+EPndQmr5yyhPqkvCGmikOy/XuKw0lkz/aC/88cKaum1CNg/V/l2GPMTulMaNGQjE2sxsuusnOv5FpXuaCEx2i3uPoeKPzOPKCOzf1zUGHvu2G/BoexDF3MSYli+iOwpuLCAaS2+Y7YZNr4v5doIM75X6o99E2+YdZW/4Bc2yu7Ls6HMUTFnQUWUoVP8byQnqDN7nsG1CMg3PufNodn4ShXoHZ7UT8mb+v5hfb2EP5G+UMRmupufNr9YchVQK+9kgmAFR6H5P8niT5h7Ph0eP45L/Ja6Jw0x8J3VGyNbExrLo/mIKXRBUt/sGX+6byHEYvTG2yuXC6AGqu1sJUVm/3jPU9tDwn4AqjFhqGhZTsCG+Izdw69wYHDXW3bIaiRU1h4cwMXV/r4EqMkxEJDfjfcS0JVPQ1hHIkj30IWJbqu5eeWDE2wGYMljqqyCzJuoZforuSpDsIKNYwBig/NuW2X34So7avNB/DFmfUcQS5qZaFRJdqwd9a6z/LtOxYMLrJP5+vYq4y8hCDBc0lI3EsooX5cuChd+1V7uKt7SeHmc6H44NCm+959exNv2S6UHFIT2+pOArKU82cmAvdV0Sk/ZjEuNdwyJ2riwIsgy1r17EqJ2w9d6oAuk1MGA73T4Fhkm/M8DJt5TiTs1zk4PLkIwuMY9pzEm09fas41M4+BFB1m/A068YNN0KCQKP66Mtnto68UEMQlwFaaKVhSFlvQR6BmmH4//Z/uAEPIBZQthNJ8Mk91FPSN2rDebrIvACs5+qpuFHP47wFu3gmj9s++qURZ2eLISCEJvvXlE3izoiuFMucGUElK1TVa+hBZZZC0O2cycqu7GngeOxByUZ++UJ1m4yPOPkcnBp1ygAwjwNhv2YaockfPiX4AjiQGeJnR7bSf6kX8Y opsDxkiu jdsk8t7I0y9IdSwjGkHEX1NUfpFzXjz4Gg3HghHfDuYlsRhi2uTjr6AiUlEe/8VDEEz9Btjx8lgwLbT6rNQLMaBCHwJ/6dHEeUuPEK2ucnwZBSwOQN0TaT0Mi+Uq4zxLdyRsHoaRRH2wanyaN+v4OB+l1t7osX7xNF055b2uuIKrVFJnrDuSwY+scwIB7MiUjycWaUWr9fKLDlozOM8zd2i7olD1xJ/aB9lVbse31pZDVlFVY61OTuOEPL7Wmj+8EZCCTNgCnXZKDW0aUzCswYBsiMxc7QylVnRbKkzaakSlB7jtYwf6z7RNI3Xb3oo2rPbJZMi3GlsLIedlaWooLardk5Z6YYLARpCVeWcXx7zDvrZykM3+F43JS+nZ52hgXYff0l0RKiQyMIi6SP7LbpS4z1VR9EomJ6haiTfxsik22SUUECO8zCwCdvtEELFyewVdy+DoAs8s2CFOKdqVnG3mAiDf+M5aRZgdjb3z+UIP3Eoqz5+AyVqPOb1WGelk7NbWjd3vGB9A/Npzwyjq1cHMo7w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: kfence_metadata is currently a static array. For the purpose of allocating scalable __kfence_pool, we first change it to runtime allocation of metadata. Since the size of an object of kfence_metadata is 1160 bytes, we can save at least 72 pages (with default 256 objects) without enabling kfence. Signed-off-by: Peng Zhang --- Changes since v2: - Fix missing renaming of kfence_alloc_pool. - Add __read_mostly for kfence_metadata and kfence_metadata_init. - Use smp_store_release() and smp_load_acquire() to access kfence_metadata. - Some tweaks to comments and git log. v1: https://lore.kernel.org/lkml/20230710032714.26200-1-zhangpeng.00@bytedance.com/ v2: https://lore.kernel.org/lkml/20230712081616.45177-1-zhangpeng.00@bytedance.com/ include/linux/kfence.h | 11 ++-- mm/kfence/core.c | 124 ++++++++++++++++++++++++++++------------- mm/kfence/kfence.h | 5 +- mm/mm_init.c | 2 +- 4 files changed, 97 insertions(+), 45 deletions(-) diff --git a/include/linux/kfence.h b/include/linux/kfence.h index 726857a4b680..401af4757514 100644 --- a/include/linux/kfence.h +++ b/include/linux/kfence.h @@ -59,15 +59,16 @@ static __always_inline bool is_kfence_address(const void *addr) } /** - * kfence_alloc_pool() - allocate the KFENCE pool via memblock + * kfence_alloc_pool_and_metadata() - allocate the KFENCE pool and KFENCE + * metadata via memblock */ -void __init kfence_alloc_pool(void); +void __init kfence_alloc_pool_and_metadata(void); /** * kfence_init() - perform KFENCE initialization at boot time * - * Requires that kfence_alloc_pool() was called before. This sets up the - * allocation gate timer, and requires that workqueues are available. + * Requires that kfence_alloc_pool_and_metadata() was called before. This sets + * up the allocation gate timer, and requires that workqueues are available. */ void __init kfence_init(void); @@ -223,7 +224,7 @@ bool __kfence_obj_info(struct kmem_obj_info *kpp, void *object, struct slab *sla #else /* CONFIG_KFENCE */ static inline bool is_kfence_address(const void *addr) { return false; } -static inline void kfence_alloc_pool(void) { } +static inline void kfence_alloc_pool_and_metadata(void) { } static inline void kfence_init(void) { } static inline void kfence_shutdown_cache(struct kmem_cache *s) { } static inline void *kfence_alloc(struct kmem_cache *s, size_t size, gfp_t flags) { return NULL; } diff --git a/mm/kfence/core.c b/mm/kfence/core.c index dad3c0eb70a0..6b526435886c 100644 --- a/mm/kfence/core.c +++ b/mm/kfence/core.c @@ -116,7 +116,15 @@ EXPORT_SYMBOL(__kfence_pool); /* Export for test modules. */ * backing pages (in __kfence_pool). */ static_assert(CONFIG_KFENCE_NUM_OBJECTS > 0); -struct kfence_metadata kfence_metadata[CONFIG_KFENCE_NUM_OBJECTS]; +struct kfence_metadata *kfence_metadata __read_mostly; + +/* + * If kfence_metadata is not NULL, it may be accessed by kfence_shutdown_cache(). + * So introduce kfence_metadata_init to initialize metadata, and then make + * kfence_metadata visible after initialization is successful. This prevents + * potential UAF or access to uninitialized metadata. + */ +static struct kfence_metadata *kfence_metadata_init __read_mostly; /* Freelist with available objects. */ static struct list_head kfence_freelist = LIST_HEAD_INIT(kfence_freelist); @@ -591,7 +599,7 @@ static unsigned long kfence_init_pool(void) __folio_set_slab(slab_folio(slab)); #ifdef CONFIG_MEMCG - slab->memcg_data = (unsigned long)&kfence_metadata[i / 2 - 1].objcg | + slab->memcg_data = (unsigned long)&kfence_metadata_init[i / 2 - 1].objcg | MEMCG_DATA_OBJCGS; #endif } @@ -610,7 +618,7 @@ static unsigned long kfence_init_pool(void) } for (i = 0; i < CONFIG_KFENCE_NUM_OBJECTS; i++) { - struct kfence_metadata *meta = &kfence_metadata[i]; + struct kfence_metadata *meta = &kfence_metadata_init[i]; /* Initialize metadata. */ INIT_LIST_HEAD(&meta->list); @@ -626,6 +634,12 @@ static unsigned long kfence_init_pool(void) addr += 2 * PAGE_SIZE; } + /* + * Make kfence_metadata visible only when initialization is successful. + * Otherwise, if the initialization fails and kfence_metadata is freed, + * it may cause UAF in kfence_shutdown_cache(). + */ + smp_store_release(&kfence_metadata, kfence_metadata_init); return 0; reset_slab: @@ -672,26 +686,10 @@ static bool __init kfence_init_pool_early(void) */ memblock_free_late(__pa(addr), KFENCE_POOL_SIZE - (addr - (unsigned long)__kfence_pool)); __kfence_pool = NULL; - return false; -} - -static bool kfence_init_pool_late(void) -{ - unsigned long addr, free_size; - addr = kfence_init_pool(); - - if (!addr) - return true; + memblock_free_late(__pa(kfence_metadata_init), KFENCE_METADATA_SIZE); + kfence_metadata_init = NULL; - /* Same as above. */ - free_size = KFENCE_POOL_SIZE - (addr - (unsigned long)__kfence_pool); -#ifdef CONFIG_CONTIG_ALLOC - free_contig_range(page_to_pfn(virt_to_page((void *)addr)), free_size / PAGE_SIZE); -#else - free_pages_exact((void *)addr, free_size); -#endif - __kfence_pool = NULL; return false; } @@ -841,19 +839,30 @@ static void toggle_allocation_gate(struct work_struct *work) /* === Public interface ===================================================== */ -void __init kfence_alloc_pool(void) +void __init kfence_alloc_pool_and_metadata(void) { if (!kfence_sample_interval) return; - /* if the pool has already been initialized by arch, skip the below. */ - if (__kfence_pool) - return; - - __kfence_pool = memblock_alloc(KFENCE_POOL_SIZE, PAGE_SIZE); - + /* + * If the pool has already been initialized by arch, there is no need to + * re-allocate the memory pool. + */ if (!__kfence_pool) + __kfence_pool = memblock_alloc(KFENCE_POOL_SIZE, PAGE_SIZE); + + if (!__kfence_pool) { pr_err("failed to allocate pool\n"); + return; + } + + /* The memory allocated by memblock has been zeroed out. */ + kfence_metadata_init = memblock_alloc(KFENCE_METADATA_SIZE, PAGE_SIZE); + if (!kfence_metadata_init) { + pr_err("failed to allocate metadata\n"); + memblock_free(__kfence_pool, KFENCE_POOL_SIZE); + __kfence_pool = NULL; + } } static void kfence_init_enable(void) @@ -895,33 +904,68 @@ void __init kfence_init(void) static int kfence_init_late(void) { - const unsigned long nr_pages = KFENCE_POOL_SIZE / PAGE_SIZE; + const unsigned long nr_pages_pool = KFENCE_POOL_SIZE / PAGE_SIZE; + const unsigned long nr_pages_meta = KFENCE_METADATA_SIZE / PAGE_SIZE; + unsigned long addr = (unsigned long)__kfence_pool; + unsigned long free_size = KFENCE_POOL_SIZE; + int err = -ENOMEM; + #ifdef CONFIG_CONTIG_ALLOC struct page *pages; - - pages = alloc_contig_pages(nr_pages, GFP_KERNEL, first_online_node, NULL); + pages = alloc_contig_pages(nr_pages_pool, GFP_KERNEL, first_online_node, + NULL); if (!pages) return -ENOMEM; + __kfence_pool = page_to_virt(pages); + pages = alloc_contig_pages(nr_pages_meta, GFP_KERNEL, first_online_node, + NULL); + if (pages) + kfence_metadata_init = page_to_virt(pages); #else - if (nr_pages > MAX_ORDER_NR_PAGES) { + if (nr_pages_pool > MAX_ORDER_NR_PAGES || + nr_pages_meta > MAX_ORDER_NR_PAGES) { pr_warn("KFENCE_NUM_OBJECTS too large for buddy allocator\n"); return -EINVAL; } + __kfence_pool = alloc_pages_exact(KFENCE_POOL_SIZE, GFP_KERNEL); if (!__kfence_pool) return -ENOMEM; + + kfence_metadata_init = alloc_pages_exact(KFENCE_METADATA_SIZE, GFP_KERNEL); #endif - if (!kfence_init_pool_late()) { - pr_err("%s failed\n", __func__); - return -EBUSY; + if (!kfence_metadata_init) + goto free_pool; + + memzero_explicit(kfence_metadata_init, KFENCE_METADATA_SIZE); + addr = kfence_init_pool(); + if (!addr) { + kfence_init_enable(); + kfence_debugfs_init(); + return 0; } - kfence_init_enable(); - kfence_debugfs_init(); + pr_err("%s failed\n", __func__); + free_size = KFENCE_POOL_SIZE - (addr - (unsigned long)__kfence_pool); + err = -EBUSY; - return 0; +#ifdef CONFIG_CONTIG_ALLOC + free_contig_range(page_to_pfn(virt_to_page((void *)kfence_metadata_init)), + nr_pages_meta); +free_pool: + free_contig_range(page_to_pfn(virt_to_page((void *)addr)), + free_size / PAGE_SIZE); +#else + free_pages_exact((void *)kfence_metadata_init, KFENCE_METADATA_SIZE); +free_pool: + free_pages_exact((void *)addr, free_size); +#endif + + kfence_metadata_init = NULL; + __kfence_pool = NULL; + return err; } static int kfence_enable_late(void) @@ -941,6 +985,10 @@ void kfence_shutdown_cache(struct kmem_cache *s) struct kfence_metadata *meta; int i; + /* Pairs with release in kfence_init_pool(). */ + if (!smp_load_acquire(&kfence_metadata)) + return; + for (i = 0; i < CONFIG_KFENCE_NUM_OBJECTS; i++) { bool in_use; diff --git a/mm/kfence/kfence.h b/mm/kfence/kfence.h index 392fb273e7bd..f46fbb03062b 100644 --- a/mm/kfence/kfence.h +++ b/mm/kfence/kfence.h @@ -102,7 +102,10 @@ struct kfence_metadata { #endif }; -extern struct kfence_metadata kfence_metadata[CONFIG_KFENCE_NUM_OBJECTS]; +#define KFENCE_METADATA_SIZE PAGE_ALIGN(sizeof(struct kfence_metadata) * \ + CONFIG_KFENCE_NUM_OBJECTS) + +extern struct kfence_metadata *kfence_metadata; static inline struct kfence_metadata *addr_to_metadata(unsigned long addr) { diff --git a/mm/mm_init.c b/mm/mm_init.c index 7f7f9c677854..3d0a63c75829 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -2721,7 +2721,7 @@ void __init mm_core_init(void) */ page_ext_init_flatmem(); mem_debugging_and_hardening_init(); - kfence_alloc_pool(); + kfence_alloc_pool_and_metadata(); report_meminit(); kmsan_init_shadow(); stack_depot_early_init(); -- 2.20.1