From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 208A3C63697 for ; Fri, 13 Nov 2020 10:40:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8E3442224B for ; Fri, 13 Nov 2020 10:40:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8E3442224B Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=suse.cz Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2A10B6B0098; Fri, 13 Nov 2020 05:40:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E67F06B009A; Fri, 13 Nov 2020 05:40:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BF62D6B0096; Fri, 13 Nov 2020 05:40:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0020.hostedemail.com [216.40.44.20]) by kanga.kvack.org (Postfix) with ESMTP id 7B1836B0095 for ; Fri, 13 Nov 2020 05:40:50 -0500 (EST) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 189691EE6 for ; Fri, 13 Nov 2020 10:40:50 +0000 (UTC) X-FDA: 77479051860.12.foot99_490f4952730e Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id E5C8018009D29 for ; Fri, 13 Nov 2020 10:40:49 +0000 (UTC) X-HE-Tag: foot99_490f4952730e X-Filterd-Recvd-Size: 8654 Received: from mx2.suse.de (mx2.suse.de [195.135.220.15]) by imf30.hostedemail.com (Postfix) with ESMTP for ; Fri, 13 Nov 2020 10:40:49 +0000 (UTC) X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.221.27]) by mx2.suse.de (Postfix) with ESMTP id 04153ACA8; Fri, 13 Nov 2020 10:40:48 +0000 (UTC) From: Vlastimil Babka To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Alexander Potapenko , Kees Cook , Michal Hocko , David Hildenbrand , Mateusz Nosek , Laura Abbott , Vlastimil Babka , Mike Rapoport Subject: [PATCH v3 1/5] mm, page_alloc: do not rely on the order of page_poison and init_on_alloc/free parameters Date: Fri, 13 Nov 2020 11:40:29 +0100 Message-Id: <20201113104033.22907-2-vbabka@suse.cz> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20201113104033.22907-1-vbabka@suse.cz> References: <20201113104033.22907-1-vbabka@suse.cz> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Enabling page_poison=3D1 together with init_on_alloc=3D1 or init_on_free=3D= 1 produces a warning in dmesg that page_poison takes precedence. However, as these warnings are printed in early_param handlers for init_on_alloc/free, they= are not printed if page_poison is enabled later on the command line (handlers= are called in the order of their parameters), or when init_on_alloc/free is a= lways enabled by the respective config option - before the page_poison early pa= ram handler is called, it is not considered to be enabled. This is inconsiste= nt. We can remove the dependency on order by making the init_on_* parameters = only set a boolean variable, and postponing the evaluation after all early par= ams have been processed. Introduce a new init_mem_debugging_and_hardening() function for that, and move the related debug_pagealloc processing there = as well. As a result init_mem_debugging_and_hardening() knows always accurately if init_on_* and/or page_poison options were enabled. Thus we can also optim= ize want_init_on_alloc() and want_init_on_free(). We don't need to check page_poisoning_enabled() there, we can instead not enable the init_on_* s= tatic keys at all, if page poisoning is enabled. This results in a simpler and = more effective code. Signed-off-by: Vlastimil Babka Reviewed-by: David Hildenbrand Reviewed-by: Mike Rapoport --- include/linux/mm.h | 20 ++--------- init/main.c | 2 +- mm/page_alloc.c | 88 ++++++++++++++++++++++------------------------ 3 files changed, 46 insertions(+), 64 deletions(-) diff --git a/include/linux/mm.h b/include/linux/mm.h index 24f47e140a4c..82ab5c894d94 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2860,6 +2860,7 @@ extern int apply_to_existing_page_range(struct mm_s= truct *mm, unsigned long address, unsigned long size, pte_fn_t fn, void *data); =20 +extern void init_mem_debugging_and_hardening(void); #ifdef CONFIG_PAGE_POISONING extern bool page_poisoning_enabled(void); extern void kernel_poison_pages(struct page *page, int numpages, int ena= ble); @@ -2869,35 +2870,20 @@ static inline void kernel_poison_pages(struct pag= e *page, int numpages, int enable) { } #endif =20 -#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON -DECLARE_STATIC_KEY_TRUE(init_on_alloc); -#else DECLARE_STATIC_KEY_FALSE(init_on_alloc); -#endif static inline bool want_init_on_alloc(gfp_t flags) { - if (static_branch_unlikely(&init_on_alloc) && - !page_poisoning_enabled()) + if (static_branch_unlikely(&init_on_alloc)) return true; return flags & __GFP_ZERO; } =20 -#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON -DECLARE_STATIC_KEY_TRUE(init_on_free); -#else DECLARE_STATIC_KEY_FALSE(init_on_free); -#endif static inline bool want_init_on_free(void) { - return static_branch_unlikely(&init_on_free) && - !page_poisoning_enabled(); + return static_branch_unlikely(&init_on_free); } =20 -#ifdef CONFIG_DEBUG_PAGEALLOC -extern void init_debug_pagealloc(void); -#else -static inline void init_debug_pagealloc(void) {} -#endif extern bool _debug_pagealloc_enabled_early; DECLARE_STATIC_KEY_FALSE(_debug_pagealloc_enabled); =20 diff --git a/init/main.c b/init/main.c index 7262d0a8861b..524089384155 100644 --- a/init/main.c +++ b/init/main.c @@ -816,7 +816,7 @@ static void __init mm_init(void) * bigger than MAX_ORDER unless SPARSEMEM. */ page_ext_init_flatmem(); - init_debug_pagealloc(); + init_mem_debugging_and_hardening(); kfence_alloc_pool(); report_meminit(); mem_init(); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 63d8d8b72c10..567060c2ad83 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -165,53 +165,26 @@ unsigned long totalcma_pages __read_mostly; =20 int percpu_pagelist_fraction; gfp_t gfp_allowed_mask __read_mostly =3D GFP_BOOT_MASK; -#ifdef CONFIG_INIT_ON_ALLOC_DEFAULT_ON -DEFINE_STATIC_KEY_TRUE(init_on_alloc); -#else DEFINE_STATIC_KEY_FALSE(init_on_alloc); -#endif EXPORT_SYMBOL(init_on_alloc); =20 -#ifdef CONFIG_INIT_ON_FREE_DEFAULT_ON -DEFINE_STATIC_KEY_TRUE(init_on_free); -#else DEFINE_STATIC_KEY_FALSE(init_on_free); -#endif EXPORT_SYMBOL(init_on_free); =20 +static bool _init_on_alloc_enabled_early __read_mostly + =3D IS_ENABLED(CONFIG_INIT_ON_ALLOC_DEFAULT_ON); static int __init early_init_on_alloc(char *buf) { - int ret; - bool bool_result; =20 - ret =3D kstrtobool(buf, &bool_result); - if (ret) - return ret; - if (bool_result && page_poisoning_enabled()) - pr_info("mem auto-init: CONFIG_PAGE_POISONING is on, will take precede= nce over init_on_alloc\n"); - if (bool_result) - static_branch_enable(&init_on_alloc); - else - static_branch_disable(&init_on_alloc); - return 0; + return kstrtobool(buf, &_init_on_alloc_enabled_early); } early_param("init_on_alloc", early_init_on_alloc); =20 +static bool _init_on_free_enabled_early __read_mostly + =3D IS_ENABLED(CONFIG_INIT_ON_FREE_DEFAULT_ON); static int __init early_init_on_free(char *buf) { - int ret; - bool bool_result; - - ret =3D kstrtobool(buf, &bool_result); - if (ret) - return ret; - if (bool_result && page_poisoning_enabled()) - pr_info("mem auto-init: CONFIG_PAGE_POISONING is on, will take precede= nce over init_on_free\n"); - if (bool_result) - static_branch_enable(&init_on_free); - else - static_branch_disable(&init_on_free); - return 0; + return kstrtobool(buf, &_init_on_free_enabled_early); } early_param("init_on_free", early_init_on_free); =20 @@ -728,19 +701,6 @@ static int __init early_debug_pagealloc(char *buf) } early_param("debug_pagealloc", early_debug_pagealloc); =20 -void init_debug_pagealloc(void) -{ - if (!debug_pagealloc_enabled()) - return; - - static_branch_enable(&_debug_pagealloc_enabled); - - if (!debug_guardpage_minorder()) - return; - - static_branch_enable(&_debug_guardpage_enabled); -} - static int __init debug_guardpage_minorder_setup(char *buf) { unsigned long res; @@ -792,6 +752,42 @@ static inline void clear_page_guard(struct zone *zon= e, struct page *page, unsigned int order, int migratetype) {} #endif =20 +/* + * Enable static keys related to various memory debugging and hardening = options. + * Some override others, and depend on early params that are evaluated i= n the + * order of appearance. So we need to first gather the full picture of w= hat was + * enabled, and then make decisions. + */ +void init_mem_debugging_and_hardening(void) +{ + if (_init_on_alloc_enabled_early) { + if (page_poisoning_enabled()) + pr_info("mem auto-init: CONFIG_PAGE_POISONING is on, " + "will take precedence over init_on_alloc\n"); + else + static_branch_enable(&init_on_alloc); + } + if (_init_on_free_enabled_early) { + if (page_poisoning_enabled()) + pr_info("mem auto-init: CONFIG_PAGE_POISONING is on, " + "will take precedence over init_on_free\n"); + else + static_branch_enable(&init_on_free); + } + +#ifdef CONFIG_DEBUG_PAGEALLOC + if (!debug_pagealloc_enabled()) + return; + + static_branch_enable(&_debug_pagealloc_enabled); + + if (!debug_guardpage_minorder()) + return; + + static_branch_enable(&_debug_guardpage_enabled); +#endif +} + static inline void set_buddy_order(struct page *page, unsigned int order= ) { set_page_private(page, order); --=20 2.29.2