From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95619C3DA6E for ; Thu, 28 Dec 2023 13:13:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 931AE6B00CF; Thu, 28 Dec 2023 08:13:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8E1EF6B00D0; Thu, 28 Dec 2023 08:13:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 75C1A6B00D1; Thu, 28 Dec 2023 08:13:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 5F3846B00CF for ; Thu, 28 Dec 2023 08:13:04 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3E77140735 for ; Thu, 28 Dec 2023 13:13:04 +0000 (UTC) X-FDA: 81616267488.30.B3BFCD8 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) by imf30.hostedemail.com (Postfix) with ESMTP id 3E6ED80016 for ; Thu, 28 Dec 2023 13:13:02 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of artem.kuzin@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=artem.kuzin@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703769182; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Vztbv+gALk2cuisUQv7nNmPHj+pBTuKYD/i4A4F8a2Y=; b=VM8lvfRoFEcHy/23HDXLzmRl5DV0TG6Xf9SV2z7XVKqWqTu8+963jvCu5YLEj28Q6mO7lv 5YUB+ljdKZ5udufrrYs38ZJ/n8zjUf++BaDhSVxkw+JfFqZn/6ZRjsqGv7AloSi8BHpDBF Y0K/VmognTI74hTrUmmZpV0R/5X9PNg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703769182; a=rsa-sha256; cv=none; b=THEZ4AX+LahyjFda+LTtgwj36yyjP75qu3vwW+4G+3NvKld1IeyzpJwapiMsPuVS7rxXnz 0kKVdG83pf0Vs6D94fNZ0xdbz7rIjxzioR1D9yWOViGVLxqqzxsmFnGJ5OmdRVxXefTNQq hohWBjeQwK8yv9GRUtisIxe4xYasd+8= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of artem.kuzin@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=artem.kuzin@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com Received: from mail.maildlp.com (unknown [172.18.186.231]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4T185x5mvkz6J9Zb; Thu, 28 Dec 2023 21:11:17 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id 83DD7140B67; Thu, 28 Dec 2023 21:13:00 +0800 (CST) Received: from mscphis00060.huawei.com (10.123.65.147) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Thu, 28 Dec 2023 13:12:58 +0000 From: To: , , , , , , , , , , , , , CC: , , , , , , , , , , Subject: [PATCH RFC 11/12] x86: add kernel modules text and rodata replication support Date: Thu, 28 Dec 2023 21:10:55 +0800 Message-ID: <20231228131056.602411-12-artem.kuzin@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231228131056.602411-1-artem.kuzin@huawei.com> References: <20231228131056.602411-1-artem.kuzin@huawei.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.123.65.147] X-ClientProxiedBy: mscpeml500004.china.huawei.com (7.188.26.250) To lhrpeml500001.china.huawei.com (7.191.163.213) X-Stat-Signature: 9uqhmo1yyo1kysktr8betjsneaf58m48 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 3E6ED80016 X-Rspam-User: X-HE-Tag: 1703769182-672564 X-HE-Meta: U2FsdGVkX1+FsH2hy4GBl1bWI23KChMKlBmMw9169CrPq6eIl1W4Rfg5YtLeer9/mRyPx4M5Qzoaz6VQ+1HQCnQsOBZOHIK3HI4bcGnNbfKdpNexWR514saXBBRjP1HEdtaHTIKbhiavBuCC7iqrO0M0haZHjiCqvMNTXI4DpQyWnctwX/G4kznvgXrkXllwSMlTWiKQnVynOPt39uamxUflsSe0dX1gJUZduZ/qGp/X1JQ3W86+iafzeMQyPI3Ohi4cTVtxsokfuS8cCTkH0emChe1IWl/MliyUzvlr55R3yYT8eOc9ZnOzBI2j50Qptsx0LUNaza12kZyOKCIKISqocDF2IOHe9avJiznSdIy2WD6h25etbHcsL7UgyCYM8BbZ039Mu1BLMr63w4UwRP9GKk3KuaowUwXvv3U6k60SmU6Ro76KqpbhwA6cBVejsVXZ16tqvn62W+Dp4+cXq7gxIAin3uBJ0LinwVaLWN+9uzB4wWmZjCxKPWRgVrSfYikCAmkHY/s6s3V7zGHhr8rHbne7iA9ZX/+smcN4iUQzLSnsjRWJuxk2pb4eSJeUEcHYssQtM8PVsthhIJS8wkzNyBfyR6apKFwqEesAeRFVkdS6t49Iov/eybBvJmNuCqtM4nZo2mgBYmicJ1k9O2/OZJ32Xz57IVJU+uUEVvNXyoP8ToGG7fPE6GfKKeQqq2UeGTt1sXcZhIK+GSN5Koo/5oTK03q/1hysu2J6Sk0SWzCj2WbS24Wgaq79p7lfL7YrCPj6ZluMHdS0Jewc1RTvxR6gaNmc4FPjFo+81F7kW9zbTp666tiORVk6rV2SQJQfrrjqWSAzwTfz6HvGzrsdHBsCX1ExLWuCP5j56eir5bGWJZn2KWs+jfDkSgKVL+gkTiudCGNwC1yNSchhGrZwiOGPJoFYCf3uWhWcDy4zlREbPPnd5q7HvfQYwxYcd8gScqsV1lySwYAVjEQ rKyuwrq/ T2Wgxx5vtfqgD/vtkoWuGTBwq7gOsA1ynTf0xLyQpQVEfZaa6yAOGGZeDgOGUf/uesc1q+oc01Jmx5w2KCcdh9vfZstPJnEHOKPSELZE2yqSNpEzmzqkZhJfEDOlo5Z+UQySHW1fHyOJ+mHy/1MB5JbZkovH4R/oCKXsR8eTxA+ie1clgABZ1LlyhMONYUf5IWfucrNQ2mUN/3+EK8YZmWKw52usw0AU0qMMh6ef1ZxuWI4krxqiaD4RXPArePBLThPjgGVKwL7HtbvBAUYZ+EDSuYg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Artem Kuzin Co-developed-by: Nikita Panov Signed-off-by: Nikita Panov Co-developed-by: Alexander Grubnikov Signed-off-by: Alexander Grubnikov Signed-off-by: Artem Kuzin --- arch/x86/kernel/module.c | 35 ++++++++++++++++++++++++++++++----- include/linux/moduleloader.h | 10 ++++++++++ kernel/module/main.c | 8 ++++++++ kernel/module/strict_rwx.c | 14 +++++++------- 4 files changed, 55 insertions(+), 12 deletions(-) diff --git a/arch/x86/kernel/module.c b/arch/x86/kernel/module.c index 5f71a0cf4399..6d74d8c33293 100644 --- a/arch/x86/kernel/module.c +++ b/arch/x86/kernel/module.c @@ -65,28 +65,53 @@ static unsigned long int get_module_load_offset(void) } #endif -void *module_alloc(unsigned long size) +static void *__module_alloc(unsigned long size, unsigned long vm_flags, int nid) { gfp_t gfp_mask = GFP_KERNEL; void *p; if (PAGE_ALIGN(size) > MODULES_LEN) return NULL; - + /* + * In case replicas vmalloc should be able to unmap/reclaim them + * somehow. Due to this fact it is necessary to account suck pages + * separately. __vmalloc_not_replicated_per_pgd_range() function + * perform this accounting using internal vmalloc buffer with size + * equal to nr_pages * nr_online_nodes. + */ p = __vmalloc_node_range(size, MODULE_ALIGN, MODULES_VADDR + get_module_load_offset(), MODULES_END, gfp_mask, PAGE_KERNEL, - VM_FLUSH_RESET_PERMS | VM_DEFER_KMEMLEAK, - NUMA_NO_NODE, __builtin_return_address(0)); + VM_FLUSH_RESET_PERMS | VM_DEFER_KMEMLEAK | vm_flags, + nid, __builtin_return_address(0)); if (p && (kasan_alloc_module_shadow(p, size, gfp_mask) < 0)) { vfree(p); return NULL; } - return p; } +#ifdef CONFIG_KERNEL_REPLICATION +void *module_alloc(unsigned long size) +{ + return __module_alloc(size, VM_NUMA_SHARED, 0); +} + +void module_replicate_numa(void *ptr) +{ + gfp_t gfp_mask = GFP_KERNEL; + + __vmalloc_node_replicate_range(ptr, gfp_mask, + PAGE_KERNEL, VM_DEFER_KMEMLEAK); +} +#else +void *module_alloc(unsigned long size) +{ + return __module_alloc(size, 0, NUMA_NO_NODE); +} +#endif /*CONFIG_KERNEL_REPLICATION*/ + #ifdef CONFIG_X86_32 int apply_relocate(Elf32_Shdr *sechdrs, const char *strtab, diff --git a/include/linux/moduleloader.h b/include/linux/moduleloader.h index 001b2ce83832..722016d36bda 100644 --- a/include/linux/moduleloader.h +++ b/include/linux/moduleloader.h @@ -29,6 +29,16 @@ unsigned int arch_mod_section_prepend(struct module *mod, unsigned int section); sections. Returns NULL on failure. */ void *module_alloc(unsigned long size); +#ifndef CONFIG_KERNEL_REPLICATION +static inline void module_replicate_numa(void *ptr) +{ + (void) ptr; +} +#else +/* Replicate memory allocated in previous function*/ +void module_replicate_numa(void *ptr); +#endif /* CONFIG_KERNEL_REPLICATION */ + /* Free memory returned from module_alloc. */ void module_memfree(void *module_region); diff --git a/kernel/module/main.c b/kernel/module/main.c index 98fedfdb8db5..2ece8a7743de 100644 --- a/kernel/module/main.c +++ b/kernel/module/main.c @@ -2718,6 +2718,12 @@ static int add_unformed_module(struct module *mod) return err; } +static void module_replicate_rodata(struct module *mod) +{ + module_replicate_numa(mod->mem[MOD_TEXT].base); + module_replicate_numa(mod->mem[MOD_RODATA].base); +} + static int complete_formation(struct module *mod, struct load_info *info) { int err; @@ -2733,6 +2739,8 @@ static int complete_formation(struct module *mod, struct load_info *info) module_bug_finalize(info->hdr, info->sechdrs, mod); module_cfi_finalize(info->hdr, info->sechdrs, mod); + module_replicate_rodata(mod); + module_enable_ro(mod, false); module_enable_nx(mod); module_enable_x(mod); diff --git a/kernel/module/strict_rwx.c b/kernel/module/strict_rwx.c index a2b656b4e3d2..23abb3b0520b 100644 --- a/kernel/module/strict_rwx.c +++ b/kernel/module/strict_rwx.c @@ -29,7 +29,7 @@ static void module_set_memory(const struct module *mod, enum mod_mem_type type, void module_enable_x(const struct module *mod) { for_class_mod_mem_type(type, text) - module_set_memory(mod, type, set_memory_x); + module_set_memory(mod, type, numa_set_memory_x); } void module_enable_ro(const struct module *mod, bool after_init) @@ -41,13 +41,13 @@ void module_enable_ro(const struct module *mod, bool after_init) return; #endif - module_set_memory(mod, MOD_TEXT, set_memory_ro); - module_set_memory(mod, MOD_INIT_TEXT, set_memory_ro); - module_set_memory(mod, MOD_RODATA, set_memory_ro); - module_set_memory(mod, MOD_INIT_RODATA, set_memory_ro); + module_set_memory(mod, MOD_TEXT, numa_set_memory_ro); + module_set_memory(mod, MOD_INIT_TEXT, numa_set_memory_ro); + module_set_memory(mod, MOD_RODATA, numa_set_memory_ro); + module_set_memory(mod, MOD_INIT_RODATA, numa_set_memory_ro); if (after_init) - module_set_memory(mod, MOD_RO_AFTER_INIT, set_memory_ro); + module_set_memory(mod, MOD_RO_AFTER_INIT, numa_set_memory_ro); } void module_enable_nx(const struct module *mod) @@ -56,7 +56,7 @@ void module_enable_nx(const struct module *mod) return; for_class_mod_mem_type(type, data) - module_set_memory(mod, type, set_memory_nx); + module_set_memory(mod, type, numa_set_memory_nx); } int module_enforce_rwx_sections(Elf_Ehdr *hdr, Elf_Shdr *sechdrs, -- 2.34.1