From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48EF5C3DA6E for ; Thu, 28 Dec 2023 13:12:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0CD816B00BC; Thu, 28 Dec 2023 08:12:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 07D7A6B00C1; Thu, 28 Dec 2023 08:12:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E867F6B00C2; Thu, 28 Dec 2023 08:12:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D7CFB6B00C0 for ; Thu, 28 Dec 2023 08:12:48 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id AC5CA1202B3 for ; Thu, 28 Dec 2023 13:12:48 +0000 (UTC) X-FDA: 81616266816.03.F7092C0 Received: from frasgout.his.huawei.com (frasgout.his.huawei.com [185.176.79.56]) by imf13.hostedemail.com (Postfix) with ESMTP id 9F36220024 for ; Thu, 28 Dec 2023 13:12:46 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=none; spf=pass (imf13.hostedemail.com: domain of artem.kuzin@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=artem.kuzin@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1703769166; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=2nbrNQsGADTF0r2JizTnrK4so/2ZZ6DRCBpb3aeabuA=; b=zWzKVHL+xKVj9hcp5zp18A5l35tBGOVEA2Ejk0CJ2Q85t/GlOdQ85guPMJd0W0hNUDuk5S sjT9cS1H15m/b92V9BTsoI35zhM+5zAnozN3Qi8at0nvvKL/Veg/kkHsSsrLqPiFB3sfuO 2FEH8jtj9SfFZaxq1W3J6a8BUlhStjo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1703769166; a=rsa-sha256; cv=none; b=NL+RaMOcS0WY/nSIaxHb069BN8JOaVQQzJLtnnUAjV0hUyrRHMPXoXiUFhq7RleSMCPtgD tm0YS8jmRfSfGiF+OAhkfsKZg1cu23GdrZ8x4YuW+Has5vFKUiFzPgsBmahWDqAe2NJ6hl w74U8hrGhTSe1WDwQvrDSxk4XSWNcGM= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; spf=pass (imf13.hostedemail.com: domain of artem.kuzin@huawei.com designates 185.176.79.56 as permitted sender) smtp.mailfrom=artem.kuzin@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com Received: from mail.maildlp.com (unknown [172.18.186.216]) by frasgout.his.huawei.com (SkyGuard) with ESMTP id 4T184x2vMbz6K8j1; Thu, 28 Dec 2023 21:10:25 +0800 (CST) Received: from lhrpeml500001.china.huawei.com (unknown [7.191.163.213]) by mail.maildlp.com (Postfix) with ESMTPS id 3A851140DD5; Thu, 28 Dec 2023 21:12:42 +0800 (CST) Received: from mscphis00060.huawei.com (10.123.65.147) by lhrpeml500001.china.huawei.com (7.191.163.213) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2507.35; Thu, 28 Dec 2023 13:12:39 +0000 From: To: , , , , , , , , , , , , , CC: , , , , , , , , , , Subject: [PATCH RFC 01/12] mm: allow per-NUMA node local PUD/PMD allocation Date: Thu, 28 Dec 2023 21:10:45 +0800 Message-ID: <20231228131056.602411-2-artem.kuzin@huawei.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231228131056.602411-1-artem.kuzin@huawei.com> References: <20231228131056.602411-1-artem.kuzin@huawei.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.123.65.147] X-ClientProxiedBy: mscpeml500004.china.huawei.com (7.188.26.250) To lhrpeml500001.china.huawei.com (7.191.163.213) X-Rspamd-Queue-Id: 9F36220024 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 4k33hnkfuf7ze1tghht95y4edxpgmdoj X-HE-Tag: 1703769166-96499 X-HE-Meta: U2FsdGVkX19xV17Jtn0aUdM3khNZ1olWf7MvUYsoC+DkhAXkLrMNgewui4eot8VMH8bFeiR/eI+3dTtI5b6eViRlFGavxQHHQyiG0exuzcGDmSwKIfU3+GHc2wOV7Xb8pedN1h4YLTRMduQnTSAY/YOHsziP7yzmfSjIkdYjbKHmh5s6l5+hLoaRwYQ0CItbvuy+cSvx0hzZusUAoUu4YtDAh8cQCweJD82wGzumIPKrmTCKgv2qLymXv+aTbGg7DNLF9yiNCqADo7UaE3h8LZEY3xaYcIDdKm+8Y0bqc3sYR9cR+64JJ0lYl+EFW5Z/BKAZwlUvhB1sLVN/fyAqKZOwa8cS5FbDod1ZWVlbgsPlKeZckhLNwvouXpgfPT1XKx5tR7qomnmv6UyRKqCX+vY5S2GXF5eV8LGcu5tQUu7MDYWt3SJMPDKtWCpdeRCCxoY44JkPwzDMcJrsz9OmFNDBSfpwfN8fzwt5l1ueLZutoW+Z3VSPVXK4RAr6QZYKlFIwP//hAGGT4yNMYgDP7Dkk+6vau7bQiDSBZBKVCjhrjC19HLX2XPVYp3bq149OY3ThxjM5aMJCzjLNPMYrVBqCji0MLBMM35ev11EZlkgFw/jlNiWR06QIHgzqhjMYLtlElsAFfkgh4roBxM6z09JYFKeVpuABzTIv13qkWoWFXUwy9Vcu9jwAyH0j9RASgjmf7oUTu1V1r3H7rKx2cOXiH6uYpmIdYuR3cUztIJw+37LyVoOAQyRkN5H0Iw2J7lOQG/yZ2vvMCcLDMU0TkJpZcZNVDI7MheKQKnXJFsLYdf1y33YRbg8IaggoCQK3jQ3JvdyrFeee8be39JtITSEAo8XluvCE01Onnu7sWVXh/oM9iV72DVHNld5xRy13MN+4gunx7aMcztwykS+MVRvIz8plzVvbENpJSsqe+DkvWxIRdZdZmkjbcJ5ctt4aeZV8+ReAi9b399TL8aj jYex8l4I guyirtqnxx5z/Mdtplv8A/paJ3P13295x+drC3hcYiwFYfazdcj6tiyj9L+UGZUVJpY48MSiKtoW4PAhKXLGM/csn/2ahHmKfG38zCO7MAYrXu7xK1/xe0i+t4FIpcAW15xoKI2bMHJV6BnigTtRvQKne0rTPFubxe6gd+74DUrTozkFc/oYDwhzvoqTvhSHqXU4jyIWRIQ4l6Ho0IdBFV2QOv45yb07bNA+Yd7C8qeY+UATeusqdVNHBiiOs8Dr2QEPFYmdtjmOZ84wtbBnJl2oJIg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Artem Kuzin Co-developed-by: Nikita Panov Signed-off-by: Nikita Panov Co-developed-by: Alexander Grubnikov Signed-off-by: Alexander Grubnikov Signed-off-by: Artem Kuzin --- include/asm-generic/pgalloc.h | 34 ++++++++++++++++++++++++++++++++++ include/linux/gfp.h | 2 ++ mm/page_alloc.c | 18 ++++++++++++++++++ 3 files changed, 54 insertions(+) diff --git a/include/asm-generic/pgalloc.h b/include/asm-generic/pgalloc.h index a7cf825befae..6364375388bf 100644 --- a/include/asm-generic/pgalloc.h +++ b/include/asm-generic/pgalloc.h @@ -132,6 +132,24 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) } return (pmd_t *)page_address(page); } + +static inline pmd_t *pmd_alloc_one_node(unsigned int nid, + struct mm_struct *mm, unsigned long addr) +{ + struct page *page; + gfp_t gfp = GFP_PGTABLE_USER; + + if (mm == &init_mm) + gfp = GFP_PGTABLE_KERNEL; + page = alloc_pages_node(nid, gfp, 0); + if (!page) + return NULL; + if (!pgtable_pmd_page_ctor(page)) { + __free_pages(page, 0); + return NULL; + } + return (pmd_t *)page_address(page); +} #endif #ifndef __HAVE_ARCH_PMD_FREE @@ -156,6 +174,16 @@ static inline pud_t *__pud_alloc_one(struct mm_struct *mm, unsigned long addr) return (pud_t *)get_zeroed_page(gfp); } +static inline pud_t *__pud_alloc_one_node(unsigned int nid, + struct mm_struct *mm, unsigned long addr) +{ + gfp_t gfp = GFP_PGTABLE_USER; + + if (mm == &init_mm) + gfp = GFP_PGTABLE_KERNEL; + return (pud_t *)get_zeroed_page_node(nid, gfp); +} + #ifndef __HAVE_ARCH_PUD_ALLOC_ONE /** * pud_alloc_one - allocate a page for PUD-level page table @@ -170,6 +198,12 @@ static inline pud_t *pud_alloc_one(struct mm_struct *mm, unsigned long addr) { return __pud_alloc_one(mm, addr); } + +static inline pud_t *pud_alloc_one_node(unsigned int nid, + struct mm_struct *mm, unsigned long addr) +{ + return __pud_alloc_one_node(nid, mm, addr); +} #endif static inline void __pud_free(struct mm_struct *mm, pud_t *pud) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index 665f06675c83..6ee0004b9774 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -288,6 +288,8 @@ static inline struct page *alloc_page_vma(gfp_t gfp, extern unsigned long __get_free_pages(gfp_t gfp_mask, unsigned int order); extern unsigned long get_zeroed_page(gfp_t gfp_mask); +extern unsigned long __get_free_pages_node(unsigned int nid, gfp_t gfp_mask, unsigned int order); +extern unsigned long get_zeroed_page_node(unsigned int nid, gfp_t gfp_mask); void *alloc_pages_exact(size_t size, gfp_t gfp_mask) __alloc_size(1); void free_pages_exact(void *virt, size_t size); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 7d3460c7a480..dc8f4a57d8b1 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4537,6 +4537,24 @@ unsigned long get_zeroed_page(gfp_t gfp_mask) } EXPORT_SYMBOL(get_zeroed_page); +unsigned long __get_free_pages_node(unsigned int nid, gfp_t gfp_mask, + unsigned int order) +{ + struct page *page; + + page = alloc_pages_node(nid, gfp_mask & ~__GFP_HIGHMEM, order); + if (!page) + return 0; + return (unsigned long) page_address(page); +} +EXPORT_SYMBOL(__get_free_pages_node); + +unsigned long get_zeroed_page_node(unsigned int nid, gfp_t gfp_mask) +{ + return __get_free_pages_node(nid, gfp_mask | __GFP_ZERO, 0); +} +EXPORT_SYMBOL(get_zeroed_page_node); + /** * __free_pages - Free pages allocated with alloc_pages(). * @page: The page pointer returned from alloc_pages(). -- 2.34.1