From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E01BEC54756 for ; Thu, 22 May 2025 09:03:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7359D6B0085; Thu, 22 May 2025 05:03:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 70D6E6B0088; Thu, 22 May 2025 05:03:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 623076B008A; Thu, 22 May 2025 05:03:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 4728D6B0085 for ; Thu, 22 May 2025 05:03:08 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id C95891A0C27 for ; Thu, 22 May 2025 09:03:07 +0000 (UTC) X-FDA: 83469954414.10.925C433 Received: from mout-p-202.mailbox.org (mout-p-202.mailbox.org [80.241.56.172]) by imf15.hostedemail.com (Postfix) with ESMTP id 2C206A0013 for ; Thu, 22 May 2025 09:03:05 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=none; spf=pass (imf15.hostedemail.com: domain of kernel@pankajraghav.com designates 80.241.56.172 as permitted sender) smtp.mailfrom=kernel@pankajraghav.com; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=samsung.com (policy=none) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1747904586; a=rsa-sha256; cv=none; b=v8hzaX5QMeuyUN4fWrssUaedzFJdKGh9bHCLaoM6DLPRQPE4yo8GuG6WtB50H5WSsRFxZ+ k0mG2cgp9gv5mqhfMQ6OPiHBhq0fqqZqz7CNqK1ky0oJjjt9t3ouRQe+tIoY/mtqtO3q/Y yk6//ZemOdeT9aGv6ywEr97IfXT8vEw= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=none; spf=pass (imf15.hostedemail.com: domain of kernel@pankajraghav.com designates 80.241.56.172 as permitted sender) smtp.mailfrom=kernel@pankajraghav.com; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=samsung.com (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1747904586; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zifPnkz1Ysc1GMFY2474ATFeY+h/iA+C8ojdXs9xI9g=; b=W9L6aQD2J888BcEvGuUjCC4va7atkO/8VMluRi5CTJEQOt9rCeZn6H4wk6Mmd3NPzMUqxh ew/Y6ToIJi1y3MNcIRmolnohYrOeTAUNbZOgeJq+0OH8IvFWgViC2eDBpOawYvY+p6O0s7 gKNZjcxvVyIdSRE5JMz6rk4j61ljroM= Received: from smtp2.mailbox.org (smtp2.mailbox.org [10.196.197.2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by mout-p-202.mailbox.org (Postfix) with ESMTPS id 4b32Pf4dHZz9scV; Thu, 22 May 2025 11:03:02 +0200 (CEST) From: Pankaj Raghav To: Suren Baghdasaryan , Vlastimil Babka , Ryan Roberts , Mike Rapoport , Michal Hocko , Thomas Gleixner , Nico Pache , Dev Jain , Baolin Wang , Borislav Petkov , Ingo Molnar , "H . Peter Anvin" , Zi Yan , Dave Hansen , David Hildenbrand , Lorenzo Stoakes , Andrew Morton , "Liam R . Howlett" , Jens Axboe Cc: linux-block@vger.kernel.org, linux-fsdevel@vger.kernel.org, "Darrick J . Wong" , gost.dev@samsung.com, kernel@pankajraghav.com, hch@lst.de, linux-kernel@vger.kernel.org, linux-mm@kvack.org, willy@infradead.org, x86@kernel.org, mcgrof@kernel.org, Pankaj Raghav Subject: [RFC v2 1/2] mm: add THP_HUGE_ZERO_PAGE_ALWAYS config option Date: Thu, 22 May 2025 11:02:42 +0200 Message-ID: <20250522090243.758943-2-p.raghav@samsung.com> In-Reply-To: <20250522090243.758943-1-p.raghav@samsung.com> References: <20250522090243.758943-1-p.raghav@samsung.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 2C206A0013 X-Stat-Signature: szx4gis9sxekdroyx5zor5dr15gmekwd X-Rspam-User: X-HE-Tag: 1747904585-159018 X-HE-Meta: U2FsdGVkX19kAoyZ8CcfDLQHxOIMag+R7MJlBDk41HhtZJvFxGeucqcjXuQ6IVfeJhhqk3gc6DpZuhDVLc1nL8Ox4vMn5G5l2ubqR3d+H0gcN5YNM/PmtBP2agzbkCi58ywUmo7Am8UBo6cXXJsuLem5iJo1GpXkAXopB5eoni6F/L2DcchZs7HKW+a7V3aYGgouEvWRmSgbSUOuIUYiW4y0alHcUxkWAxmyVibdJ0rKlUy+zxkiQkd0kVhn3TwXDx6ezIsgVxRyuc9U6X3JzT3TIIOICpr43azcfS82WoLGR6hDoJE7hoN5cd8qVjQ2GkL294+dTIrQ0g/XxnDI0KOrxtDdq7IkIkrS5EZ/gX2kLFW96VTPaqbgfIMtkG6VG6ZCYAdB2wV9r/xC/gX8sLR8Psl+7FzuBib8/JRoyVr7hhguuJqd6LRIi1kjheDt6FcNNwQXFD5jgFHB1vAuJlrKU4wkIKVzA5p7O7ljmhWVWUqkzJYq91UiAK+7Lgb960CMk30e00B2EmLLNTkr8xRQfSKT/un46cXdKU92RPFzSENgNks2nLfrZ3AACPX+1G2Bx3kJxiSon035ylqh5x4E2xug3uutmG+i0iYs0drEeuLmFCaVizqfkcwHSSu608JNfjfyCQk8Mnv6elKlE210lHTQXq//E8mQ6Xx0p/Qg0kZNFDDYc9lR3Y1VdDOtzZhipDNUu6drkxCMN2DBZQQCkAlsOhctBxSQTaLZ13m4tXMj+njBiEC5nQ3VWmtVwzaS/yNyjFXXG0qcUYiOms2aFrh9bUI8hF2KzjgOk8QR3004Lmfeb4VkEAqZip0cBkpisKpYGE8X1XjTe+ITh+RoBm6sErOYXeqqrI12gQiWhkbjPAuWCeiRF9W0SlyhKUtOKwfKtRa9BvwEpiTQcsfXWIT8eCQl0UTPSTc+MGCTy1T8XY4l6CwmO+iQMRmCBuMigUc6Pj02wxB51ik XRa1fhed OBdO43z0Z6vV5qEgLpXs0JulBAjCx2NPvUW4asn40qCHcM9zJMzAS8GcAn6HvCOcdppqnl1QSyze8pbYOzwVKxJiklgPmFaVEuqe7XkzkBoGTgSioy+bxgOmoqCyGpWZBmhle X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There are many places in the kernel where we need to zeroout larger chunks but the maximum segment we can zeroout at a time by ZERO_PAGE is limited by PAGE_SIZE. This is especially annoying in block devices and filesystems where we attach multiple ZERO_PAGEs to the bio in different bvecs. With multipage bvec support in block layer, it is much more efficient to send out larger zero pages as a part of single bvec. This concern was raised during the review of adding LBS support to XFS[1][2]. Usually huge_zero_folio is allocated on demand, and it will be deallocated by the shrinker if there are no users of it left. Add a config option THP_HUGE_ZERO_PAGE_ALWAYS that will always allocate the huge_zero_folio, and it will never be freed. This makes using the huge_zero_folio without having to pass any mm struct and call put_folio in the destructor. We can enable it by default for x86_64 where the PMD size is 2M. It is good compromise between the memory and efficiency. As a THP zero page might be wasteful for architectures with bigger page sizes, let's not enable it for them. [1] https://lore.kernel.org/linux-xfs/20231027051847.GA7885@lst.de/ [2] https://lore.kernel.org/linux-xfs/ZitIK5OnR7ZNY0IG@infradead.org/ Suggested-by: David Hildenbrand Signed-off-by: Pankaj Raghav --- arch/x86/Kconfig | 1 + mm/Kconfig | 12 +++++++++ mm/huge_memory.c | 63 ++++++++++++++++++++++++++++++++++++++---------- 3 files changed, 63 insertions(+), 13 deletions(-) diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index 055204dc211d..2e1527580746 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -152,6 +152,7 @@ config X86 select ARCH_WANT_OPTIMIZE_HUGETLB_VMEMMAP if X86_64 select ARCH_WANT_HUGETLB_VMEMMAP_PREINIT if X86_64 select ARCH_WANTS_THP_SWAP if X86_64 + select ARCH_WANTS_THP_ZERO_PAGE_ALWAYS if X86_64 select ARCH_HAS_PARANOID_L1D_FLUSH select BUILDTIME_TABLE_SORT select CLKEVT_I8253 diff --git a/mm/Kconfig b/mm/Kconfig index bd08e151fa1b..a2994e7d55ba 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -823,6 +823,9 @@ config ARCH_WANT_GENERAL_HUGETLB config ARCH_WANTS_THP_SWAP def_bool n +config ARCH_WANTS_THP_ZERO_PAGE_ALWAYS + def_bool n + config MM_ID def_bool n @@ -895,6 +898,15 @@ config READ_ONLY_THP_FOR_FS support of file THPs will be developed in the next few release cycles. +config THP_ZERO_PAGE_ALWAYS + def_bool y + depends on TRANSPARENT_HUGEPAGE && ARCH_WANTS_THP_ZERO_PAGE_ALWAYS + help + Typically huge_zero_folio, which is a THP of zeroes, is allocated + on demand and deallocated when not in use. This option will always + allocate huge_zero_folio for zeroing and it is never deallocated. + Not suitable for memory constrained systems. + config NO_PAGE_MAPCOUNT bool "No per-page mapcount (EXPERIMENTAL)" help diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d3e66136e41a..1a0556ca3839 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -247,9 +247,16 @@ static void put_huge_zero_page(void) BUG_ON(atomic_dec_and_test(&huge_zero_refcount)); } +/* + * If THP_ZERO_PAGE_ALWAYS is enabled, @mm can be NULL, i.e, the huge_zero_folio + * is not associated with any mm_struct. + */ struct folio *mm_get_huge_zero_folio(struct mm_struct *mm) { - if (test_bit(MMF_HUGE_ZERO_PAGE, &mm->flags)) + if (!IS_ENABLED(CONFIG_THP_ZERO_PAGE_ALWAYS) && !mm) + return NULL; + + if (IS_ENABLED(CONFIG_THP_ZERO_PAGE_ALWAYS) || test_bit(MMF_HUGE_ZERO_PAGE, &mm->flags)) return READ_ONCE(huge_zero_folio); if (!get_huge_zero_page()) @@ -263,6 +270,9 @@ struct folio *mm_get_huge_zero_folio(struct mm_struct *mm) void mm_put_huge_zero_folio(struct mm_struct *mm) { + if (IS_ENABLED(CONFIG_THP_ZERO_PAGE_ALWAYS)) + return; + if (test_bit(MMF_HUGE_ZERO_PAGE, &mm->flags)) put_huge_zero_page(); } @@ -274,14 +284,21 @@ static unsigned long shrink_huge_zero_page_count(struct shrinker *shrink, return atomic_read(&huge_zero_refcount) == 1 ? HPAGE_PMD_NR : 0; } +static void _put_huge_zero_folio(void) +{ + struct folio *zero_folio; + + zero_folio = xchg(&huge_zero_folio, NULL); + BUG_ON(zero_folio == NULL); + WRITE_ONCE(huge_zero_pfn, ~0UL); + folio_put(zero_folio); +} + static unsigned long shrink_huge_zero_page_scan(struct shrinker *shrink, struct shrink_control *sc) { if (atomic_cmpxchg(&huge_zero_refcount, 1, 0) == 1) { - struct folio *zero_folio = xchg(&huge_zero_folio, NULL); - BUG_ON(zero_folio == NULL); - WRITE_ONCE(huge_zero_pfn, ~0UL); - folio_put(zero_folio); + _put_huge_zero_folio(); return HPAGE_PMD_NR; } @@ -850,10 +867,6 @@ static inline void hugepage_exit_sysfs(struct kobject *hugepage_kobj) static int __init thp_shrinker_init(void) { - huge_zero_page_shrinker = shrinker_alloc(0, "thp-zero"); - if (!huge_zero_page_shrinker) - return -ENOMEM; - deferred_split_shrinker = shrinker_alloc(SHRINKER_NUMA_AWARE | SHRINKER_MEMCG_AWARE | SHRINKER_NONSLAB, @@ -863,14 +876,21 @@ static int __init thp_shrinker_init(void) return -ENOMEM; } - huge_zero_page_shrinker->count_objects = shrink_huge_zero_page_count; - huge_zero_page_shrinker->scan_objects = shrink_huge_zero_page_scan; - shrinker_register(huge_zero_page_shrinker); - deferred_split_shrinker->count_objects = deferred_split_count; deferred_split_shrinker->scan_objects = deferred_split_scan; shrinker_register(deferred_split_shrinker); + if (IS_ENABLED(CONFIG_THP_ZERO_PAGE_ALWAYS)) + return 0; + + huge_zero_page_shrinker = shrinker_alloc(0, "thp-zero"); + if (!huge_zero_page_shrinker) + return -ENOMEM; + + huge_zero_page_shrinker->count_objects = shrink_huge_zero_page_count; + huge_zero_page_shrinker->scan_objects = shrink_huge_zero_page_scan; + shrinker_register(huge_zero_page_shrinker); + return 0; } @@ -880,6 +900,17 @@ static void __init thp_shrinker_exit(void) shrinker_free(deferred_split_shrinker); } +static int __init huge_zero_page_init(void) { + + if (!IS_ENABLED(CONFIG_THP_ZERO_PAGE_ALWAYS)) + return 0; + + if (!get_huge_zero_page()) { + return -ENOMEM; + } + return 0; +} + static int __init hugepage_init(void) { int err; @@ -903,6 +934,10 @@ static int __init hugepage_init(void) if (err) goto err_slab; + err = huge_zero_page_init(); + if (err) + goto err_huge_zero_page; + err = thp_shrinker_init(); if (err) goto err_shrinker; @@ -925,6 +960,8 @@ static int __init hugepage_init(void) err_khugepaged: thp_shrinker_exit(); err_shrinker: + _put_huge_zero_folio(); +err_huge_zero_page: khugepaged_destroy(); err_slab: hugepage_exit_sysfs(hugepage_kobj); -- 2.47.2