From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB240C4345F for ; Fri, 3 May 2024 09:54:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BA1546B009F; Fri, 3 May 2024 05:54:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A7D836B009D; Fri, 3 May 2024 05:54:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 790766B00A3; Fri, 3 May 2024 05:54:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 52FFD6B00A0 for ; Fri, 3 May 2024 05:54:06 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 0E4FC1C1782 for ; Fri, 3 May 2024 09:54:06 +0000 (UTC) X-FDA: 82076623692.03.2501121 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf10.hostedemail.com (Postfix) with ESMTP id 2767CC0004 for ; Fri, 3 May 2024 09:54:02 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=ERDQV5qf; dmarc=fail reason="No valid SPF, DKIM not aligned (relaxed)" header.from=kernel.org (policy=none); spf=none (imf10.hostedemail.com: domain of mcgrof@infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=mcgrof@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714730043; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4I9xqhTy2lFMzABeOddmnumdvFNlTh4tfqYx9OGQOzQ=; b=0FyJZ/fV98y4hw4jfIR5rMQ+4kXz2DoQnCR3VzJzkZ4Mo827+tYHjkGqbrlKn0hmvcwSpG mguV96Zp+WpB1lZBBiRF92sw/JaoTVe6kV2UN800Eari2qSebxU8i0/clesqVUD6F9Heb6 4QWi8AMuh6fXPpxLEOgUFUfi/hAv49g= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714730043; a=rsa-sha256; cv=none; b=IqxBC70aS3Q9Jdz5tveJXvzsbk2czyWn0pFGF8756ASki5FFU0sBG7H/zZ1VeB6apCYQ84 izlZ2bB9Rytwg+csyKCdqgmP4k1Awuiugd+uG5Zvq2xj0ufkG46e+c9JrZiDB+2OU/0hKd GlPYIPSmVJ2zqdt35i0GU6Hmd0pnnEM= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=infradead.org header.s=bombadil.20210309 header.b=ERDQV5qf; dmarc=fail reason="No valid SPF, DKIM not aligned (relaxed)" header.from=kernel.org (policy=none); spf=none (imf10.hostedemail.com: domain of mcgrof@infradead.org has no SPF policy when checking 198.137.202.133) smtp.mailfrom=mcgrof@infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=4I9xqhTy2lFMzABeOddmnumdvFNlTh4tfqYx9OGQOzQ=; b=ERDQV5qfxEQHLjpvZHSwQlvFVW /w89WTHFzx2qowoqlcIaD3WLdLj2cnvL+gDbVmEifTdEIu4xIR09zhDBobRvkK1tZG2yG6Vydn1jb qpoe813VDwBJ31mJFbVwxU00mqf8jA+7S0khgN2NyNv7afb0Wx0v/UZcQMM/plI7HtLhTFFcfPlfh 0KJfgzdPiPwyxU3xS6vz+mDI3MgBX7ELgK8ghzPEwHGitM5Ke1RUSlhjkkNUObMCkZV3nCQxbYQDS MJGXiVvH4D/DAkF9rl3xIso/903CJYKq/at2hIu2eHYQZ1VU7q/gUVX0TC8IurL29fpVZH+sC2rlE MxOwOfvA==; Received: from mcgrof by bombadil.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1s2pc3-0000000Fw3i-0pnP; Fri, 03 May 2024 09:53:55 +0000 From: Luis Chamberlain To: akpm@linux-foundation.org, willy@infradead.org, djwong@kernel.org, brauner@kernel.org, david@fromorbit.com, chandan.babu@oracle.com Cc: hare@suse.de, ritesh.list@gmail.com, john.g.garry@oracle.com, ziy@nvidia.com, linux-fsdevel@vger.kernel.org, linux-xfs@vger.kernel.org, linux-mm@kvack.org, linux-block@vger.kernel.org, gost.dev@samsung.com, p.raghav@samsung.com, kernel@pankajraghav.com, mcgrof@kernel.org Subject: [PATCH v5 05/11] mm: split a folio in minimum folio order chunks Date: Fri, 3 May 2024 02:53:47 -0700 Message-ID: <20240503095353.3798063-6-mcgrof@kernel.org> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240503095353.3798063-1-mcgrof@kernel.org> References: <20240503095353.3798063-1-mcgrof@kernel.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 2767CC0004 X-Stat-Signature: wa9rf61nn87x7ecujbcxzmgii4u8wdbg X-Rspam-User: X-HE-Tag: 1714730042-95376 X-HE-Meta: U2FsdGVkX185xOPOGErXG9Gc4+HP9SYoon3f1wl4//2yfthoWYn5w9ugsPZH4/SvjV6HowPdqGRQJ7bausSOuDODpd6qSOgT1zk/TmVGOlkPqz1/DMC8NL3mVG8YYGzTZADbhPBZkoaE5FwzWVFzewStjxISUkx3BB/1Mum2JpXrfseQ7ujs3PMHC02D0vNoXgND3PKdRDw66gZQp9d97hlXJJvXABwpsuAil4XT94LGLNt8NSUMx3RukkUxTBELjY0o2WrNSK79ZL0SxeODY4jRUIst51zDAD1WGfeMxZDaR7T4rrJS0cEgR4S0PTNCON7hN1E3nSkMazQTPtf3gelxrCHUllNlEuuYKJYe+Ylg6pu2rmaSakRLzyFqyBiFjc0qwhmNTj8ABKN65ZCVtEtv4kdNIrZK8xLXzTMnj5pFGqTxxC521xvb9cvlGNpTlk1WDbLFqSThHxDuC6BzpzCDL/THzf3jQCUB5wOYYcEVh1n9sNhyaYCwxL7AwgIhqau0sXTOU4bn5d6eU7e4uEld9Ft9XM+4grHu/OtjvsXgmnJNC4btw7SEUQoZuLv1mIxBOrHUrLluTvFZkiH1GIAt0vb8qYnzsxns7fWkx1t34p/OIGTQiUccUJtGOPW0b1WaWQRtkr9SLxaMO54czThyMl18nhDJMo05vHTZRPkKnq6p09K3lgLiGvA2jCv1W3HCz86mMMw+QollggqVz3syc4CEjx/EHJoeLJmGI8dh0Ue9Jte9qeLi5KJpkUaQV8XgEKrOInmzaA0eaI+tkvuRERIcEkGljnJY9Cp3CyMyJSQlft8XlPmaF0+hsm//rIP4j6fPrIl/803U1TyLnuh0JNRx4ktr4c7/SpUVZ4Pw4/dsykt3SLK+z9yh1r2lrB0NIYdjM6eKEIyHjDk64Y4/uIdkGjL71BpxtKgwo0thvarr2FM++g43RD4buxcck2W5Jxtss0W8QP2nGr7 DVTg+Vzl ucy29FUKEgdRs6BIDbykwJH1QV55ICDRXQ5jJH11mJcaCJ/4hZ+Z4aQtJ3iNtDnEKbP8cwCiZQAyzTxxzx/PYic2LCmmjxslvU1dCdPSdqKtW7duVNAcPKQZmHIXIMn4cZ4CduumrWEUP2DB0LlwgtESrnDqHhNiDlvxe0MhIOCi0xrorxHTpyWd3NJxwUD9O8UwAhrOe2XcnqlSpEpyfoLdivc2/nVDCwQWu9UItm2cRURAtgVrVUuvqfRhmiiZTDnplxTTBcEhcZLrifTvB8WW/JmYUH29Js6pO5jvKSpl31kVVk6PBS/tvvUfP06zBTOiMkajtYhXLTE4k0QjajHTFg/EPh5k4YHKJ8MvIk9e+iyGI5YYsFVLBYowtFW6gc/3UY2UBk3drbbNX+wr0X2Xeu5ZOLCDrC2qzimtg5BoUaHGB39i0D3noqL8sfbHEVW2l8/4PuTOHOpZrjmsAwtmXxHCrjFFPJJyMwZCGYCuemGo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: split_folio() and split_folio_to_list() assume order 0, to support minorder we must expand these to check the folio mapping order and use that. Set new_order to be at least minimum folio order if it is set in split_huge_page_to_list() so that we can maintain minimum folio order requirement in the page cache. Update the debugfs write files used for testing to ensure the order is respected as well. We simply enforce the min order when a file mapping is used. Signed-off-by: Pankaj Raghav Signed-off-by: Luis Chamberlain --- include/linux/huge_mm.h | 12 ++++++---- mm/huge_memory.c | 50 ++++++++++++++++++++++++++++++++++++++--- 2 files changed, 55 insertions(+), 7 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index de0c89105076..06748a8fa43b 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -87,6 +87,8 @@ extern struct kobj_attribute shmem_enabled_attr; #define thp_vma_allowable_order(vma, vm_flags, smaps, in_pf, enforce_sysfs, order) \ (!!thp_vma_allowable_orders(vma, vm_flags, smaps, in_pf, enforce_sysfs, BIT(order))) +#define split_folio(f) split_folio_to_list(f, NULL) + #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define HPAGE_PMD_SHIFT PMD_SHIFT #define HPAGE_PMD_SIZE ((1UL) << HPAGE_PMD_SHIFT) @@ -267,9 +269,10 @@ void folio_prep_large_rmappable(struct folio *folio); bool can_split_folio(struct folio *folio, int *pextra_pins); int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, unsigned int new_order); +int split_folio_to_list(struct folio *folio, struct list_head *list); static inline int split_huge_page(struct page *page) { - return split_huge_page_to_list_to_order(page, NULL, 0); + return split_folio(page_folio(page)); } void deferred_split_folio(struct folio *folio); @@ -432,6 +435,10 @@ static inline int split_huge_page(struct page *page) { return 0; } +static inline int split_folio_to_list(struct page *page, struct list_head *list) +{ + return 0; +} static inline void deferred_split_folio(struct folio *folio) {} #define split_huge_pmd(__vma, __pmd, __address) \ do { } while (0) @@ -532,9 +539,6 @@ static inline int split_folio_to_order(struct folio *folio, int new_order) return split_folio_to_list_to_order(folio, NULL, new_order); } -#define split_folio_to_list(f, l) split_folio_to_list_to_order(f, l, 0) -#define split_folio(f) split_folio_to_order(f, 0) - /* * archs that select ARCH_WANTS_THP_SWAP but don't support THP_SWP due to * limitations in the implementation like arm64 MTE can override this to diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 89f58c7603b2..c0cc8f32fe42 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -3035,6 +3035,9 @@ bool can_split_folio(struct folio *folio, int *pextra_pins) * Returns 0 if the hugepage is split successfully. * Returns -EBUSY if the page is pinned or if anon_vma disappeared from under * us. + * + * Callers should ensure that the order respects the address space mapping + * min-order if one is set. */ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, unsigned int new_order) @@ -3107,6 +3110,7 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, mapping = NULL; anon_vma_lock_write(anon_vma); } else { + unsigned int min_order; gfp_t gfp; mapping = folio->mapping; @@ -3117,6 +3121,14 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, goto out; } + min_order = mapping_min_folio_order(folio->mapping); + if (new_order < min_order) { + VM_WARN_ONCE(1, "Cannot split mapped folio below min-order: %u", + min_order); + ret = -EINVAL; + goto out; + } + gfp = current_gfp_context(mapping_gfp_mask(mapping) & GFP_RECLAIM_MASK); @@ -3227,6 +3239,21 @@ int split_huge_page_to_list_to_order(struct page *page, struct list_head *list, return ret; } +int split_folio_to_list(struct folio *folio, struct list_head *list) +{ + unsigned int min_order = 0; + + if (!folio_test_anon(folio)) { + if (!folio->mapping) { + count_vm_event(THP_SPLIT_PAGE_FAILED); + return -EBUSY; + } + min_order = mapping_min_folio_order(folio->mapping); + } + + return split_huge_page_to_list_to_order(&folio->page, list, min_order); +} + void folio_undo_large_rmappable(struct folio *folio) { struct deferred_split *ds_queue; @@ -3466,6 +3493,7 @@ static int split_huge_pages_pid(int pid, unsigned long vaddr_start, struct vm_area_struct *vma = vma_lookup(mm, addr); struct page *page; struct folio *folio; + unsigned int target_order = new_order; if (!vma) break; @@ -3502,7 +3530,18 @@ static int split_huge_pages_pid(int pid, unsigned long vaddr_start, if (!folio_trylock(folio)) goto next; - if (!split_folio_to_order(folio, new_order)) + if (!folio_test_anon(folio)) { + unsigned int min_order; + + if (!folio->mapping) + goto next; + + min_order = mapping_min_folio_order(folio->mapping); + if (new_order < target_order) + target_order = min_order; + } + + if (!split_folio_to_order(folio, target_order)) split++; folio_unlock(folio); @@ -3545,14 +3584,19 @@ static int split_huge_pages_in_file(const char *file_path, pgoff_t off_start, for (index = off_start; index < off_end; index += nr_pages) { struct folio *folio = filemap_get_folio(mapping, index); + unsigned int min_order, target_order = new_order; nr_pages = 1; if (IS_ERR(folio)) continue; - if (!folio_test_large(folio)) + if (!folio->mapping || !folio_test_large(folio)) goto next; + min_order = mapping_min_folio_order(mapping); + if (new_order < min_order) + target_order = min_order; + total++; nr_pages = folio_nr_pages(folio); @@ -3562,7 +3606,7 @@ static int split_huge_pages_in_file(const char *file_path, pgoff_t off_start, if (!folio_trylock(folio)) goto next; - if (!split_folio_to_order(folio, new_order)) + if (!split_folio_to_order(folio, target_order)) split++; folio_unlock(folio); -- 2.43.0