From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.8 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,UNWANTED_LANGUAGE_BODY, URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DB04C433E0 for ; Fri, 5 Mar 2021 04:20:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3F99264FE4 for ; Fri, 5 Mar 2021 04:20:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3F99264FE4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CF0976B0010; Thu, 4 Mar 2021 23:20:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CA03E6B0012; Thu, 4 Mar 2021 23:20:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B69116B0022; Thu, 4 Mar 2021 23:20:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0171.hostedemail.com [216.40.44.171]) by kanga.kvack.org (Postfix) with ESMTP id 98F986B0010 for ; Thu, 4 Mar 2021 23:20:42 -0500 (EST) Received: from smtpin14.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 6233C364F for ; Fri, 5 Mar 2021 04:20:42 +0000 (UTC) X-FDA: 77884519524.14.B5FCA09 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf14.hostedemail.com (Postfix) with ESMTP id 84797C0001FA for ; Fri, 5 Mar 2021 04:20:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=dP5UMVOfza3vmGEPtSLp/DyWFDFLrJcWbOeP/yqlT/g=; b=jZ+XY+kY4DkzL8xI9W+ptjKuDC BE6f7v9Dile/BYECrEL77EDUjJToAGJcpex9iNwpqZdJTbUZKT9s16e/o6dfzwRq75RLKn373Yb5P OQUvCoW6CEstjPO9EBcrTtJwgidoihgFdviLyWlBXRtnBuHZswteNy69ffc6QPnrrA8qbUAbUcf3+ rgjY56IS5mg6BSHFA8YENtL69WVzAAtUw1e+bMM3Z8fKph7ftqW20HVWs/K+QhfSPWkmPIo66NRgo LE4DsJ8wSRZEdBpEQ8XKdRYGVEoXmuruQJtrFIFWWh+m1ynnzsYRV/0XUI2PBRUxl46/6WCkPFdj2 xLyQWlBw==; Received: from willy by casper.infradead.org with local (Exim 4.94 #2 (Red Hat Linux)) id 1lI1w5-00A3To-HA; Fri, 05 Mar 2021 04:19:42 +0000 From: "Matthew Wilcox (Oracle)" To: linux-mm@kvack.org Cc: "Matthew Wilcox (Oracle)" , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: [PATCH v4 07/25] mm: Create FolioFlags Date: Fri, 5 Mar 2021 04:18:43 +0000 Message-Id: <20210305041901.2396498-8-willy@infradead.org> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210305041901.2396498-1-willy@infradead.org> References: <20210305041901.2396498-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 84797C0001FA X-Stat-Signature: pqfqosa6ebxqouxurwbg9xq3x74kkd4h Received-SPF: none (infradead.org>: No applicable sender policy available) receiver=imf14; identity=mailfrom; envelope-from=""; helo=casper.infradead.org; client-ip=90.155.50.34 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614918039-85097 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: These new functions are the folio analogues of the PageFlags functions. If CONFIG_DEBUG_VM_PGFLAGS is enabled, we check the folio is not a tail page at every invocation. Note that this will also catch the PagePoisone= d case as a poisoned page has every bit set, which would include PageTail. This saves 1740 bytes of text with the distro-derived config that I'm testing due to removing a double call to compound_head() in PageSwapCache(). Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/fscache.h | 5 ++ include/linux/page-flags.h | 104 ++++++++++++++++++++++++++++++------- 2 files changed, 89 insertions(+), 20 deletions(-) diff --git a/include/linux/fscache.h b/include/linux/fscache.h index a1c928fe98e7..c274006f4037 100644 --- a/include/linux/fscache.h +++ b/include/linux/fscache.h @@ -38,6 +38,11 @@ #define ClearPageFsCache(page) ClearPagePrivate2((page)) #define TestSetPageFsCache(page) TestSetPagePrivate2((page)) #define TestClearPageFsCache(page) TestClearPagePrivate2((page)) +#define FolioFsCache(folio) FolioPrivate2((folio)) +#define SetFolioFsCache(folio) SetFolioPrivate2((folio)) +#define ClearFolioFsCache(folio) ClearFolioPrivate2((folio)) +#define TestSetFolioFsCache(folio) TestSetFolioPrivate2((folio)) +#define TestClearFolioFsCache(folio) TestClearFolioPrivate2((folio)) =20 /* pattern used to fill dead space in an index entry */ #define FSCACHE_INDEX_DEADFILL_PATTERN 0x79 diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 04a34c08e0a6..90381858d901 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -212,6 +212,12 @@ static inline void page_init_poison(struct page *pag= e, size_t size) } #endif =20 +static unsigned long *folio_flags(struct folio *folio) +{ + VM_BUG_ON_PGFLAGS(PageTail(&folio->page), &folio->page); + return &folio->page.flags; +} + /* * Page flags policies wrt compound pages * @@ -260,30 +266,44 @@ static inline void page_init_poison(struct page *pa= ge, size_t size) * Macros to create function definitions for page flags */ #define TESTPAGEFLAG(uname, lname, policy) \ +static __always_inline int Folio##uname(struct folio *folio) \ + { return test_bit(PG_##lname, folio_flags(folio)); } \ static __always_inline int Page##uname(struct page *page) \ { return test_bit(PG_##lname, &policy(page, 0)->flags); } =20 #define SETPAGEFLAG(uname, lname, policy) \ +static __always_inline void SetFolio##uname(struct folio *folio) \ + { set_bit(PG_##lname, folio_flags(folio)); } \ static __always_inline void SetPage##uname(struct page *page) \ { set_bit(PG_##lname, &policy(page, 1)->flags); } =20 #define CLEARPAGEFLAG(uname, lname, policy) \ +static __always_inline void ClearFolio##uname(struct folio *folio) \ + { clear_bit(PG_##lname, folio_flags(folio)); } \ static __always_inline void ClearPage##uname(struct page *page) \ { clear_bit(PG_##lname, &policy(page, 1)->flags); } =20 #define __SETPAGEFLAG(uname, lname, policy) \ +static __always_inline void __SetFolio##uname(struct folio *folio) \ + { __set_bit(PG_##lname, folio_flags(folio)); } \ static __always_inline void __SetPage##uname(struct page *page) \ { __set_bit(PG_##lname, &policy(page, 1)->flags); } =20 #define __CLEARPAGEFLAG(uname, lname, policy) \ +static __always_inline void __ClearFolio##uname(struct folio *folio) \ + { __clear_bit(PG_##lname, folio_flags(folio)); } \ static __always_inline void __ClearPage##uname(struct page *page) \ { __clear_bit(PG_##lname, &policy(page, 1)->flags); } =20 #define TESTSETFLAG(uname, lname, policy) \ +static __always_inline int TestSetFolio##uname(struct folio *folio) \ + { return test_and_set_bit(PG_##lname, folio_flags(folio)); } \ static __always_inline int TestSetPage##uname(struct page *page) \ { return test_and_set_bit(PG_##lname, &policy(page, 1)->flags); } =20 #define TESTCLEARFLAG(uname, lname, policy) \ +static __always_inline int TestClearFolio##uname(struct folio *folio) \ + { return test_and_clear_bit(PG_##lname, folio_flags(folio)); } \ static __always_inline int TestClearPage##uname(struct page *page) \ { return test_and_clear_bit(PG_##lname, &policy(page, 1)->flags); } =20 @@ -302,21 +322,27 @@ static __always_inline int TestClearPage##uname(str= uct page *page) \ TESTCLEARFLAG(uname, lname, policy) =20 #define TESTPAGEFLAG_FALSE(uname) \ +static inline int Folio##uname(const struct folio *folio) { return 0; } = \ static inline int Page##uname(const struct page *page) { return 0; } =20 #define SETPAGEFLAG_NOOP(uname) \ +static inline void SetFolio##uname(struct folio *folio) { } \ static inline void SetPage##uname(struct page *page) { } =20 #define CLEARPAGEFLAG_NOOP(uname) \ +static inline void ClearFolio##uname(struct folio *folio) { } \ static inline void ClearPage##uname(struct page *page) { } =20 #define __CLEARPAGEFLAG_NOOP(uname) \ +static inline void __ClearFolio##uname(struct folio *folio) { } \ static inline void __ClearPage##uname(struct page *page) { } =20 #define TESTSETFLAG_FALSE(uname) \ +static inline int TestSetFolio##uname(struct folio *folio) { return 0; }= \ static inline int TestSetPage##uname(struct page *page) { return 0; } =20 #define TESTCLEARFLAG_FALSE(uname) \ +static inline int TestClearFolio##uname(struct folio *folio) { return 0;= } \ static inline int TestClearPage##uname(struct page *page) { return 0; } =20 #define PAGEFLAG_FALSE(uname) TESTPAGEFLAG_FALSE(uname) \ @@ -393,14 +419,18 @@ PAGEFLAG_FALSE(HighMem) #endif =20 #ifdef CONFIG_SWAP -static __always_inline int PageSwapCache(struct page *page) +static __always_inline bool FolioSwapCache(struct folio *folio) { -#ifdef CONFIG_THP_SWAP - page =3D compound_head(page); -#endif - return PageSwapBacked(page) && test_bit(PG_swapcache, &page->flags); + return FolioSwapBacked(folio) && + test_bit(PG_swapcache, folio_flags(folio)); =20 } + +static __always_inline bool PageSwapCache(struct page *page) +{ + return FolioSwapCache(page_folio(page)); +} + SETPAGEFLAG(SwapCache, swapcache, PF_NO_TAIL) CLEARPAGEFLAG(SwapCache, swapcache, PF_NO_TAIL) #else @@ -478,10 +508,14 @@ static __always_inline int PageMappingFlags(struct = page *page) return ((unsigned long)page->mapping & PAGE_MAPPING_FLAGS) !=3D 0; } =20 -static __always_inline int PageAnon(struct page *page) +static __always_inline bool FolioAnon(struct folio *folio) { - page =3D compound_head(page); - return ((unsigned long)page->mapping & PAGE_MAPPING_ANON) !=3D 0; + return ((unsigned long)folio->page.mapping & PAGE_MAPPING_ANON) !=3D 0; +} + +static __always_inline bool PageAnon(struct page *page) +{ + return FolioAnon(page_folio(page)); } =20 static __always_inline int __PageMovable(struct page *page) @@ -509,18 +543,16 @@ TESTPAGEFLAG_FALSE(Ksm) =20 u64 stable_page_flags(struct page *page); =20 -static inline int PageUptodate(struct page *page) +static inline int FolioUptodate(struct folio *folio) { - int ret; - page =3D compound_head(page); - ret =3D test_bit(PG_uptodate, &(page)->flags); + int ret =3D test_bit(PG_uptodate, folio_flags(folio)); /* * Must ensure that the data we read out of the page is loaded * _after_ we've loaded page->flags to check for PageUptodate. * We can skip the barrier if the page is not uptodate, because * we wouldn't be reading anything from it. * - * See SetPageUptodate() for the other side of the story. + * See SetFolioUptodate() for the other side of the story. */ if (ret) smp_rmb(); @@ -528,23 +560,36 @@ static inline int PageUptodate(struct page *page) return ret; } =20 -static __always_inline void __SetPageUptodate(struct page *page) +static inline int PageUptodate(struct page *page) +{ + return FolioUptodate(page_folio(page)); +} + +static __always_inline void __SetFolioUptodate(struct folio *folio) { - VM_BUG_ON_PAGE(PageTail(page), page); smp_wmb(); - __set_bit(PG_uptodate, &page->flags); + __set_bit(PG_uptodate, folio_flags(folio)); } =20 -static __always_inline void SetPageUptodate(struct page *page) +static __always_inline void SetFolioUptodate(struct folio *folio) { - VM_BUG_ON_PAGE(PageTail(page), page); /* * Memory barrier must be issued before setting the PG_uptodate bit, * so that all previous stores issued in order to bring the page * uptodate are actually visible before PageUptodate becomes true. */ smp_wmb(); - set_bit(PG_uptodate, &page->flags); + set_bit(PG_uptodate, folio_flags(folio)); +} + +static __always_inline void __SetPageUptodate(struct page *page) +{ + __SetFolioUptodate((struct folio *)page); +} + +static __always_inline void SetPageUptodate(struct page *page) +{ + SetFolioUptodate((struct folio *)page); } =20 CLEARPAGEFLAG(Uptodate, uptodate, PF_NO_TAIL) @@ -569,6 +614,17 @@ static inline void set_page_writeback_keepwrite(stru= ct page *page) =20 __PAGEFLAG(Head, head, PF_ANY) CLEARPAGEFLAG(Head, head, PF_ANY) =20 +/* Whether there are one or multiple pages in a folio */ +static inline bool FolioSingle(struct folio *folio) +{ + return !FolioHead(folio); +} + +static inline bool FolioMulti(struct folio *folio) +{ + return FolioHead(folio); +} + static __always_inline void set_compound_head(struct page *page, struct = page *head) { WRITE_ONCE(page->compound_head, (unsigned long)head + 1); @@ -592,12 +648,15 @@ static inline void ClearPageCompound(struct page *p= age) #ifdef CONFIG_HUGETLB_PAGE int PageHuge(struct page *page); int PageHeadHuge(struct page *page); +static inline bool FolioHuge(struct folio *folio) +{ + return PageHeadHuge(&folio->page); +} #else TESTPAGEFLAG_FALSE(Huge) TESTPAGEFLAG_FALSE(HeadHuge) #endif =20 - #ifdef CONFIG_TRANSPARENT_HUGEPAGE /* * PageHuge() only returns true for hugetlbfs pages, but not for @@ -844,6 +903,11 @@ static inline int page_has_private(struct page *page= ) return !!(page->flags & PAGE_FLAGS_PRIVATE); } =20 +static inline bool folio_has_private(struct folio *folio) +{ + return page_has_private(&folio->page); +} + #undef PF_ANY #undef PF_HEAD #undef PF_ONLY_HEAD --=20 2.30.0