From: Yu Zhao <yuzhao@google.com>
To: Andrew Morton <akpm@linux-foundation.org>
Cc: vbabka@suse.cz, alex.shi@linux.alibaba.com, guro@fb.com,
hannes@cmpxchg.org, hughd@google.com,
linux-kernel@vger.kernel.org, linux-mm@kvack.org,
mhocko@kernel.org, vdavydov.dev@gmail.com, willy@infradead.org
Subject: Re: [PATCH] mm: test page->flags directly in page_lru()
Date: Wed, 24 Feb 2021 12:57:29 -0700 [thread overview]
Message-ID: <YDavqVVajik8Z+SR@google.com> (raw)
In-Reply-To: <20210224051558.79e31c60eea2c088f4a1b300@linux-foundation.org>
On Wed, Feb 24, 2021 at 05:15:58AM -0800, Andrew Morton wrote:
> On Wed, 24 Feb 2021 01:48:07 -0700 Yu Zhao <yuzhao@google.com> wrote:
>
> > Currently page_lru() uses Page{Active,Unevictable} to determine which
> > lru list a page belongs to. Page{Active,Unevictable} contain
> > compound_head() and therefore page_lru() essentially tests
> > PG_{active,unevictable} against compound_head(page)->flags. Once an
> > lru list is determined, page->lru, rather than
> > compound_head(page)->lru, will be added to or deleted from it.
> >
> > Though not bug, having compound_head() in page_lru() increases the
> > size of vmlinux by O(KB) because page_lru() is inlined many places.
> > And removing compound_head() entirely from Page{Active,Unevictable}
> > may not be the best option (for the moment) either because there
> > may be other cases that need compound_head(). This patch makes
> > page_lru() and __clear_page_lru_flags(), which are used immediately
> > before and after operations on page->lru, test
> > PG_{active,unevictable} directly against page->flags instead.
>
> Oh geeze.
>
> > --- a/include/linux/mm_inline.h
> > +++ b/include/linux/mm_inline.h
> > @@ -46,14 +46,12 @@ static __always_inline void __clear_page_lru_flags(struct page *page)
> > {
> > VM_BUG_ON_PAGE(!PageLRU(page), page);
> >
> > - __ClearPageLRU(page);
> > -
> > /* this shouldn't happen, so leave the flags to bad_page() */
> > - if (PageActive(page) && PageUnevictable(page))
> > + if ((page->flags & (BIT(PG_active) | BIT(PG_unevictable))) ==
> > + (BIT(PG_active) | BIT(PG_unevictable)))
> > return;
>
> This isn't very nice. At the very least we should have (documented!)
> helper functions for this:
You are right. Now when I look at this, I s/dislike/hate/ it.
> /* comment goes here */
> static inline bool RawPageActive(struct page *page)
> {
> ...
> }
>
>
>
> However.
>
> Here's what the preprocessor produces for an allmodconfig version of
> PageActive():
>
> static inline __attribute__((__gnu_inline__)) __attribute__((__unused__)) __attribute__((no_instrument_function)) __attribute__((__always_inline__)) int PageActive(struct page *page)
> {
> return test_bit(PG_active, &({ do { if (__builtin_expect(!!(PagePoisoned(compound_head(page))), 0)) { dump_page(compound_head(page), "VM_BUG_ON_PAGE(" "PagePoisoned(compound_head(page))"")"); do { ({ asm volatile("%c0: nop\n\t" ".pushsection .discard.instr_begin\n\t" ".long %c0b - .\n\t" ".popsection\n\t" : : "i" (373)); }); do { asm volatile("1:\t" ".byte 0x0f, 0x0b" "\n" ".pushsection __bug_table,\"aw\"\n" "2:\t" ".long " "1b" " - 2b" "\t# bug_entry::bug_addr\n" "\t" ".long " "%c0" " - 2b" "\t# bug_entry::file\n" "\t.word %c1" "\t# bug_entry::line\n" "\t.word %c2" "\t# bug_entry::flags\n" "\t.org 2b+%c3\n" ".popsection" : : "i" ("./include/linux/page-flags.h"), "i" (338), "i" (0), "i" (sizeof(struct bug_entry))); } while (0); do { ({ asm volatile("%c0:\n\t" ".pushsection .discard.unreachable\n\t" ".long %c0b - .\n\t" ".popsection\n\t" : : "i" (374)); }); asm volatile(""); __builtin_unreachable(); } while (0); } while (0); } } while (0); compound_head(page); })->flags);
>
> }
>
> That's all to test a single bit!
I hear you. Let me spend a couple of days and focus on PG_{lru,active,
unevictable,swapbacked} first. They are mostly used with lru-related
operations and therefore can be switched to a compound_head()-free
policy easily. My estimate is we could save ~8KB by doing so :)
Weaning off compound_head() completely is a larger commitment neither
I or Alex are willing to make at the moment -- I did suggest this to
him last night when I asked him to help test build with GCC, which is
their default compiler (we've switched to Clang).
Another good point he has raised is they did see a slowdown on ARM64
after compound_head() was first introduced. My point is there may be
measurable performance benefit too if we could get rid of those
excessive calls to compound_head(). And I'd be happy to work with
somebody if they are interested in doing this.
Fair?
> Four calls to compound_head().
>
> Compiling this:
>
> int wibble(struct page *page)
> {
> return PageActive(page);
> }
>
>
> to the below assembly output (allmodconfig build) and it appears that
> the compiler did not CSE these calls. Perhaps it would be beneficial
> to give it a bit of help.
Another interesting thing I've noticed is the following change from
patch 10 also makes vmlinux a couple of hundreds bytes larger with
my GCC 4.9.x.
-unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru, int zone_idx)
+static unsigned long lruvec_lru_size(struct lruvec *lruvec, enum lru_list lru,
+ int zone_idx)
> This is all nuts. How much of this inlining is really justifiable? Do
> we know we wouldn't get a better kernel if we did
>
> mv mm-inline.h mm-not-inline-any-more.c
>
> ?
>
> Methinks that mm-inline.c needs some serious work...
Agreed. I'll send another series of patches on top of the lru cleanup
series this week.
> .type wibble, @function
> wibble:
> 1: call __fentry__
> .section __mcount_loc, "a",@progbits
> .quad 1b
> .previous
> pushq %r12 #
> pushq %rbp #
> pushq %rbx #
> # mm/swap.c:1156: {
> movq %rdi, %rbx # page, page
> movq %rbx, %rbp # page, _14
> # ./include/linux/page-flags.h:184: unsigned long head = READ_ONCE(page->compound_head);
> call __sanitizer_cov_trace_pc #
> movq 8(%rbx), %r12 # page_2(D)->D.14210.D.14188.compound_head, _8
> # ./include/linux/page-flags.h:186: if (unlikely(head & 1))
> testb $1, %r12b #, _8
> je .L2945 #,
> # ./include/linux/page-flags.h:187: return (struct page *) (head - 1);
> call __sanitizer_cov_trace_pc #
> leaq -1(%r12), %rbp #, _14
> jmp .L2945 #
> .L2945:
> call __sanitizer_cov_trace_pc #
> # ./include/linux/page-flags.h:338: PAGEFLAG(Active, active, PF_HEAD) __CLEARPAGEFLAG(Active, active, PF_HEAD)
> cmpq $-1, 0(%rbp) #, MEM[(long unsigned int *)_15]
> jne .L2946 #,
> # ./include/linux/page-flags.h:184: unsigned long head = READ_ONCE(page->compound_head);
> call __sanitizer_cov_trace_pc #
> movq 8(%rbx), %rbp #, _16
> # ./include/linux/page-flags.h:186: if (unlikely(head & 1))
> testb $1, %bpl #, _16
> je .L2947 #,
> # ./include/linux/page-flags.h:187: return (struct page *) (head - 1);
> leaq -1(%rbp), %rbx #, page
> call __sanitizer_cov_trace_pc #
> .L2947:
> # ./include/linux/page-flags.h:338: PAGEFLAG(Active, active, PF_HEAD) __CLEARPAGEFLAG(Active, active, PF_HEAD)
> call __sanitizer_cov_trace_pc #
> movq $.LC20, %rsi #,
> movq %rbx, %rdi # page,
> call dump_page #
> #APP
> # 338 "./include/linux/page-flags.h" 1
> 373: nop #
> .pushsection .discard.instr_begin
> .long 373b - . #
> .popsection
>
> # 0 "" 2
> # 338 "./include/linux/page-flags.h" 1
> 1: .byte 0x0f, 0x0b
> .pushsection __bug_table,"aw"
> 2: .long 1b - 2b # bug_entry::bug_addr
> .long .LC3 - 2b # bug_entry::file #
> .word 338 # bug_entry::line #
> .word 0 # bug_entry::flags #
> .org 2b+12 #
> .popsection
> # 0 "" 2
> # 338 "./include/linux/page-flags.h" 1
> 374: #
> .pushsection .discard.unreachable
> .long 374b - . #
> .popsection
>
> # 0 "" 2
> #NO_APP
> .L2946:
> # ./include/linux/page-flags.h:184: unsigned long head = READ_ONCE(page->compound_head);
> call __sanitizer_cov_trace_pc #
> movq 8(%rbx), %rbp #, _28
> # ./include/linux/page-flags.h:186: if (unlikely(head & 1))
> testb $1, %bpl #, _28
> je .L2948 #,
> # ./include/linux/page-flags.h:187: return (struct page *) (head - 1);
> leaq -1(%rbp), %rbx #, page
> call __sanitizer_cov_trace_pc #
> .L2948:
> # ./arch/x86/include/asm/bitops.h:207: (addr[nr >> _BITOPS_LONG_SHIFT])) != 0;
> call __sanitizer_cov_trace_pc #
> movq (%rbx), %rax # MEM[(const long unsigned int *)_35], _24
> # mm/swap.c:1158: }
> popq %rbx #
> popq %rbp #
> # ./arch/x86/include/asm/bitops.h:207: (addr[nr >> _BITOPS_LONG_SHIFT])) != 0;
> shrq $5, %rax #, tmp107
> # ./include/linux/page-flags.h:338: PAGEFLAG(Active, active, PF_HEAD) __CLEARPAGEFLAG(Active, active, PF_HEAD)
> andl $1, %eax #, tmp106
> # mm/swap.c:1158: }
> popq %r12 #
> ret
>
next prev parent reply other threads:[~2021-02-24 19:57 UTC|newest]
Thread overview: 49+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-22 22:05 [PATCH v2 00/10] mm: lru related cleanups Yu Zhao
2021-01-22 22:05 ` [PATCH v2 01/10] mm: use add_page_to_lru_list() Yu Zhao
2021-01-26 18:57 ` Vlastimil Babka
2021-01-27 2:12 ` Miaohe Lin
2021-01-22 22:05 ` [PATCH v2 02/10] mm: shuffle lru list addition and deletion functions Yu Zhao
2021-01-26 18:58 ` Vlastimil Babka
2021-01-27 2:14 ` Miaohe Lin
2021-01-22 22:05 ` [PATCH v2 03/10] mm: don't pass "enum lru_list" to lru list addition functions Yu Zhao
2021-01-26 19:13 ` Vlastimil Babka
2021-01-26 21:34 ` Yu Zhao
2021-01-27 10:51 ` Vlastimil Babka
2021-01-26 22:01 ` Matthew Wilcox
2021-01-26 22:14 ` Yu Zhao
2021-02-23 22:50 ` Andrew Morton
2021-02-24 5:29 ` Yu Zhao
2021-02-24 8:06 ` Alex Shi
2021-02-24 8:37 ` Yu Zhao
2021-02-24 9:01 ` Alex Shi
2021-01-22 22:05 ` [PATCH v2 04/10] mm: don't pass "enum lru_list" to trace_mm_lru_insertion() Yu Zhao
2021-01-22 22:05 ` [PATCH v2 05/10] mm: don't pass "enum lru_list" to del_page_from_lru_list() Yu Zhao
2021-01-22 22:05 ` [PATCH v2 06/10] mm: add __clear_page_lru_flags() to replace page_off_lru() Yu Zhao
2021-01-22 22:05 ` [PATCH v2 07/10] mm: VM_BUG_ON lru page flags Yu Zhao
2021-01-22 22:05 ` [PATCH v2 08/10] mm: fold page_lru_base_type() into its sole caller Yu Zhao
2021-01-22 22:05 ` [PATCH v2 09/10] mm: fold __update_lru_size() " Yu Zhao
2021-01-22 22:06 ` [PATCH v2 10/10] mm: make lruvec_lru_size() static Yu Zhao
2021-02-24 8:48 ` [PATCH] mm: test page->flags directly in page_lru() Yu Zhao
2021-02-24 13:15 ` Andrew Morton
2021-02-24 19:57 ` Yu Zhao [this message]
2021-02-24 21:56 ` Matthew Wilcox
2021-02-24 22:34 ` Yu Zhao
2021-02-24 22:48 ` Matthew Wilcox
2021-02-24 23:50 ` Yu Zhao
2021-02-25 3:55 ` Matthew Wilcox
2021-02-25 5:22 ` Yu Zhao
2021-02-25 12:12 ` Matthew Wilcox
2021-02-26 9:17 ` [PATCH v2 0/3] trim the uses of compound_head() Yu Zhao
2021-02-26 9:17 ` [PATCH v2 1/3] mm: bypass compound_head() for PF_NO_TAIL when enforce=1 Yu Zhao
2021-02-26 9:17 ` [PATCH v2 2/3] mm: use PF_NO_TAIL for PG_lru Yu Zhao
2021-02-26 20:22 ` Yu Zhao
2021-02-26 9:17 ` [PATCH v2 3/3] mm: use PF_ONLY_HEAD for PG_active and PG_unevictable Yu Zhao
2021-02-26 12:13 ` Matthew Wilcox
2021-02-26 19:49 ` Yu Zhao
2021-02-26 20:27 ` Matthew Wilcox
2021-03-01 11:50 ` Kirill A. Shutemov
2021-03-01 19:58 ` Yu Zhao
2021-03-01 20:16 ` Hugh Dickins
2021-03-01 20:26 ` Matthew Wilcox
2021-02-26 10:52 ` [PATCH v2 0/3] trim the uses of compound_head() Vlastimil Babka
2021-02-26 19:04 ` Yu Zhao
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=YDavqVVajik8Z+SR@google.com \
--to=yuzhao@google.com \
--cc=akpm@linux-foundation.org \
--cc=alex.shi@linux.alibaba.com \
--cc=guro@fb.com \
--cc=hannes@cmpxchg.org \
--cc=hughd@google.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@kernel.org \
--cc=vbabka@suse.cz \
--cc=vdavydov.dev@gmail.com \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox