From: Hugh Dickins <hughd@google.com>
To: Andrew Morton <akpm@linux-foundation.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>,
Johannes Weiner <hannes@cmpxcg.org>,
"Kirill A. Shutemov" <kirill@shutemov.name>,
Matthew Wilcox <willy@infradead.org>,
David Hildenbrand <david@redhat.com>,
Vlastimil Babka <vbabka@suse.cz>, Peter Xu <peterx@redhat.com>,
Yang Shi <shy828301@gmail.com>,
John Hubbard <jhubbard@nvidia.com>,
Mike Kravetz <mike.kravetz@oracle.com>,
Sidhartha Kumar <sidhartha.kumar@oracle.com>,
Muchun Song <songmuchun@bytedance.com>,
Miaohe Lin <linmiaohe@huawei.com>,
Naoya Horiguchi <naoya.horiguchi@linux.dev>,
Mina Almasry <almasrymina@google.com>,
James Houghton <jthoughton@google.com>,
Zach O'Keefe <zokeefe@google.com>,
linux-kernel@vger.kernel.org, linux-mm@kvack.org
Subject: [PATCH 4/3] mm,thp,rmap: handle the normal !PageCompound case first
Date: Wed, 9 Nov 2022 18:18:49 -0800 (PST) [thread overview]
Message-ID: <fca2f694-2098-b0ef-d4e-f1d8b94d318c@google.com> (raw)
In-Reply-To: <5f52de70-975-e94f-f141-543765736181@google.com>
Commit ("mm,thp,rmap: lock_compound_mapcounts() on THP mapcounts")
propagated the "if (compound) {lock} else if (PageCompound) {lock} else
{atomic}" pattern throughout; but Linus hated the way that gives primacy
to the uncommon case: switch to "if (!PageCompound) {atomic} else if
(compound) {lock} else {lock}" throughout. Linus has a bigger idea
for how to improve it all, but here just make that rearrangement.
Signed-off-by: Hugh Dickins <hughd@google.com>
---
mm/rmap.c | 54 +++++++++++++++++++++++++++---------------------------
1 file changed, 27 insertions(+), 27 deletions(-)
diff --git a/mm/rmap.c b/mm/rmap.c
index 512e53cae2ca..4833d28c5e1a 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -1311,7 +1311,11 @@ void page_add_anon_rmap(struct page *page,
else
VM_BUG_ON_PAGE(!PageLocked(page), page);
- if (compound && PageTransHuge(page)) {
+ if (likely(!PageCompound(page))) {
+ first = atomic_inc_and_test(&page->_mapcount);
+ nr = first;
+
+ } else if (compound && PageTransHuge(page)) {
lock_compound_mapcounts(page, &mapcounts);
first = !mapcounts.compound_mapcount;
mapcounts.compound_mapcount++;
@@ -1321,8 +1325,7 @@ void page_add_anon_rmap(struct page *page,
nr = nr_subpages_unmapped(page, nr_pmdmapped);
}
unlock_compound_mapcounts(page, &mapcounts);
-
- } else if (PageCompound(page)) {
+ } else {
struct page *head = compound_head(page);
lock_compound_mapcounts(head, &mapcounts);
@@ -1330,10 +1333,6 @@ void page_add_anon_rmap(struct page *page,
first = subpage_mapcount_inc(page);
nr = first && !mapcounts.compound_mapcount;
unlock_compound_mapcounts(head, &mapcounts);
-
- } else {
- first = atomic_inc_and_test(&page->_mapcount);
- nr = first;
}
VM_BUG_ON_PAGE(!first && (flags & RMAP_EXCLUSIVE), page);
@@ -1373,20 +1372,23 @@ void page_add_anon_rmap(struct page *page,
void page_add_new_anon_rmap(struct page *page,
struct vm_area_struct *vma, unsigned long address)
{
- const bool compound = PageCompound(page);
- int nr = compound ? thp_nr_pages(page) : 1;
+ int nr;
VM_BUG_ON_VMA(address < vma->vm_start || address >= vma->vm_end, vma);
__SetPageSwapBacked(page);
- if (compound) {
+
+ if (likely(!PageCompound(page))) {
+ /* increment count (starts at -1) */
+ atomic_set(&page->_mapcount, 0);
+ nr = 1;
+ } else {
VM_BUG_ON_PAGE(!PageTransHuge(page), page);
/* increment count (starts at -1) */
atomic_set(compound_mapcount_ptr(page), 0);
+ nr = thp_nr_pages(page);
__mod_lruvec_page_state(page, NR_ANON_THPS, nr);
- } else {
- /* increment count (starts at -1) */
- atomic_set(&page->_mapcount, 0);
}
+
__mod_lruvec_page_state(page, NR_ANON_MAPPED, nr);
__page_set_anon_rmap(page, vma, address, 1);
}
@@ -1409,7 +1411,11 @@ void page_add_file_rmap(struct page *page,
VM_BUG_ON_PAGE(compound && !PageTransHuge(page), page);
lock_page_memcg(page);
- if (compound && PageTransHuge(page)) {
+ if (likely(!PageCompound(page))) {
+ first = atomic_inc_and_test(&page->_mapcount);
+ nr = first;
+
+ } else if (compound && PageTransHuge(page)) {
lock_compound_mapcounts(page, &mapcounts);
first = !mapcounts.compound_mapcount;
mapcounts.compound_mapcount++;
@@ -1419,8 +1425,7 @@ void page_add_file_rmap(struct page *page,
nr = nr_subpages_unmapped(page, nr_pmdmapped);
}
unlock_compound_mapcounts(page, &mapcounts);
-
- } else if (PageCompound(page)) {
+ } else {
struct page *head = compound_head(page);
lock_compound_mapcounts(head, &mapcounts);
@@ -1428,10 +1433,6 @@ void page_add_file_rmap(struct page *page,
first = subpage_mapcount_inc(page);
nr = first && !mapcounts.compound_mapcount;
unlock_compound_mapcounts(head, &mapcounts);
-
- } else {
- first = atomic_inc_and_test(&page->_mapcount);
- nr = first;
}
if (nr_pmdmapped)
@@ -1471,7 +1472,11 @@ void page_remove_rmap(struct page *page,
lock_page_memcg(page);
/* page still mapped by someone else? */
- if (compound && PageTransHuge(page)) {
+ if (likely(!PageCompound(page))) {
+ last = atomic_add_negative(-1, &page->_mapcount);
+ nr = last;
+
+ } else if (compound && PageTransHuge(page)) {
lock_compound_mapcounts(page, &mapcounts);
mapcounts.compound_mapcount--;
last = !mapcounts.compound_mapcount;
@@ -1481,8 +1486,7 @@ void page_remove_rmap(struct page *page,
nr = nr_subpages_unmapped(page, nr_pmdmapped);
}
unlock_compound_mapcounts(page, &mapcounts);
-
- } else if (PageCompound(page)) {
+ } else {
struct page *head = compound_head(page);
lock_compound_mapcounts(head, &mapcounts);
@@ -1490,10 +1494,6 @@ void page_remove_rmap(struct page *page,
last = subpage_mapcount_dec(page);
nr = last && !mapcounts.compound_mapcount;
unlock_compound_mapcounts(head, &mapcounts);
-
- } else {
- last = atomic_add_negative(-1, &page->_mapcount);
- nr = last;
}
if (nr_pmdmapped) {
--
2.35.3
next prev parent reply other threads:[~2022-11-10 2:18 UTC|newest]
Thread overview: 54+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-11-03 1:44 [PATCH 0/3] mm,huge,rmap: unify and speed up compound mapcounts Hugh Dickins
2022-11-03 1:48 ` [PATCH 1/3] mm,hugetlb: use folio fields in second tail page Hugh Dickins
2022-11-03 21:18 ` Sidhartha Kumar
2022-11-04 4:29 ` Hugh Dickins
2022-11-10 0:11 ` Sidhartha Kumar
2022-11-10 2:10 ` Hugh Dickins
2022-11-10 2:13 ` [PATCH 1/3 fix] mm,hugetlb: use folio fields in second tail page: fix Hugh Dickins
2022-11-05 19:13 ` [PATCH 1/3] mm,hugetlb: use folio fields in second tail page Kirill A. Shutemov
2022-11-10 1:58 ` Hugh Dickins
2022-11-03 1:51 ` [PATCH 2/3] mm,thp,rmap: simplify compound page mapcount handling Hugh Dickins
2022-11-05 19:51 ` Kirill A. Shutemov
2022-11-10 2:49 ` Hugh Dickins
2022-11-03 1:53 ` [PATCH 3/3] mm,thp,rmap: lock_compound_mapcounts() on THP mapcounts Hugh Dickins
2022-11-05 20:06 ` Kirill A. Shutemov
2022-11-10 3:31 ` Hugh Dickins
2022-11-10 2:18 ` Hugh Dickins [this message]
2022-11-10 3:23 ` [PATCH 4/3] mm,thp,rmap: handle the normal !PageCompound case first Linus Torvalds
2022-11-10 4:21 ` Hugh Dickins
2022-11-10 16:31 ` Matthew Wilcox
2022-11-10 16:58 ` Linus Torvalds
2022-11-18 9:08 ` [PATCH 0/3] mm,thp,rmap: rework the use of subpages_mapcount Hugh Dickins
2022-11-18 9:12 ` [PATCH 1/3] mm,thp,rmap: subpages_mapcount of PTE-mapped subpages Hugh Dickins
2022-11-19 0:12 ` Yu Zhao
2022-11-19 0:37 ` Hugh Dickins
2022-11-19 1:35 ` [PATCH 1/3 fix] mm,thp,rmap: subpages_mapcount of PTE-mapped subpages: fix Hugh Dickins
2022-11-21 12:38 ` Kirill A. Shutemov
2022-11-22 9:13 ` Hugh Dickins
2022-11-21 12:36 ` [PATCH 1/3] mm,thp,rmap: subpages_mapcount of PTE-mapped subpages Kirill A. Shutemov
2022-11-22 9:03 ` Hugh Dickins
2022-11-18 9:14 ` [PATCH 2/3] mm,thp,rmap: subpages_mapcount COMPOUND_MAPPED if PMD-mapped Hugh Dickins
2022-11-21 13:09 ` Kirill A. Shutemov
2022-11-22 9:33 ` Hugh Dickins
2022-11-18 9:16 ` [PATCH 3/3] mm,thp,rmap: clean up the end of __split_huge_pmd_locked() Hugh Dickins
2022-11-21 13:24 ` Kirill A. Shutemov
2022-11-18 20:18 ` [PATCH 0/3] mm,thp,rmap: rework the use of subpages_mapcount Linus Torvalds
2022-11-18 20:42 ` Johannes Weiner
2022-11-18 20:51 ` Hugh Dickins
2022-11-18 22:03 ` Andrew Morton
2022-11-18 22:07 ` Linus Torvalds
2022-11-18 22:10 ` Hugh Dickins
2022-11-18 22:23 ` Andrew Morton
2022-11-21 16:59 ` Shakeel Butt
2022-11-21 17:16 ` Linus Torvalds
2022-11-22 16:27 ` Shakeel Butt
2022-11-21 18:52 ` Johannes Weiner
2022-11-22 1:32 ` Hugh Dickins
2022-11-22 5:57 ` Matthew Wilcox
2022-11-22 6:55 ` Johannes Weiner
2022-11-22 16:30 ` Shakeel Butt
2022-11-22 9:38 ` [PATCH v2 " Hugh Dickins
2022-11-22 9:42 ` [PATCH v2 1/3] mm,thp,rmap: subpages_mapcount of PTE-mapped subpages Hugh Dickins
2022-11-22 9:49 ` [PATCH v2 2/3] mm,thp,rmap: subpages_mapcount COMPOUND_MAPPED if PMD-mapped Hugh Dickins
2022-11-22 9:51 ` [PATCH v2 3/3] mm,thp,rmap: clean up the end of __split_huge_pmd_locked() Hugh Dickins
2022-12-05 1:38 ` Hugh Dickins
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=fca2f694-2098-b0ef-d4e-f1d8b94d318c@google.com \
--to=hughd@google.com \
--cc=akpm@linux-foundation.org \
--cc=almasrymina@google.com \
--cc=david@redhat.com \
--cc=hannes@cmpxcg.org \
--cc=jhubbard@nvidia.com \
--cc=jthoughton@google.com \
--cc=kirill@shutemov.name \
--cc=linmiaohe@huawei.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mike.kravetz@oracle.com \
--cc=naoya.horiguchi@linux.dev \
--cc=peterx@redhat.com \
--cc=shy828301@gmail.com \
--cc=sidhartha.kumar@oracle.com \
--cc=songmuchun@bytedance.com \
--cc=torvalds@linux-foundation.org \
--cc=vbabka@suse.cz \
--cc=willy@infradead.org \
--cc=zokeefe@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox