From: "Zach O'Keefe" <zokeefe@google.com>
To: Andrew Morton <akpm@linux-foundation.org>
Cc: linux-mm@kvack.org, Hugh Dickins <hughd@google.com>,
Yang Shi <shy828301@gmail.com>,
Miaohe Lin <linmiaohe@huawei.com>,
"Zach O'Keefe" <zokeefe@google.com>
Subject: [PATCH mm-unstable 2/4] mm/khugepaged: consistently order cc->is_khugepaged and pte_* checks
Date: Wed, 20 Jul 2022 07:06:01 -0700 [thread overview]
Message-ID: <20220720140603.1958773-3-zokeefe@google.com> (raw)
In-Reply-To: <20220720140603.1958773-1-zokeefe@google.com>
cc->is_khugepaged is used to predicate the khugepaged-only behavior
of enforcing khugepaged heuristics limited by the sysfs knobs
khugepaged_max_ptes_[none|swap|shared].
In branches where khugepaged_max_ptes_* is checked, consistently check
cc->is_khugepaged first. Also, local counters (for comparison vs
khugepaged_max_ptes_* limits) were previously incremented in the
comparison expression. Some of these counters (unmapped) are
additionally used outside of khugepaged_max_ptes_* enforcement, and
all counters are communicated in tracepoints. Move the correct
accounting of these counters before branching statements to avoid future
errors due to C's short-circuiting evaluation.
Fixes: 9fab4752a181 ("mm/khugepaged: add flag to predicate khugepaged-only behavior")
Link: https://lore.kernel.org/linux-mm/Ys2qJm6FaOQcxkha@google.com/
Signed-off-by: Zach O'Keefe <zokeefe@google.com>
---
mm/khugepaged.c | 49 +++++++++++++++++++++++++++++--------------------
1 file changed, 29 insertions(+), 20 deletions(-)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index ecd28bfeab60..290422577172 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -574,9 +574,10 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,
pte_t pteval = *_pte;
if (pte_none(pteval) || (pte_present(pteval) &&
is_zero_pfn(pte_pfn(pteval)))) {
+ ++none_or_zero;
if (!userfaultfd_armed(vma) &&
- (++none_or_zero <= khugepaged_max_ptes_none ||
- !cc->is_khugepaged)) {
+ (!cc->is_khugepaged ||
+ none_or_zero <= khugepaged_max_ptes_none)) {
continue;
} else {
result = SCAN_EXCEED_NONE_PTE;
@@ -596,11 +597,14 @@ static int __collapse_huge_page_isolate(struct vm_area_struct *vma,
VM_BUG_ON_PAGE(!PageAnon(page), page);
- if (cc->is_khugepaged && page_mapcount(page) > 1 &&
- ++shared > khugepaged_max_ptes_shared) {
- result = SCAN_EXCEED_SHARED_PTE;
- count_vm_event(THP_SCAN_EXCEED_SHARED_PTE);
- goto out;
+ if (page_mapcount(page) > 1) {
+ ++shared;
+ if (cc->is_khugepaged &&
+ shared > khugepaged_max_ptes_shared) {
+ result = SCAN_EXCEED_SHARED_PTE;
+ count_vm_event(THP_SCAN_EXCEED_SHARED_PTE);
+ goto out;
+ }
}
if (PageCompound(page)) {
@@ -1170,8 +1174,9 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,
_pte++, _address += PAGE_SIZE) {
pte_t pteval = *_pte;
if (is_swap_pte(pteval)) {
- if (++unmapped <= khugepaged_max_ptes_swap ||
- !cc->is_khugepaged) {
+ ++unmapped;
+ if (!cc->is_khugepaged ||
+ unmapped <= khugepaged_max_ptes_swap) {
/*
* Always be strict with uffd-wp
* enabled swap entries. Please see
@@ -1189,9 +1194,10 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,
}
}
if (pte_none(pteval) || is_zero_pfn(pte_pfn(pteval))) {
+ ++none_or_zero;
if (!userfaultfd_armed(vma) &&
- (++none_or_zero <= khugepaged_max_ptes_none ||
- !cc->is_khugepaged)) {
+ (!cc->is_khugepaged ||
+ none_or_zero <= khugepaged_max_ptes_none)) {
continue;
} else {
result = SCAN_EXCEED_NONE_PTE;
@@ -1221,12 +1227,14 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm,
goto out_unmap;
}
- if (cc->is_khugepaged &&
- page_mapcount(page) > 1 &&
- ++shared > khugepaged_max_ptes_shared) {
- result = SCAN_EXCEED_SHARED_PTE;
- count_vm_event(THP_SCAN_EXCEED_SHARED_PTE);
- goto out_unmap;
+ if (page_mapcount(page) > 1) {
+ ++shared;
+ if (cc->is_khugepaged &&
+ shared > khugepaged_max_ptes_shared) {
+ result = SCAN_EXCEED_SHARED_PTE;
+ count_vm_event(THP_SCAN_EXCEED_SHARED_PTE);
+ goto out_unmap;
+ }
}
page = compound_head(page);
@@ -1961,8 +1969,9 @@ static int khugepaged_scan_file(struct mm_struct *mm, struct file *file,
continue;
if (xa_is_value(page)) {
+ ++swap;
if (cc->is_khugepaged &&
- ++swap > khugepaged_max_ptes_swap) {
+ swap > khugepaged_max_ptes_swap) {
result = SCAN_EXCEED_SWAP_PTE;
count_vm_event(THP_SCAN_EXCEED_SWAP_PTE);
break;
@@ -2013,8 +2022,8 @@ static int khugepaged_scan_file(struct mm_struct *mm, struct file *file,
rcu_read_unlock();
if (result == SCAN_SUCCEED) {
- if (present < HPAGE_PMD_NR - khugepaged_max_ptes_none &&
- cc->is_khugepaged) {
+ if (cc->is_khugepaged &&
+ present < HPAGE_PMD_NR - khugepaged_max_ptes_none) {
result = SCAN_EXCEED_NONE_PTE;
count_vm_event(THP_SCAN_EXCEED_NONE_PTE);
} else {
--
2.37.0.170.g444d1eabd0-goog
next prev parent reply other threads:[~2022-07-20 14:06 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-07-20 14:05 [PATCH mm-unstable 0/4] mm: fixes for userspace hugepage collapse, v7 Zach O'Keefe
2022-07-20 14:06 ` [PATCH mm-unstable 1/4] mm/khugepaged: Use minimal bits to store num page < HPAGE_PMD_NR Zach O'Keefe
2022-07-20 17:26 ` Yang Shi
2022-07-21 0:41 ` David Rientjes
2022-07-20 14:06 ` Zach O'Keefe [this message]
2022-07-20 17:27 ` [PATCH mm-unstable 2/4] mm/khugepaged: consistently order cc->is_khugepaged and pte_* checks Yang Shi
2022-07-20 19:09 ` Zach O'Keefe
2022-07-21 0:42 ` David Rientjes
2022-07-20 14:06 ` [PATCH mm-unstable 3/4] mm/khugepaged: delay computation of hpage boundaries until use Zach O'Keefe
2022-07-20 17:30 ` Yang Shi
2022-07-21 0:43 ` David Rientjes
2022-07-20 14:06 ` [PATCH mm-unstable 4/4] Revert "mm/madvise: add huge_memory:mm_madvise_collapse tracepoint" Zach O'Keefe
2022-07-20 17:30 ` Yang Shi
2022-07-21 0:43 ` David Rientjes
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220720140603.1958773-3-zokeefe@google.com \
--to=zokeefe@google.com \
--cc=akpm@linux-foundation.org \
--cc=hughd@google.com \
--cc=linmiaohe@huawei.com \
--cc=linux-mm@kvack.org \
--cc=shy828301@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox