From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A53A3C61D85 for ; Thu, 23 Nov 2023 07:01:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 42C9D6B053C; Thu, 23 Nov 2023 02:01:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3DD2C6B053D; Thu, 23 Nov 2023 02:01:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2A4A76B05AB; Thu, 23 Nov 2023 02:01:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 1C46D6B053C for ; Thu, 23 Nov 2023 02:01:14 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id CA9524075E for ; Thu, 23 Nov 2023 07:01:13 +0000 (UTC) X-FDA: 81488322426.27.0586D8D Received: from out0-201.mail.aliyun.com (out0-201.mail.aliyun.com [140.205.0.201]) by imf05.hostedemail.com (Postfix) with ESMTP id ACD6B100021 for ; Thu, 23 Nov 2023 07:01:11 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=antgroup.com; spf=pass (imf05.hostedemail.com: domain of henry.hj@antgroup.com designates 140.205.0.201 as permitted sender) smtp.mailfrom=henry.hj@antgroup.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700722872; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=E+oScbwHyUU41wMxgpjjJBi2Mu5nq8JOAP1ZxPXdc+I=; b=zhaHqjgMiaycJTnKD3B0lrLgrfDGAg5E5w4NZMWaMCrK55+fyvwHCeQF1rN1r3uUw+rnwX w4BpvwvmYBsp+uXOjKG9ZSpMA4wfc3jmBOmMziOF7dYx3zaDlZymJSbQByMAJDjoAO393K cvUqOoETDPtjmT7O7T4Kw3Wvh1g5j/g= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=none; dmarc=pass (policy=quarantine) header.from=antgroup.com; spf=pass (imf05.hostedemail.com: domain of henry.hj@antgroup.com designates 140.205.0.201 as permitted sender) smtp.mailfrom=henry.hj@antgroup.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700722872; a=rsa-sha256; cv=none; b=ygyMrHn9x3RmZZw1wVRzINNyfQDbHwZ/nkFIVty6UC/xEem1wkXmx1eWm8OGDIrozmH/OS JxDYLJ8M/drjq7YJM8jNDlpVSc8VI7VMc03dpAENHUbIEyfmHKlBfzhe+5lHMuCwsu8EUw tt8gHhSWcxCntQ3mb+9xcuTImp7yFPE= X-Alimail-AntiSpam:AC=PASS;BC=-1|-1;BR=01201311R111e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=ay29a033018047205;MF=henry.hj@antgroup.com;NM=1;PH=DS;RN=6;SR=0;TI=SMTPD_---.VU7afWl_1700722866; Received: from localhost(mailfrom:henry.hj@antgroup.com fp:SMTPD_---.VU7afWl_1700722866) by smtp.aliyun-inc.com; Thu, 23 Nov 2023 15:01:07 +0800 From: "=?UTF-8?B?6buE6Z2WKOm7hOmdlik=?=" To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: "=?UTF-8?B?6LCI6Ym06ZSL?=" , "=?UTF-8?B?5pyx6L6JKOiMtuawtCk=?=" , , "=?UTF-8?B?6buE6Z2WKOm7hOmdlik=?=" Subject: [RFC] mm: Multi-Gen LRU: fix use mm/page_idle/bitmap Date: Thu, 23 Nov 2023 14:58:11 +0800 Message-ID: X-Mailer: git-send-email 2.43.0 In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: ACD6B100021 X-Stat-Signature: np7ejw44s3k68puamx78gkxou8g3693g X-Rspam-User: X-HE-Tag: 1700722871-639378 X-HE-Meta: U2FsdGVkX19uh/Wjxv4969H5dP97eDMWoZDiZv679CezrjbOohmP+X/El6by9u2ehXfOHIEvWOP8OYCuTJWkhPnKlttxrZfgjZthkmv46Kq3TUanxDPyDDqOmHdfaX6XiTiqy62gG8//AKlzp5WqT6jSL9tSAmTadrSuWJpxVcVxj10vEBXc0CzN0nSu7Bjln4zfGK1CLa/PqoudX/5hQHW+agYpx4fqWAaqAOsOqYr6SBBAYuUzwmxuNdOFGvTNNGv9wNgcbjD6N9YrSBDq6KMbcTZkg6UTcp6ctacv0LA81QUkR0JbCzmV/doVjm8ddZdlN5YsYqDi6nVGzf+nShHuZOpUk59ykJLgfXFE7SsDjT9CHFmxzQwS/eQ8xvrviZDw7X9HDJDBoK8Ayfje6eQ/SDuhkn4cta9379VW9y2v7DUDGtWMnkQEoVVCN+YfWQSdUnFxIZYTD4QvUmDFrhbh5yT1QwutSCYBEHwmFVZBkfoLxPWWkY8UxZvsgO9g75pFJWW7dk8SoqsoxZgjaNacncadmssX1+9AYypGsjU8u1k/SxzTEdERU0ljnS0sGQpk7WVWOezzZzNFamW1KUN6fG9N+2O1DVBwc36wOKOtf6YZTe9ToNHWpvl97ZNmCdG+yKnahAf3GOFA2i7pm+IId9rLxH7ulay+1CZnkpaOxjvOhphEINGqg6mCTzAhMbCRCFRimYgl3Ku+EzC22/9WPJbvK98R+oiveMjuwD6A4jsg6SW6KdH0rcPcua8syEynwhUtihp8AeWo7wGguhvWXQrbonLFICdr6Wujsd6Jb6biQR1fn6PUDQH58v0nxbbZjEiT0s/03FAAbACPFCYEUdHq1/d2iHf9IbNGv674bXdh1RaHEAI9C9UfbrGKNhcmpkYvrucDzvAT0pllvm5GrNQOW1Yx+2ElnDxbvE1WqcbYKQnuFckaXJSDKEWzvw0mbGhrz5fTKuDaKEF twNXrKjF 6hOuSQO8xvjotbSDr7z93Ibb902m8ZwJq1uTdCqLE9RQlpaRH4GD4D2Hrwna/LC8OLYQ8pPycthTzuathnDDwiNHtA6R8mSqBJT103HF/b4qmrvZPzbxsUShh53eEdRiLmhwqzE4OpCJnnPnjDrdZRdxrdRpcPHuVcGrJtqNasU1/nUFk4v4qcq/nPNX1wqmmo2gVu3LexaCrU9I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Multi-Gen LRU page-table walker clears pte young flag, but it doesn't clear page idle flag. When we use /sys/kernel/mm/page_idle/bitmap to check whether one page is accessed, it would tell us this page is idle, but actually this page has been accessed. For those unmapped filecache pages, page idle flag would not been cleared in folio_mark_accessed if Multi-Gen LRU is enabled. So we couln't use /sys/kernel/mm/page_idle/bitmap to check whether a filecache page is read or written. What's more, /sys/kernel/mm/page_idle/bitmap also clears pte young flag. If one page is accessed, it would set page young flag. Multi-Gen LRU page-table walker should check both page&pte young flags. how-to-reproduce-problem idle_page_track a tools to track process accessed memory during a specific time usage idle_page_track $pid $time how-it-works 1. scan process vma from /proc/$pid/maps 2. vfn --> pfn from /proc/$pid/pagemap 3. write /sys/kernel/mm/page_idle/bitmap to mark phy page idle flag and clear pte young flag 4. sleep $time 5. read /sys/kernel/mm/page_idle/bitmap to test_and_clear pte young flag and return whether phy page is accessed test ---- test program #include #include #include #include #include #include #include int main(int argc, const char *argv[]) { char *buf = NULL; char pipe_info[4096]; int n; int fd = -1; buf = malloc(1024*1024*1024UL); memset(buf, 0, 1024*1024*1024UL); fd = open("access.pipe", O_RDONLY); if (fd < 0) goto out; while (1) { n = read(fd, pipe_info, sizeof(pipe_info)); if (!n) { sleep(1); continue; } else if (n < 0) { break; } memset(buf, 0, 1024*1024*1024UL); puts("finish access"); } out: if (fd >=0) close(fd); if (buf) free(buf); return 0; } prepare: mkfifo access.pipe ./test ps -ef | grep test root 4106 3148 8 06:47 pts/0 00:00:01 ./test We use /sys/kernel/debug/lru_gen to simulate mglru page-table scan. case 1: mglru walker break page_idle ./idle_page_track 4106 60 & sleep 5; echo 1 > access.pipe sleep 5; echo '+ 8 0 6 1 1' > /sys/kernel/debug/lru_gen the output of idle_page_track is: Est(s) Ref(MB) 64.822 1.00 only found 1MB were accessed during 64.822s, but actually 1024MB were accessed. case 2: page_idle break mglru walker echo 1 > access.pipe ./idle_page_track 4106 10 echo '+ 8 0 7 1 1' > /sys/kernel/debug/lru_gen lru gen status: memcg 8 /user.slice node 0 5 772458 1065 9735 6 737435 262244 72 7 538053 1184 632 8 59404 6422 0 almost pages should be in max_seq-1 queue, but actually not. Signed-off-by: Henry Huang --- mm/swap.c | 3 +++ mm/vmscan.c | 37 ++++++++++++++++++++++++++----------- 2 files changed, 29 insertions(+), 11 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index cd8f0150ba3a..4bd14aabdc10 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -416,6 +416,9 @@ static void folio_inc_refs(struct folio *folio) { unsigned long new_flags, old_flags = READ_ONCE(folio->flags); + if (folio_test_idle(folio)) + folio_clear_idle(folio); + if (folio_test_unevictable(folio)) return; diff --git a/mm/vmscan.c b/mm/vmscan.c index 96abaa5a973e..4f41bef5bca5 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3355,6 +3355,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, unsigned long pfn; struct folio *folio; pte_t ptent = ptep_get(pte + i); + int is_pte_young; total++; walk->mm_stats[MM_LEAF_TOTAL]++; @@ -3363,18 +3364,22 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, if (pfn == -1) continue; - if (!pte_young(ptent)) { - walk->mm_stats[MM_LEAF_OLD]++; - continue; - } - folio = get_pfn_folio(pfn, memcg, pgdat, walk->can_swap); if (!folio) continue; - if (!ptep_test_and_clear_young(args->vma, addr, pte + i)) + is_pte_young = pte_young(ptent); + if (!folio_test_clear_young(folio) && !is_pte_young) { + walk->mm_stats[MM_LEAF_OLD]++; + continue; + } + + if (is_pte_young && !ptep_test_and_clear_young(args->vma, addr, pte + i)) VM_WARN_ON_ONCE(true); + if (folio_test_idle(folio)) + folio_clear_idle(folio); + young++; walk->mm_stats[MM_LEAF_YOUNG]++; @@ -3435,6 +3440,7 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area do { unsigned long pfn; struct folio *folio; + int is_pmd_young; /* don't round down the first address */ addr = i ? (*first & PMD_MASK) + i * PMD_SIZE : *first; @@ -3453,9 +3459,13 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area if (!folio) goto next; - if (!pmdp_test_and_clear_young(vma, addr, pmd + i)) + is_pmd_young = pmdp_test_and_clear_young(vma, addr, pmd + i); + if (!folio_test_clear_young(folio) && !is_pmd_young) goto next; + if (folio_test_idle(folio)) + folio_clear_idle(folio); + walk->mm_stats[MM_LEAF_YOUNG]++; if (pmd_dirty(pmd[i]) && !folio_test_dirty(folio) && @@ -4025,21 +4035,26 @@ void lru_gen_look_around(struct page_vma_mapped_walk *pvmw) for (i = 0, addr = start; addr != end; i++, addr += PAGE_SIZE) { unsigned long pfn; pte_t ptent = ptep_get(pte + i); + int is_pte_young; pfn = get_pte_pfn(ptent, pvmw->vma, addr); if (pfn == -1) continue; - if (!pte_young(ptent)) - continue; - folio = get_pfn_folio(pfn, memcg, pgdat, can_swap); if (!folio) continue; - if (!ptep_test_and_clear_young(pvmw->vma, addr, pte + i)) + is_pte_young = pte_young(ptent); + if (!folio_test_clear_young(folio) && !is_pte_young) + continue; + + if (is_pte_young && !ptep_test_and_clear_young(pvmw->vma, addr, pte + i)) VM_WARN_ON_ONCE(true); + if (folio_test_idle(folio)) + folio_clear_idle(folio); + young++; if (pte_dirty(ptent) && !folio_test_dirty(folio) && -- 2.43.0