From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADEF9C4829A for ; Wed, 14 Feb 2024 06:06:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C9F98D0016; Wed, 14 Feb 2024 01:06:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 37A5B8D0001; Wed, 14 Feb 2024 01:06:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 225758D0016; Wed, 14 Feb 2024 01:06:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 0BAAE8D0001 for ; Wed, 14 Feb 2024 01:06:08 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id ADFA7A02C0 for ; Wed, 14 Feb 2024 06:06:07 +0000 (UTC) X-FDA: 81789373974.22.675FBE6 Received: from mail-pf1-f202.google.com (mail-pf1-f202.google.com [209.85.210.202]) by imf17.hostedemail.com (Postfix) with ESMTP id 1488B4000B for ; Wed, 14 Feb 2024 06:06:05 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=HgjvG71x; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3TFjMZQgKCGMLJOTFZIPHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--kinseyho.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3TFjMZQgKCGMLJOTFZIPHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--kinseyho.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707890766; a=rsa-sha256; cv=none; b=0nAwvFwCyovaQzV75sK8ZdgcACRqICpCZQTa+xZlalPffOfO2WAtQBJQy/zDaJwInKM8Tb h0PSDs1ba9vCJqCrT4A0Vmz70zIv7OsQJmWr1q9N7+kjcpJ3saNGaV+KXNRDOOjZLgtcrs LDtvMQbgac5r5C7kO81tQ2d7CRzmOrQ= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=HgjvG71x; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of 3TFjMZQgKCGMLJOTFZIPHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--kinseyho.bounces.google.com designates 209.85.210.202 as permitted sender) smtp.mailfrom=3TFjMZQgKCGMLJOTFZIPHPPHMF.DPNMJOVY-NNLWBDL.PSH@flex--kinseyho.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707890766; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QH8wNJ7lAHDhvm3bFylVV17pyKkkH6HT7TXZodNBMaI=; b=2PHrJ9S6n+k9OjwI9hiy3QEhZEnqlLMdSizhZpuEHLvZbFberT1SsB7ldv5Z3btCvV5ap7 fJ7WC4Gf8tWNOWN4P7KxSZ/9jhfGSopPdzWqH3BvrCUVA0dY1XkxuxCjZehscvAPUIBc6x CU8bln5+4iHWFhI+JaGZl/mASz1mCW8= Received: by mail-pf1-f202.google.com with SMTP id d2e1a72fcca58-6e0ab5dbe7cso4235266b3a.1 for ; Tue, 13 Feb 2024 22:06:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1707890765; x=1708495565; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=QH8wNJ7lAHDhvm3bFylVV17pyKkkH6HT7TXZodNBMaI=; b=HgjvG71xXQ8HYGevz1EHc0cjtqB9Qy3MXYuTjN1Mi6ixGX/VifrehMuF4sYLHanzTW YPkctt/2DDepEW6tBJpi0dvH6RFjvwUzPArZ2pPU94AJDxy5cgAAHlXHqW7HA3O8ZqZR pxkq5wwR3oS1DW+LjBEwnzBhC6WxR8g49TFYZVisb+H6JQrZgKEsa7Su0npb7MeLvSBG rWJgLZTPK7YLwwdojsLDRGcVBVL5DM3c9XrRi2NcxxcR2Y93YvurzY1KVzioZGEHICEM V6l+hU8Roh9QQEyCY3Z7ZciE2iJkC/snzQ6F23vizj4fCD5yKxTBBdHxNiDjyRM1oJy8 yx/w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1707890765; x=1708495565; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=QH8wNJ7lAHDhvm3bFylVV17pyKkkH6HT7TXZodNBMaI=; b=e0QRgkWg1CzmBwGwNrXl4jCelQTWQ88adhmmXUn7gRAg+38CrFYGZoHdoNhg8VSzaP TxZBO3FxfC8f9Jhiqsb0FVZfSgZrMQa+KEnpnNy+N06WJH2+BLNib312xNI2W5eOk7rs yGX8aw4NkeJBQoZYjtginnPV5U5n9q6D2YHOFXoxfVKutzAt3iWb0Pmtz/ZYa0b+ZIKJ ZY4q+McfpTARgoLjUxWFD7zQuKfbgvkzkdqSQE2iMzUfYA1HOsUNiMIL18Tp9/3FTEdD LEN4iAmK5lIkY1hGQqaWVcijJvjfPi4t017tNZG7BIQF4vAEC9hAd00QSGF4jqtt1513 +Bvw== X-Gm-Message-State: AOJu0Yxzn6APCHgrlLCV8vWnOiGuVAwNubAZ8k+RbBTKdSiGNl+zK/A1 t2IROMVOIutmi3lhQlVQgsSvnSy1K0txQSu0jprdOvrdIoNSwR+GZh6RP/Fm06LogrJcRPUUOIh xlcGZqcr5ig== X-Google-Smtp-Source: AGHT+IHQunAmjkzKIPCb68OjVMfuFVv2mPMH+2t6Wiy6R/pOgsm2Z2RPiWQL7A62g5M5NjpLQpMX+lR14ejk/A== X-Received: from kinseyct.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:46b]) (user=kinseyho job=sendgmr) by 2002:a05:6a00:2d9e:b0:6e0:a9bc:ef63 with SMTP id fb30-20020a056a002d9e00b006e0a9bcef63mr112938pfb.0.1707890764929; Tue, 13 Feb 2024 22:06:04 -0800 (PST) Date: Wed, 14 Feb 2024 06:05:37 +0000 In-Reply-To: <20240214060538.3524462-1-kinseyho@google.com> Mime-Version: 1.0 References: <20240214060538.3524462-1-kinseyho@google.com> X-Mailer: git-send-email 2.43.0.687.g38aa6559b0-goog Message-ID: <20240214060538.3524462-5-kinseyho@google.com> Subject: [PATCH mm-unstable v1 4/5] mm/mglru: improve struct lru_gen_mm_walk From: Kinsey Ho To: Andrew Morton Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yu Zhao , Donet Tom , "Aneesh Kumar K . V" , Kinsey Ho Content-Type: text/plain; charset="UTF-8" X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 1488B4000B X-Stat-Signature: jxtqijkojny7bnhfnrtc5uoa3tjm6hpx X-HE-Tag: 1707890765-890400 X-HE-Meta: U2FsdGVkX1/I+hKfI/NDh+WW9oUa8FQXqSVfPdUqK/OZnlDjibXC90hYUcrQ89EC2DhBAEcT3qSTEcrb5SzsVFXaUNNKELlTvH978SYEbz0oyNIDtLD8SStJdW7bYmS4uE9U3YVM2cdZ0p8riVGdIlANSJzZUlef26h+dLF+/D51E7YdKkOPE70ueK8TGGFdqi3yKNRYyuhnvrf97bt0HpPtAj97TuT1o+9XHaD9Yhqv29ZgAFoobpvIam1l7QPHSSNIXB6PGZp+CcKvt1lUn26kRlQx4tB4wBlE0gzfCQRTH1P4O+3cvv35BObdHRZ67bUBU15JxiDfEqBWZ08sYN4ciGYpjEue7FrrFWJHFj6jeyAqYJ0j61vI6s2WCKhaUQlgGh/YukzRVHa0stEseEvqGlnSkm+ecSmmmjv/0/I7vflWL5IHnE0woP2loVxl6fE8hOTlPBzQ1rR8zsn7ONVEBg1orglEUsl+YNctYP6sGwMKFkjmJauxNC56CyTufnfYg/cTx+ZGfRZXlEkM5EM9zrgBwtp9b9y7975lUIXC4dJp/S0OG01e6VBTsOaZB6kAqNTemJJdZZ//TxzP5zmW6PI/qgFs3RgJqqsGDVyuWQ3kNIc6tc7UNMPP9Fe8q90MTc3mmkDfl9JLo14OjPthCzt1Wo60JYM+NxzjdrOxrbGn27iMafvoKiA40wsQz2qOtRF8Ez+tkoju9uUnfvj0S63n+710U1+qfoa9vKFrdRf8w198uEMFhLDQgX3v53MDqIa8+i99AXu0xt6ZRJMhcWUc5W/8vGYL7RYoPFtuLTknj+1sRVa9lqsUthW7/b98JGripmRm/GCxasE+933jG3YsrxiPha1SORAQ4Tu2D7feUiWIRKqn8zguGI1T5MPwvCH8jtqSQetgIrTIHZ2xcGuWbAaey4sUbccqlNtaGVhhvLfuUbLc7Adjk1ST4wZ39pCYBIp9HZzJq5E vZlPhF2Q noZIOQcg6LxDxrq/3BQKdGTXLvSOq1HuWE5gwSQuRSSNHuN2eHqQF6lgUoEFNCa5PaJJfxEGQuL8plhwxj7NrdS/jw8a+BpDQjh5rbeMMA8EpWhmuBAY0OeQMa+uulKIylqwrSGId0WQjzDYifu+z4/43+QdQV5vkO5/6QnC+1dywDzUTPG8xJYvg7g77f7fRA1rBcmjsM46q0blWQOV/vnTxn12qKg4KAj/HkIXL9S6YBGQOdKexPqHSML9U/Qi5n5dtv0HtmZfoeZBO6hb3vTaRdjoghwTClA7J5IcItfD/8EmaDH142dD5pG/63dzkECk/g9tnIXzPnrZHHrpfEPcCI5AsUOxBToXcuRT7649PMnMHzYxAwL4MaE9bETKYc0xCDStus0U6uXpByFuCRuIKACZOU0yhWqWm X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Rename max_seq to seq in struct lru_gen_mm_walk to keep consistent with struct lru_gen_mm_state. Note that seq is not always up to date with max_seq from lru_gen_folio. No functional changes. Signed-off-by: Kinsey Ho --- include/linux/mmzone.h | 6 ++--- mm/vmscan.c | 50 ++++++++++++++++++++++-------------------- 2 files changed, 29 insertions(+), 27 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 22fd46c77840..e51c75270e82 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -468,7 +468,7 @@ enum { #define NR_BLOOM_FILTERS 2 struct lru_gen_mm_state { - /* set to max_seq after each iteration */ + /* synced with max_seq after each iteration */ unsigned long seq; /* where the current iteration continues after */ struct list_head *head; @@ -483,8 +483,8 @@ struct lru_gen_mm_state { struct lru_gen_mm_walk { /* the lruvec under reclaim */ struct lruvec *lruvec; - /* unstable max_seq from lru_gen_folio */ - unsigned long max_seq; + /* max_seq from lru_gen_folio: can be out of date */ + unsigned long seq; /* the next address within an mm to scan */ unsigned long next_addr; /* to batch promoted pages */ diff --git a/mm/vmscan.c b/mm/vmscan.c index 8c77c332a67a..f1019afa7813 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2906,7 +2906,7 @@ static void reset_mm_stats(struct lru_gen_mm_walk *walk, bool last) lockdep_assert_held(&get_mm_list(lruvec_memcg(lruvec))->lock); - hist = lru_hist_from_seq(walk->max_seq); + hist = lru_hist_from_seq(walk->seq); for (i = 0; i < NR_MM_STATS; i++) { WRITE_ONCE(mm_state->stats[hist][i], @@ -2915,7 +2915,7 @@ static void reset_mm_stats(struct lru_gen_mm_walk *walk, bool last) } if (NR_HIST_GENS > 1 && last) { - hist = lru_hist_from_seq(walk->max_seq + 1); + hist = lru_hist_from_seq(walk->seq + 1); for (i = 0; i < NR_MM_STATS; i++) WRITE_ONCE(mm_state->stats[hist][i], 0); @@ -2944,9 +2944,9 @@ static bool iterate_mm_list(struct lru_gen_mm_walk *walk, struct mm_struct **ite */ spin_lock(&mm_list->lock); - VM_WARN_ON_ONCE(mm_state->seq + 1 < walk->max_seq); + VM_WARN_ON_ONCE(mm_state->seq + 1 < walk->seq); - if (walk->max_seq <= mm_state->seq) + if (walk->seq <= mm_state->seq) goto done; if (!mm_state->head) @@ -2976,7 +2976,7 @@ static bool iterate_mm_list(struct lru_gen_mm_walk *walk, struct mm_struct **ite spin_unlock(&mm_list->lock); if (mm && first) - reset_bloom_filter(mm_state, walk->max_seq + 1); + reset_bloom_filter(mm_state, walk->seq + 1); if (*iter) mmput_async(*iter); @@ -2986,7 +2986,7 @@ static bool iterate_mm_list(struct lru_gen_mm_walk *walk, struct mm_struct **ite return last; } -static bool iterate_mm_list_nowalk(struct lruvec *lruvec, unsigned long max_seq) +static bool iterate_mm_list_nowalk(struct lruvec *lruvec, unsigned long seq) { bool success = false; struct mem_cgroup *memcg = lruvec_memcg(lruvec); @@ -2995,9 +2995,9 @@ static bool iterate_mm_list_nowalk(struct lruvec *lruvec, unsigned long max_seq) spin_lock(&mm_list->lock); - VM_WARN_ON_ONCE(mm_state->seq + 1 < max_seq); + VM_WARN_ON_ONCE(mm_state->seq + 1 < seq); - if (max_seq > mm_state->seq) { + if (seq > mm_state->seq) { mm_state->head = NULL; mm_state->tail = NULL; WRITE_ONCE(mm_state->seq, mm_state->seq + 1); @@ -3348,7 +3348,8 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, struct lru_gen_mm_walk *walk = args->private; struct mem_cgroup *memcg = lruvec_memcg(walk->lruvec); struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec); - int old_gen, new_gen = lru_gen_from_seq(walk->max_seq); + DEFINE_MAX_SEQ(walk->lruvec); + int old_gen, new_gen = lru_gen_from_seq(max_seq); pte = pte_offset_map_nolock(args->mm, pmd, start & PMD_MASK, &ptl); if (!pte) @@ -3415,7 +3416,8 @@ static void walk_pmd_range_locked(pud_t *pud, unsigned long addr, struct vm_area struct lru_gen_mm_walk *walk = args->private; struct mem_cgroup *memcg = lruvec_memcg(walk->lruvec); struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec); - int old_gen, new_gen = lru_gen_from_seq(walk->max_seq); + DEFINE_MAX_SEQ(walk->lruvec); + int old_gen, new_gen = lru_gen_from_seq(max_seq); VM_WARN_ON_ONCE(pud_leaf(*pud)); @@ -3546,7 +3548,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end, walk_pmd_range_locked(pud, addr, vma, args, bitmap, &first); } - if (!walk->force_scan && !test_bloom_filter(mm_state, walk->max_seq, pmd + i)) + if (!walk->force_scan && !test_bloom_filter(mm_state, walk->seq, pmd + i)) continue; walk->mm_stats[MM_NONLEAF_FOUND]++; @@ -3557,7 +3559,7 @@ static void walk_pmd_range(pud_t *pud, unsigned long start, unsigned long end, walk->mm_stats[MM_NONLEAF_ADDED]++; /* carry over to the next generation */ - update_bloom_filter(mm_state, walk->max_seq + 1, pmd + i); + update_bloom_filter(mm_state, walk->seq + 1, pmd + i); } walk_pmd_range_locked(pud, -1, vma, args, bitmap, &first); @@ -3628,7 +3630,7 @@ static void walk_mm(struct mm_struct *mm, struct lru_gen_mm_walk *walk) err = -EBUSY; /* another thread might have called inc_max_seq() */ - if (walk->max_seq != max_seq) + if (walk->seq != max_seq) break; /* folio_update_gen() requires stable folio_memcg() */ @@ -3765,7 +3767,7 @@ static bool try_to_inc_min_seq(struct lruvec *lruvec, bool can_swap) return success; } -static bool inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, +static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, bool can_swap, bool force_scan) { bool success; @@ -3773,14 +3775,14 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, int type, zone; struct lru_gen_folio *lrugen = &lruvec->lrugen; restart: - if (max_seq < READ_ONCE(lrugen->max_seq)) + if (seq < READ_ONCE(lrugen->max_seq)) return false; spin_lock_irq(&lruvec->lru_lock); VM_WARN_ON_ONCE(!seq_is_valid(lruvec)); - success = max_seq == lrugen->max_seq; + success = seq == lrugen->max_seq; if (!success) goto unlock; @@ -3833,7 +3835,7 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, return success; } -static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, +static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long seq, bool can_swap, bool force_scan) { bool success; @@ -3842,13 +3844,13 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, struct lru_gen_folio *lrugen = &lruvec->lrugen; struct lru_gen_mm_state *mm_state = get_mm_state(lruvec); - VM_WARN_ON_ONCE(max_seq > READ_ONCE(lrugen->max_seq)); + VM_WARN_ON_ONCE(seq > READ_ONCE(lrugen->max_seq)); if (!mm_state) - return inc_max_seq(lruvec, max_seq, can_swap, force_scan); + return inc_max_seq(lruvec, seq, can_swap, force_scan); /* see the comment in iterate_mm_list() */ - if (max_seq <= READ_ONCE(mm_state->seq)) + if (seq <= READ_ONCE(mm_state->seq)) return false; /* @@ -3858,18 +3860,18 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, * is less efficient, but it avoids bursty page faults. */ if (!should_walk_mmu()) { - success = iterate_mm_list_nowalk(lruvec, max_seq); + success = iterate_mm_list_nowalk(lruvec, seq); goto done; } walk = set_mm_walk(NULL, true); if (!walk) { - success = iterate_mm_list_nowalk(lruvec, max_seq); + success = iterate_mm_list_nowalk(lruvec, seq); goto done; } walk->lruvec = lruvec; - walk->max_seq = max_seq; + walk->seq = seq; walk->can_swap = can_swap; walk->force_scan = force_scan; @@ -3880,7 +3882,7 @@ static bool try_to_inc_max_seq(struct lruvec *lruvec, unsigned long max_seq, } while (mm); done: if (success) { - success = inc_max_seq(lruvec, max_seq, can_swap, force_scan); + success = inc_max_seq(lruvec, seq, can_swap, force_scan); WARN_ON_ONCE(!success); } -- 2.43.0.687.g38aa6559b0-goog