From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 15B81C3271E for ; Mon, 8 Jul 2024 23:42:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1C1856B0092; Mon, 8 Jul 2024 19:42:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 170A66B0095; Mon, 8 Jul 2024 19:42:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 010496B0096; Mon, 8 Jul 2024 19:42:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D7F4F6B0092 for ; Mon, 8 Jul 2024 19:42:31 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 4997C1A15A3 for ; Mon, 8 Jul 2024 23:42:31 +0000 (UTC) X-FDA: 82318212102.11.2F52B6C Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) by imf25.hostedemail.com (Postfix) with ESMTP id 7BF92A0017 for ; Mon, 8 Jul 2024 23:42:29 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=yELvcT64; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.176 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720482125; a=rsa-sha256; cv=none; b=UkdZUY9fojQApNS8CfXqBqvozDUy4pd2v3NXsVmRqgHAawASA2+z3hTyl1QqQeyx0Uomzd dETh2z+QZghetI1MHBSLOtkYuFgD90Fq4vZok9H8F1KqlSizBUlg7WTS92G/0uJRWid2fT 1SvDEElZye21eJJ2/dIsCoBQVPcXWAc= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=yELvcT64; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.176 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720482125; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nM7n3F0up3r61YyqcXWbM5R0i4/2z/TwS0sdB2eyKMU=; b=sBo2aSQFTmz0lolZQU2OrhRZzqlvSuZfCfNwMe1ie/sJcP/WSRydHTPnPOsCRnASHIbnVd 6vJEgxU0u/qWhKP+AXXireEZgh1/gxek95jsC9QXWkpJij/AglveofQdvG/TCOSW/lj+4d 5K2v9Gz9Psy6SQW8Pv92YwlyhcxlHjg= Received: by mail-qt1-f176.google.com with SMTP id d75a77b69052e-447f8aa87bfso165701cf.0 for ; Mon, 08 Jul 2024 16:42:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720482148; x=1721086948; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=nM7n3F0up3r61YyqcXWbM5R0i4/2z/TwS0sdB2eyKMU=; b=yELvcT64/y4UU4o3HfImxHt+XGhGyXfrVLOcwRWS4ymQqsZm+QmSLaTStj2W1ZDaAB pCEqb6zs3CApZHWo5UN8Z1JLyA6ckfrxSZNWnaApyHfceIpGGXRn7grJv2NkqXJ7nznT 9JaD6v2rBhUM6rX+8Wss7fhS7EiFCNfOTArO+Zd849e5BCrda3XjaJoqbu4mZt6Ea4XH NumJTy4EGDPo1bYolzxlb7QSYtKLxOu0j4pr1bt/ltXNaCddOucWvaNWiPacjtl9MEKM HmkUf6c6cdgL3HqC+UAo70aiVIvoRXTgeT1tJ5/GVHAYKHKyIRVfDpy71RCckSF1P2Xz uSSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720482148; x=1721086948; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nM7n3F0up3r61YyqcXWbM5R0i4/2z/TwS0sdB2eyKMU=; b=madN2g+/D7ELUByGlQnPr9w8mJHZC/fQf6LyCh/EFV6fujyzA0Ut1X3miGBfsbuFwP KAOk1PZS8Bi/OOfrGhrKqSVRimve4NmOLUi4Lyij7GjzSpyydtjzbXIiPS+eCc1AFSzN HhQ8+OtIZGYL7ZA4LVuXVgNdqnZpsHVCiBifiDmZC8XR8w33T7MULk9m4ApWfFlmn7St uUkFRP+vQO5BPSQ93VZNzsX8dBxh8VajmrS3AJgPWUZTK1LQmxHG7/+FfmKa+CFOpz46 ggNoCuuXtOAU9o7NFsbU4KpCp9NpzGjzBEOrK4ygRorsn5ykumBR1U2z9LWW1VZmXt8U Lafg== X-Forwarded-Encrypted: i=1; AJvYcCXhJONOrIFlOXhs+4ZqXNp19U3gGCp8fVeIydHEBW5+eXmNeprBZY3reVuRgC5B+2tpdgqb3AwTeOo348XHqW54ry8= X-Gm-Message-State: AOJu0Yy/Tats56mBZW2cZD+1YL//SlebLzL6ZWzJqXR4ZYksYnXwQS+t KsuEaBt4i/3J04fyjhj/34YlcO9IoxCUnsOAk7WIMu/neYTURJjY7we5TSOofApIqTiU9mFpK8s GXhLZEYexOZH6BHoOolLoJpuFnDvFrsEcGCwK X-Google-Smtp-Source: AGHT+IF5waWqD1+AOGNDtNFnA/E3lGLbDiFUJIIuSfBZaHyKwMMyMoyshkPlNYgy3P4wdxcBdGual2ZDEbwUTDImpqE= X-Received: by 2002:ac8:6742:0:b0:447:e4cb:bf50 with SMTP id d75a77b69052e-447fb2f6c08mr1579121cf.8.1720482148269; Mon, 08 Jul 2024 16:42:28 -0700 (PDT) MIME-Version: 1.0 References: <20240611002145.2078921-1-jthoughton@google.com> <20240611002145.2078921-9-jthoughton@google.com> In-Reply-To: From: Yu Zhao Date: Mon, 8 Jul 2024 17:41:49 -0600 Message-ID: Subject: Re: [PATCH v5 8/9] mm: multi-gen LRU: Have secondary MMUs participate in aging To: James Houghton Cc: Andrew Morton , Paolo Bonzini , Ankit Agrawal , Axel Rasmussen , Catalin Marinas , David Matlack , David Rientjes , James Morse , Jonathan Corbet , Marc Zyngier , Oliver Upton , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Suzuki K Poulose , Wei Xu , Will Deacon , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 7BF92A0017 X-Stat-Signature: eihcuncd3ajhwfyrr846j8qc9f8g3rgf X-Rspam-User: X-HE-Tag: 1720482149-846456 X-HE-Meta: U2FsdGVkX18/jTKUn5KtS2gPJjdrScfBgNroY3Zow7YQVmTxyrJjjVLUO/kcmsmmmbTS5LRPp9S+URoQFcXXJcD84IcFDEw/twb2RVFBqud9BQVOpZ9L5G478oWhq5YrkLlVNYg5E2ujn4WiA3ka7Lp7wUDysN3XRLDFMw15fDsRjKAJiZaJhj0bdxnxJY2XA9+BAVXKrjk/HfrTYR989Q5Jg1NxpjImBlbnaTlv9l9Cm6IhgiTfr/wcQzcLAG4nhDibbqZpAToMggpgfeASchMzqr08Xq9J0x64lOUVsFyB51a5qWgS3gNidosA/qmVU41wIe0/b0u9UVn1FxvIFGwMsZcELNCcyxSLhJa8BvMWgv7q7SCfnvmtANNZtGtf2rFp6v6NUAmqjr5HmWPQS/zt7k4Ihrg8WOfDHsDfob52SVcKmInSPg4DAW2XfqkAV+iHnoHL61LG2dZ7813wkxGGGGT+06vvtIooscS+eXmqHJ+eNKnBdL3+M03R9KxV6dz+pJqgiPbm+5QrUjKyhwkvT7oAXjKzCD3lT/2FN2+bKc1gJOqe5zF+EBK67CO4dxUxzYNy6fwjDSpWIXH0dFubO4uT7QXWuh/Pn06anvOPpaiSTdAMByK6AsjJkMrIQC3oti3IKo26HZMUEyADyhuZlES+LJZvwAKBOkiE2Q1dxYLHSOk7p907tUBQGzyYHmQoFCKYdvYVBsAG7ICOmWB+bhnS+9BO2+BiN21tADRA4ahlxIW/eV4TD4z2cNFwaBLXTdgPB1MIJyaDA0C8cn6UhWHzGKWnBAE4Rcla5b39NgJ504CPBNuwshbvKqbKdd/fkeowvzoAHEM50rW42wGPfJAptdK8drBYctnRPhW5ypGpPUfm1Uv5OBKQ4OZbWgsSv2ENJXjEUfNtXLoT6nfBalXisFAzs1Rs6QOW8XsxZkHHnAOSfwEy8arCFMGDOtGIoSQfxZqol3VbtzJ 6L7Ql8Y2 5tBXxDXMBBQll8bjDl1U9MoWsAdtGpN/g8iBPX3QAXvUI/uK7z1/Bj/mpcwmGUZpLyGxMI8Zj3/3gpKpFUcy2bIv8iZ20DcW5k4pj1SMvq6/+PUi9H3ZjhrwXHdXe5qS5R0XF5/qQ4KJDYFnnh/7ucvbQgOhonPn6VR49Ke6jpKUjmpK7S7VYH0nz07U3lJqbUE8wrpIPsTV5H5hGMUpY9/O3ZVBGn0bKTfZkzaYWJlF8DVS8IoX8HFTK5QCeJb2yaOHC X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Jul 8, 2024 at 11:31=E2=80=AFAM James Houghton wrote: > > On Fri, Jul 5, 2024 at 11:36=E2=80=AFAM Yu Zhao wrote= : > > > > On Mon, Jun 10, 2024 at 6:22=E2=80=AFPM James Houghton wrote: > > > > > > Secondary MMUs are currently consulted for access/age information at > > > eviction time, but before then, we don't get accurate age information= . > > > That is, pages that are mostly accessed through a secondary MMU (like > > > guest memory, used by KVM) will always just proceed down to the oldes= t > > > generation, and then at eviction time, if KVM reports the page to be > > > young, the page will be activated/promoted back to the youngest > > > generation. > > > > > > The added feature bit (0x8), if disabled, will make MGLRU behave as i= f > > > there are no secondary MMUs subscribed to MMU notifiers except at > > > eviction time. > > > > > > Implement aging with the new mmu_notifier_test_clear_young_fast_only(= ) > > > notifier. For architectures that do not support this notifier, this > > > becomes a no-op. For architectures that do implement it, it should be > > > fast enough to make aging worth it. > > > > > > Suggested-by: Yu Zhao > > > Signed-off-by: James Houghton > > > --- > > > > > > Notes: > > > should_look_around() can sometimes use two notifiers now instead = of one. > > > > > > This simply comes from restricting myself from not changing > > > mmu_notifier_clear_young() to return more than just "young or not= ". > > > > > > I could change mmu_notifier_clear_young() (and > > > mmu_notifier_test_young()) to return if it was fast or not. At th= at > > > point, I could just as well combine all the notifiers into one no= tifier, > > > like what was in v2 and v3. > > > > > > Documentation/admin-guide/mm/multigen_lru.rst | 6 +- > > > include/linux/mmzone.h | 6 +- > > > mm/rmap.c | 9 +- > > > mm/vmscan.c | 185 ++++++++++++++--= -- > > > 4 files changed, 164 insertions(+), 42 deletions(-) > > > > ... > > > > > static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned= long end, > > > struct mm_walk *args) > > > { > > > @@ -3357,8 +3416,9 @@ static bool walk_pte_range(pmd_t *pmd, unsigned= long start, unsigned long end, > > > struct pglist_data *pgdat =3D lruvec_pgdat(walk->lruvec); > > > DEFINE_MAX_SEQ(walk->lruvec); > > > int old_gen, new_gen =3D lru_gen_from_seq(max_seq); > > > + struct mm_struct *mm =3D args->mm; > > > > > > - pte =3D pte_offset_map_nolock(args->mm, pmd, start & PMD_MASK= , &ptl); > > > + pte =3D pte_offset_map_nolock(mm, pmd, start & PMD_MASK, &ptl= ); > > > if (!pte) > > > return false; > > > if (!spin_trylock(ptl)) { > > > @@ -3376,11 +3436,12 @@ static bool walk_pte_range(pmd_t *pmd, unsign= ed long start, unsigned long end, > > > total++; > > > walk->mm_stats[MM_LEAF_TOTAL]++; > > > > > > - pfn =3D get_pte_pfn(ptent, args->vma, addr); > > > + pfn =3D get_pte_pfn(ptent, args->vma, addr, pgdat); > > > if (pfn =3D=3D -1) > > > continue; > > > > > > - if (!pte_young(ptent)) { > > > + if (!pte_young(ptent) && > > > + !lru_gen_notifier_test_young(mm, addr)) { > > > walk->mm_stats[MM_LEAF_OLD]++; > > > continue; > > > } > > > @@ -3389,8 +3450,9 @@ static bool walk_pte_range(pmd_t *pmd, unsigned= long start, unsigned long end, > > > if (!folio) > > > continue; > > > > > > - if (!ptep_test_and_clear_young(args->vma, addr, pte += i)) > > > - VM_WARN_ON_ONCE(true); > > > + lru_gen_notifier_clear_young(mm, addr, addr + PAGE_SI= ZE); > > > + if (pte_young(ptent)) > > > + ptep_test_and_clear_young(args->vma, addr, pt= e + i); > > > > > > young++; > > > walk->mm_stats[MM_LEAF_YOUNG]++; > > > > > > There are two ways to structure the test conditions in walk_pte_range()= : > > 1. a single pass into the MMU notifier (combine test/clear) which > > causes a cache miss from get_pfn_page() if the page is NOT young. > > 2. two passes into the MMU notifier (separate test/clear) if the page > > is young, which does NOT cause a cache miss if the page is NOT young. > > > > v2 can batch up to 64 PTEs, i.e., it only goes into the MMU notifier > > twice every 64 PTEs, and therefore the second option is a clear win. > > > > But you are doing twice per PTE. So what's the rationale behind going > > with the second option? Was the first option considered? > > Hi Yu, > > I didn't consider changing this from your v2[1]. Thanks for bringing it u= p. > > The only real change I have made is that I reordered the > (!test_spte_young() && !pte_young()) to what it is now (!pte_young() > && !lru_gen_notifier_test_young()) because pte_young() can be > evaluated much faster. > > I am happy to change the initial test_young() notifier to a > clear_young() (and drop the later clear_young(). In fact, I think I > should. Making the condition (!pte_young() && > !lru_gen_notifier_clear_young()) makes sense to me. This returns the > same result as if it were !lru_gen_notifier_test_young() instead, > there is no need for a second clear_young(), and we don't call > get_pfn_folio() on pages that are not young. We don't want to do that because we would lose the A-bit for a folio that's beyond the current reclaim scope, i.e., the cases where get_pfn_folio() returns NULL (a folio from another memcg, e.g.). > WDYT? Have I misunderstood your comment? I hope this is clear enough: @@ -3395,7 +3395,7 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, if (pfn =3D=3D -1) continue; - if (!pte_young(ptent)) { + if (!pte_young(ptent) && !mm_has_notifiers(args->mm)) { walk->mm_stats[MM_LEAF_OLD]++; continue; } @@ -3404,8 +3404,8 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, if (!folio) continue; - if (!ptep_test_and_clear_young(args->vma, addr, pte + i)) - VM_WARN_ON_ONCE(true); + if (!ptep_clear_young_notify(args->vma, addr, pte + i)) + continue; young++; walk->mm_stats[MM_LEAF_YOUNG]++; > Also, I take it your comment was not just about walk_pte_range() but > about the similar bits in lru_gen_look_around() as well, so I'll make > whatever changes we agree on there too (or maybe factor out the common > bits). > > [1]: https://lore.kernel.org/kvmarm/20230526234435.662652-11-yuzhao@googl= e.com/ > > > In addition, what about the non-lockless cases? Would this change make > > them worse by grabbing the MMU lock twice per PTE? > > That's a good point. Yes I think calling the notifier twice here would > indeed exacerbate problems with a non-lockless notifier. I think so too, but I haven't verified it. Please do?