From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2C99C27C4F for ; Tue, 11 Jun 2024 00:22:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DCC186B0092; Mon, 10 Jun 2024 20:22:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CDC876B0093; Mon, 10 Jun 2024 20:22:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B07456B0095; Mon, 10 Jun 2024 20:22:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 7C17C6B0092 for ; Mon, 10 Jun 2024 20:22:05 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 040B91409FA for ; Tue, 11 Jun 2024 00:22:04 +0000 (UTC) X-FDA: 82216705410.25.D8D62CC Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf26.hostedemail.com (Postfix) with ESMTP id 397AF140015 for ; Tue, 11 Jun 2024 00:22:02 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=tMTj2ewB; spf=pass (imf26.hostedemail.com: domain of 3qphnZgoKCLMcmahnZamhgZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3qphnZgoKCLMcmahnZamhgZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718065323; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6j3rWVw+E6yiyAll83pvyiHc59e6Q6WaEN3pW8ayg5o=; b=k1Qk1bMtzENt3xhaS5tlKNbf2Xn1MdJe70uZCp+0FdZ9rd1MRcICu2z/Htgej9aoY2iVJ9 xvYDEY9s0nHkDvzXwidAl9bgd3vsEvfhxGbi8GyoU8yvMWCDTT79ULwmvStJoN/oFvys+y 2aow2IofaJ31DwqchmAT9vHq9CVdaiM= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=tMTj2ewB; spf=pass (imf26.hostedemail.com: domain of 3qphnZgoKCLMcmahnZamhgZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3qphnZgoKCLMcmahnZamhgZhhZeX.Vhfebgnq-ffdoTVd.hkZ@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718065323; a=rsa-sha256; cv=none; b=dUxOri3Y2Hsp8xedXClZrmzgoWJW+LRpVNzD/kLQx4eRCKFzIuDSW6+G0oJujq+RLyoMcx Fma4VW7Q2/8saZuKKVxCzV1okJpxECpFpel3LdyrE3085wL07b1VTzhUlIEd8gXIYx1jMk jPOSQvojc26BK2+2ZftS4vWJbzd3Uzs= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-62cf01a20d5so9727307b3.1 for ; Mon, 10 Jun 2024 17:22:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1718065322; x=1718670122; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=6j3rWVw+E6yiyAll83pvyiHc59e6Q6WaEN3pW8ayg5o=; b=tMTj2ewBkPXjv8pwSq+AL3pF6VDGxDlekLEf6uAmPMFFyRP6FpY1uSbxTyvPwX/+Zp w9JEgto/SJ6tRr2uSgaASouw90oY1sHnQOmOd34V+qMHbnwOq7WbjH4Omjs6n8nFqWHG biv5cSMWnipEdrzgTubnztn3NDi/8cnDpH69KpeUt7OwgyBxnSsQFo1HNLyRAPDh7cxy +OKTbFzr/2BsOxpOhbiMDxsVSRWrCgRCVXJ2UrXiOtsiI6JhhHZngs3A1XWA1/NdIPTj aG1bGLTs76RVcFOig4YwkDWFb7hax/EOrr557v0qrmi6MB2uOIsUG+yyuJxDubVLNrRs Qn2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718065322; x=1718670122; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6j3rWVw+E6yiyAll83pvyiHc59e6Q6WaEN3pW8ayg5o=; b=KFPJbWMPJMYx5VvPsRRIFJhrWNAZb/MI7B7AOegHq3Rj6PVbIj4o6Yv9UN4kES37Q1 WKLo70Pu4YsmOJgJcuRA2XALk6wxEdt0fXVqxYf98t9OuM8A4RPNh65sSegcTjzmwlXi PukvPuuMPraAoPuRytMIcXgXU2v4rUEiuy6KOK0tvZg78zLYLQMalLqnYUqimn2aqaGH W4kWBAzfB4Cnbmevvt8dFfeh42w06aWeEaKRGRl8HbsCRRMoFo9stud/zR1DY2DDK7ds tgqGyzmuA2wpeTVtLEtY8OimebMZLETygxW39gn9lcvtEeYBB59OnbP6E7Gf38x4Z0DF mj+g== X-Forwarded-Encrypted: i=1; AJvYcCWTjgnHnCn15zDWs7/n1EgDgHZcaDqvcicU1qr4jdXbI6KSVcu11FKZeMP5jIdFt8gHYN8wLgkCKDLBfHoRbmv8das= X-Gm-Message-State: AOJu0YzGkNzKcwlkCv9YeNC8DD+VahsW3CEvfYhQ7Qhnv2W9OVgHKRth 2XTzWeMelcem0uDVRJ5+Ey+OKHW9PQq+jBjzkpz4NayVwsUqH2YJz2WBVSahb6sDSL0PEJTOpP+ GCooMOGiBfwRckBCFhA== X-Google-Smtp-Source: AGHT+IEvrN0k+l/fqR0GYBVcFmKI7zO7AJD796Zxk8tVrT0hIaj/Sm704UQ/Nq6b1QsHmo+//ovsajrvliWZ5C+2 X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:690c:4485:b0:62d:fbf:920a with SMTP id 00721157ae682-62d0fbf9666mr11256657b3.10.1718065322252; Mon, 10 Jun 2024 17:22:02 -0700 (PDT) Date: Tue, 11 Jun 2024 00:21:40 +0000 In-Reply-To: <20240611002145.2078921-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240611002145.2078921-1-jthoughton@google.com> X-Mailer: git-send-email 2.45.2.505.gda0bf45e8d-goog Message-ID: <20240611002145.2078921-5-jthoughton@google.com> Subject: [PATCH v5 4/9] mm: Add test_clear_young_fast_only MMU notifier From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Ankit Agrawal , Axel Rasmussen , Catalin Marinas , David Matlack , David Rientjes , James Houghton , James Morse , Jonathan Corbet , Marc Zyngier , Oliver Upton , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Suzuki K Poulose , Wei Xu , Will Deacon , Yu Zhao , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 397AF140015 X-Rspam-User: X-Rspamd-Server: rspam09 X-Stat-Signature: kwk5nesy4h3twtdyc5dggpbpfz43a8yo X-HE-Tag: 1718065322-500041 X-HE-Meta: U2FsdGVkX1+S9o5C1WXCKsNvdT/s3e/JcQJXM31AqNxvOIpgZOSCor7WeEevu8AirJfXXc4YgIsMIJjFVLRTVmhFoAh4KJpae9EtX1Mp5EgMDHZSWKNiXymvCMltEtsmL2XjqLbxytXRgJ9IgNTMNwdiFu/xFxVE4+LntCZ+GkFhubuPKSzdGY0bd3ecOX9iXCgxuxw66CEQo9UxchNybif0K3HyKPewqK6j5XVEAFDs2CvpIljKu52svKZlOyNzWhkC1rfY6wnX15uOYZ6gbmYHm8PlF03am9CkhAMpvtPkIP6ZlD67HieAEXRa2eIS58DAgV8DE9yD1361OAWMc8tftryqGnI1iev6gDnbidKGx3uoiWxwVHPhV4AbiGl/GVBiNn0Oo6XcZzDxe7N9NEi/hnwGmKLoc6My/sSDZsO9dy7nqWxhNHmkKVCktGGta0yLTEenDyxPR08opfI2Bi4CVQ8j835nd+4hyzURZv30qVF02vlQptmXoR0j/PAT2D/q5WeWmBunxay3tTM/ocXtsWvvXCmIOl+gwCaimj+rvmH6b0jLcnSlMn/U77JrTK2qcR6ZqTd9W1Yv8rIGzGxETgBHb9DkrcCZM1iYjFoC5t85XnOf4KLnja2ZY9J4olwWmU1MzNx6LC8sgWLqpB348EaxodFtBVC78a54T7JXdCUnumgNP+DLdSF9qVaogja8VZ64fvb2Jt1s6HRuHDsuERGV+i9SAQU5RUipHRm5MSOc3acTvk487vvI0x0dgikrNbY8yVDRuUnUr7oBduRXU7/wjsT33F/ifnsqycNvmJQdUlVxwo7RevImUloh6LeLTBYvUbk0vtIFXuORdASYcJbgG0hU6PwZqoPTC6TRgbJf8Zw/poytyfIzzS2N6EFdjMwuAQzFbxTkIXMIb1fFhE0pdpOx2HhEvwowKI3VGC4R5zxvU9eFBMtA9maUjzQATwexvFw5bz3Kt54 20x6FGPe z5JWLauzuRA2p6A5BfnWah+/ONYYM4ajKobKEYhGqAquWj+ZKvf83FBttKgrpuTR279LS1E+1ZPA7m5OVLTKjuIYHYQS2vaJ2x2a6yLsmd+zpfh2vKK17evuX6ujvnbc25xS3bqkPf7k/I8cAopiedfl9qS6vyA3clRP9mBf95KlWE1aGRft/BG2F8Yvv+Im2K5vLVcb2HjT2PvMF9n/dYiR/JJuNttPEz4D53Hyr3ATzbp8pa1FHapszsGEmDZ2kVmrvXqnfKiT31eYUlxCZLpXegHyxoOy8mWEC7K+sRkQqQ+3Hf2vTah5XI0tGrSGFIMfDosmDq2Cywk5VJlx/4vpOyVBZtLHX+AJD2Oc6fEgW0/H7o+WcZzmS+YsVR4Qy3Mqcm+ITo1WLWbdUo1gevygF3TRJlrfaVnVpOOfNMhdKoyQa43PWf7QYzpzy5Guvwjqe7KIVWvyFd3eAIIqxcMymwaAW+WjPkZESTlZMeXvED2UZpu0l/17Avg3202jAqRvSEc2CpQlF0f4mv17deoA2JhiryrWOBx6ZzeeOysxZcXfcapG/fgoF94mcGlyj3FnVyflYdy2kLmBBtZ+Y3f6Z+4zE5k1oBMTd X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This new notifier is for multi-gen LRU specifically, as it wants to be able to get and clear age information from secondary MMUs only if it can be done "fast". By having this notifier specifically created for MGLRU, what "fast" means comes down to what is "fast" enough to improve MGLRU's ability to reclaim most of the time. Signed-off-by: James Houghton --- include/linux/mmu_notifier.h | 50 ++++++++++++++++++++++++++++++++++++ mm/mmu_notifier.c | 26 +++++++++++++++++++ 2 files changed, 76 insertions(+) diff --git a/include/linux/mmu_notifier.h b/include/linux/mmu_notifier.h index d39ebb10caeb..2655d841a409 100644 --- a/include/linux/mmu_notifier.h +++ b/include/linux/mmu_notifier.h @@ -61,6 +61,15 @@ enum mmu_notifier_event { #define MMU_NOTIFIER_RANGE_BLOCKABLE (1 << 0) +/* + * Bits in the return value for test_clear_young_fast_only. + * + * MMU_NOTIFIER_FAST_YOUNG: notifier succeeded, secondary MMU reports young. + * MMU_NOTIFIER_FAST_FAILED: notifier failed. + */ +#define MMU_NOTIFIER_FAST_YOUNG (1 << 0) +#define MMU_NOTIFIER_FAST_FAILED (1 << 1) + struct mmu_notifier_ops { /* * Called either by mmu_notifier_unregister or when the mm is @@ -122,6 +131,24 @@ struct mmu_notifier_ops { struct mm_struct *mm, unsigned long address); + /* + * test_clear_young_fast_only is called to check (and optionally clear) + * the young/accessed bitflag in the secondary pte such that the + * secondary MMU must implement it in a way that will not significantly + * disrupt other MMU operations. In other words, speed is more + * important than accuracy. + * + * Returns MMU_NOTIFIER_FAST_YOUNG if the secondary pte(s) were young. + * Returns MMU_NOTIFIER_FAST_FAILED if the secondary MMU could not do + * an accurate fast-only test and/or clear of the young/accessed + * flag. + */ + int (*test_clear_young_fast_only)(struct mmu_notifier *subscription, + struct mm_struct *mm, + unsigned long start, + unsigned long end, + bool clear); + /* * invalidate_range_start() and invalidate_range_end() must be * paired and are called only when the mmap_lock and/or the @@ -383,6 +410,10 @@ extern int __mmu_notifier_clear_young(struct mm_struct *mm, unsigned long end); extern int __mmu_notifier_test_young(struct mm_struct *mm, unsigned long address); +extern int __mmu_notifier_test_clear_young_fast_only(struct mm_struct *mm, + unsigned long start, + unsigned long end, + bool clear); extern int __mmu_notifier_invalidate_range_start(struct mmu_notifier_range *r); extern void __mmu_notifier_invalidate_range_end(struct mmu_notifier_range *r); extern void __mmu_notifier_arch_invalidate_secondary_tlbs(struct mm_struct *mm, @@ -428,6 +459,17 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm, return 0; } +static inline int mmu_notifier_test_clear_young_fast_only(struct mm_struct *mm, + unsigned long start, + unsigned long end, + bool clear) +{ + if (mm_has_notifiers(mm)) + return __mmu_notifier_test_clear_young_fast_only(mm, start, end, + clear); + return 0; +} + static inline void mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range) { @@ -612,6 +654,14 @@ static inline int mmu_notifier_test_young(struct mm_struct *mm, return 0; } +static inline int mmu_notifier_test_clear_young_fast_only(struct mm_struct *mm, + unsigned long start, + unsigned long end, + bool clear) +{ + return 0; +} + static inline void mmu_notifier_invalidate_range_start(struct mmu_notifier_range *range) { diff --git a/mm/mmu_notifier.c b/mm/mmu_notifier.c index 8982e6139d07..7b77ad6cf833 100644 --- a/mm/mmu_notifier.c +++ b/mm/mmu_notifier.c @@ -424,6 +424,32 @@ int __mmu_notifier_test_young(struct mm_struct *mm, return young; } +int __mmu_notifier_test_clear_young_fast_only(struct mm_struct *mm, + unsigned long start, + unsigned long end, + bool clear) +{ + struct mmu_notifier *subscription; + int ret = 0, id; + + id = srcu_read_lock(&srcu); + hlist_for_each_entry_rcu(subscription, + &mm->notifier_subscriptions->list, hlist, + srcu_read_lock_held(&srcu)) { + if (subscription->ops->test_clear_young_fast_only) { + ret = subscription->ops->test_clear_young_fast_only( + subscription, mm, start, end, clear); + if (ret & MMU_NOTIFIER_FAST_FAILED) + break; + if (!clear && (ret & MMU_NOTIFIER_FAST_YOUNG)) + break; + } + } + srcu_read_unlock(&srcu, id); + + return ret; +} + static int mn_itree_invalidate(struct mmu_notifier_subscriptions *subscriptions, const struct mmu_notifier_range *range) { -- 2.45.2.505.gda0bf45e8d-goog