From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81C6FC27C4F for ; Tue, 11 Jun 2024 00:22:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A48AA6B0093; Mon, 10 Jun 2024 20:22:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D32C6B0095; Mon, 10 Jun 2024 20:22:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7FAC86B0096; Mon, 10 Jun 2024 20:22:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 5EA316B0093 for ; Mon, 10 Jun 2024 20:22:06 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 16CDF12084D for ; Tue, 11 Jun 2024 00:22:06 +0000 (UTC) X-FDA: 82216705452.21.EBC3D65 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf29.hostedemail.com (Postfix) with ESMTP id 43178120012 for ; Tue, 11 Jun 2024 00:22:04 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="yZ/47khF"; spf=pass (imf29.hostedemail.com: domain of 3q5hnZgoKCLQdnbioabnihaiiafY.Wigfchor-ggepUWe.ila@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3q5hnZgoKCLQdnbioabnihaiiafY.Wigfchor-ggepUWe.ila@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718065324; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JyxqE8JQbjAibPP1L18H/WB5ae5upDraVd7kSoejAF4=; b=XUQ3x9c3y+a/A+tZ82w0Ezy9GS3dmpQAstO2aCghfTeER9DSqr+l5efHd7OShEo/ePHvcU 8cfdxoT5gXxypO+N/fs33AJ1LKUDjQSGStoCjoC4pJ3P2mXzQPUWdBMBLtYuol7AKdQwS8 0xiqe3Klr0ditqRzkD1/bCb572lRe+E= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718065324; a=rsa-sha256; cv=none; b=6KQC90FzpVr57hqIBLtc9aVy6NKX7tckry6b6FduW5rvpXBgxd1lu/g6OwDJ590V+ax2Qg E7nPPazLUOU6tbkigEiXVSITXDzY9jAQxndNfr6gKEQI8zgycKzCJpudET3AVcRVfdF00q T/Fm/0Zpk9FsJ7dl9nTh/BAAwIiGu9g= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="yZ/47khF"; spf=pass (imf29.hostedemail.com: domain of 3q5hnZgoKCLQdnbioabnihaiiafY.Wigfchor-ggepUWe.ila@flex--jthoughton.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3q5hnZgoKCLQdnbioabnihaiiafY.Wigfchor-ggepUWe.ila@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-627956be166so99972647b3.0 for ; Mon, 10 Jun 2024 17:22:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1718065323; x=1718670123; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=JyxqE8JQbjAibPP1L18H/WB5ae5upDraVd7kSoejAF4=; b=yZ/47khFrCJDbfaloZDz0eVAlCF6mkMZM2Z3wTx+Tdp7hlDTrIzQCm8wG7WDohpzcA GCxEgapAF02JxggDqJTq1wQ5Ah/jGF4bJI18Kda5Rh2i5QCF+NgJvSRGOp/98sng0obG JLlkYwn42yvH3Ec6XmImrvVjJqL/0UIZ7hWahh1NRi8+dHEwf4crqzMlOmCVzRX0oxeA odu0PzV1953Rg0w9O6r5ppgNYjSnyIFtSLURuwo7IQTnGJXIWsqWVjYfXGBywFfqSWdr rUoKeJB/LKtsv1ZoRI0z5uJo+XDLecQhF50Zsesys9omaVf3K93dHTklkUCkg9Y48JkV kN3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718065323; x=1718670123; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=JyxqE8JQbjAibPP1L18H/WB5ae5upDraVd7kSoejAF4=; b=ufbzh9RzNeQclNDWOyEm5XW2+te14mmytLjvi4DJ/li/KEqHHR4cd1zS2+9Gq9pUwQ 6h/+l9p07b7bYB/e93nwkhhf7y7QDyU3wmbffB9QqJHJaUKL9cdZV7/f7EkVGuiAVdZH SpbJmyWPPHJNkXAqs6hyZEVgsqm3UTZT4SldBFCSB12y4NHBzXhikv9VyS1vKo/yrjUF mKxJee35eRiTFkDWIgmVVXtE3YF2G8ccxUohRKhQsgL0bAxON2IKQcSYjb+1mOuPlDBk 6DhnZVnz2ntFMO5WeiMhx55VzhpBDhsgfcT6Lme15kVunu78w5/Om7FJvFWPIQo5TEIR QoVg== X-Forwarded-Encrypted: i=1; AJvYcCVnVbT86HoyXzBiqNXHFLwuwEpl44S9++ed447Rz0KSLur5Qs+um5U/ggQ5tBBjJS4wYZT6RN1xSJTkLslZ10rMmts= X-Gm-Message-State: AOJu0YxO3RnQkE5enKhhFRu28YoiDPrpipvaC23s3c/zgRjuKPzyGBka Opz4TOnnXwCxURnsuyrv5SZ2/MUbxhsZjvOWHpiUAIwl1bd584l3itLknuUP9Agei3N/Ht/23Q/ /vVbQIMPiXxsu6wAYLw== X-Google-Smtp-Source: AGHT+IE2AKcYsstInOqgudHKTQp6Jdm0LGHMNfnLB4IVzasoq6oW0Y69RVWgTVatpoFZnw0nHcf4J+vZvg80MMrE X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:690c:6101:b0:62a:2a39:ccd9 with SMTP id 00721157ae682-62cd5663caemr27714337b3.6.1718065323356; Mon, 10 Jun 2024 17:22:03 -0700 (PDT) Date: Tue, 11 Jun 2024 00:21:41 +0000 In-Reply-To: <20240611002145.2078921-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240611002145.2078921-1-jthoughton@google.com> X-Mailer: git-send-email 2.45.2.505.gda0bf45e8d-goog Message-ID: <20240611002145.2078921-6-jthoughton@google.com> Subject: [PATCH v5 5/9] KVM: Add kvm_fast_age_gfn and kvm_fast_test_age_gfn From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Ankit Agrawal , Axel Rasmussen , Catalin Marinas , David Matlack , David Rientjes , James Houghton , James Morse , Jonathan Corbet , Marc Zyngier , Oliver Upton , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Suzuki K Poulose , Wei Xu , Will Deacon , Yu Zhao , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 43178120012 X-Rspam-User: X-Rspamd-Server: rspam08 X-Stat-Signature: o5epmmx6uab7e3qmdw7f9ts6kpep9n86 X-HE-Tag: 1718065324-26316 X-HE-Meta: U2FsdGVkX18U1L3VueVRz74oquFnNmXOF55eVP+25QTKYNw0mmWnHAnEm4GD9XYoKXydNmesdq3n5a1Cq3m+BLbKe0QRz25fmcXZDgt/7x0XPTif1ZPRH7spqguot5yj0MHyR47zWy1k2zQHoGyE/mwq1/s8fZoo3ZnQxpmDE4YqorIFel2yhaPhRnMsRBcoRDCVI+5gN4J5dhKgKPZWC9L1g+3xJe9/9YcW86+DXG6ShnN9wECry9Shjw0Oa9dKqvp1Sq7G5iqYA7CmzM0HalOF40RpROosPYsV9TBBig/QZUIJeIRcj0n60HnIbp565ShTIu0JXY8w5eXT+1BhsqD3F70x3JHS2Q93h55hzatz8NfgnnKG1sEbrS2DkRtjKSWrMFbnceGtOofrY4WrfZcAGAtk6yNf4Zcx8ysZKD+Q8AMZyB9hPZggAlaNFL2aUkjXxPm9a+B88W1Mr2xQ4qU+o/uKqGi3dkZkE5Nc7XWc66k4jmC+IUW7b38KWtalxz1Ac7ybKJdNsmfPnvdbSeLRzgMKNaN6IK+nmC+unQtZbwpefISoS4NZpSLX3XKQqDpADyawwtNPnZYQvQkcBowmFhhcatrsl/3xvQfhCpGH1lvCWn5gXHz/w7cae0Y9HlzVVFaU2gBoQP5smKipGNP3Aq4v1qlIv7UlYRgtj12AQCKsYT4TtR3wooG0xQQD9QgyB3E78z7nrQ/f7lCD2JWCRTYJMaCx199wL01tj35geY72m2XbSYSBClfljrzle8l/l4jt+xnDKD9aLjPhlvmqWppbiayF7LLKyvjV5c4tLsca605FAnsrDLLvg/p12GdH9rbBKFX1J58RcCNETe03RXdx1zMFanHRmU2MehLP2RC17xSS7txkAuJdfjxLPPEGEtpalxk8jlNtwsN/XgZbFiVkh/A0ajskjhi36Qjce1Hklrds5R1gpQWfvbuA/uCjw9fte9tPW3Em7NC 8nil3sw7 FidUMitZ6QI4oNMb9ep/r8Omwcu+V73O6RsFBlQXHUwUDxArisw2oXA+RfS0rZGda8w+0dLB19WffqHyD6RTKGCkKBfsFswg6waALj/vcIvYFlY+jI0jKX68rg+e3mqOdIkzWT0e18xMen2SMSfhhuqO+IbTRrgtn9qTRlyYnwSyZpW88N/X/I6gL2JrpVR2njGUxUcz1FpMMno2/3lo9W+P4+wXZJf6xLJ6vPeHPLdLfleBrthJmDhh0e5YaO/s/dOGbNGKvIhWiBBi2ldyAC3+VhzV6idLdL55mrHDHzZVUZyZqgR3V2s0xC92m8bs4pIqd8lfUD7Hx2DPVZ9HyjIM8FU5QYfJd11DIQWqnm4hV1hxYLDmzVR/KkuwOtcZuMVgvxvk7dEto+9BZChTMIfIl+D+fnPZ9iyJWam8n5N/DJjaSsN5CfX7QBzGXdEaRL6ROyPd45khola1C0gUeifoFlKaEx701tnyks1hyaNua4hevWFTzoQ+nT9A099eQ1DSMQpuIlDhRXJpsDpjQ5L9td1WgtV0XQgJVfAwqMZ1CyceEqpjLWD7jknhS+AKKi9C7dt4G1DJj2+82v6t/OZQFmd3SgEzAGObm X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Provide the basics for allowing architectures to implement mmu_notifier_test_clear_young_fast_only(). Add CONFIG_HAVE_KVM_YOUNG_FAST_ONLY_NOTIFIER that architectures will set if they implement the fast-only notifier. kvm_fast_age_gfn and kvm_fast_test_age_gfn both need to support returning a tri-state state of: 1. fast && young, 2. fast && !young, 3. !fast This could be done by making gfn_handler_t return int, but that would mean a lot of churn. Instead, include a new kvm_mmu_notifier_arg 'bool *failed' for kvm_fast_{test,}_age_gfn to optionally use. Signed-off-by: James Houghton --- include/linux/kvm_host.h | 7 ++++++ include/trace/events/kvm.h | 22 ++++++++++++++++++ virt/kvm/Kconfig | 4 ++++ virt/kvm/kvm_main.c | 47 ++++++++++++++++++++++++++++++-------- 4 files changed, 71 insertions(+), 9 deletions(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 4d7c3e8632e6..e4efeba51222 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -258,6 +258,9 @@ int kvm_async_pf_wakeup_all(struct kvm_vcpu *vcpu); #ifdef CONFIG_KVM_GENERIC_MMU_NOTIFIER union kvm_mmu_notifier_arg { unsigned long attributes; +#ifdef CONFIG_HAVE_KVM_YOUNG_FAST_ONLY_NOTIFIER + bool *failed; +#endif }; struct kvm_gfn_range { @@ -271,7 +274,11 @@ struct kvm_gfn_range { bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range); bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); +#ifdef CONFIG_HAVE_KVM_YOUNG_FAST_ONLY_NOTIFIER +bool kvm_fast_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); +bool kvm_fast_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); #endif +#endif /* CONFIG_KVM_GENERIC_MMU_NOTIFIER */ enum { OUTSIDE_GUEST_MODE, diff --git a/include/trace/events/kvm.h b/include/trace/events/kvm.h index 74e40d5d4af4..7ba6c35c2426 100644 --- a/include/trace/events/kvm.h +++ b/include/trace/events/kvm.h @@ -489,6 +489,28 @@ TRACE_EVENT(kvm_test_age_hva, TP_printk("mmu notifier test age hva: %#016lx", __entry->hva) ); +TRACE_EVENT(kvm_fast_test_age_hva, + TP_PROTO(unsigned long start, unsigned long end, bool clear), + TP_ARGS(start, end, clear), + + TP_STRUCT__entry( + __field( unsigned long, start ) + __field( unsigned long, end ) + __field( bool, clear ) + ), + + TP_fast_assign( + __entry->start = start; + __entry->end = end; + __entry->clear = clear; + ), + + TP_printk("mmu notifier fast test age: hva: %#016lx -- %#016lx " + "clear: %d", + __entry->start, __entry->end, + __entry->clear) +); + #endif /* _TRACE_KVM_MAIN_H */ /* This part must be outside protection */ diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig index 0404857c1702..77ac680af60c 100644 --- a/virt/kvm/Kconfig +++ b/virt/kvm/Kconfig @@ -100,6 +100,10 @@ config KVM_GENERIC_MMU_NOTIFIER config KVM_MMU_NOTIFIER_YOUNG_LOCKLESS bool +config HAVE_KVM_YOUNG_FAST_ONLY_NOTIFIER + select KVM_GENERIC_MMU_NOTIFIER + bool + config KVM_GENERIC_MEMORY_ATTRIBUTES depends on KVM_GENERIC_MMU_NOTIFIER bool diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index d8fa0d617f12..aa930a8b903f 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -699,7 +699,8 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn, static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn, unsigned long start, unsigned long end, - gfn_handler_t handler) + gfn_handler_t handler, + bool *failed) { struct kvm *kvm = mmu_notifier_to_kvm(mn); const struct kvm_mmu_notifier_range range = { @@ -711,6 +712,7 @@ static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn .may_block = false, .lockless = IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), + .arg.failed = failed, }; return __kvm_handle_hva_range(kvm, &range).ret; @@ -901,7 +903,7 @@ static int kvm_mmu_notifier_clear_young(struct mmu_notifier *mn, * cadence. If we find this inaccurate, we might come up with a * more sophisticated heuristic later. */ - return kvm_handle_hva_range_no_flush(mn, start, end, kvm_age_gfn); + return kvm_handle_hva_range_no_flush(mn, start, end, kvm_age_gfn, NULL); } static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, @@ -911,9 +913,32 @@ static int kvm_mmu_notifier_test_young(struct mmu_notifier *mn, trace_kvm_test_age_hva(address); return kvm_handle_hva_range_no_flush(mn, address, address + 1, - kvm_test_age_gfn); + kvm_test_age_gfn, NULL); } +#ifdef CONFIG_HAVE_KVM_YOUNG_FAST_ONLY_NOTIFIER +static int kvm_mmu_notifier_test_clear_young_fast_only(struct mmu_notifier *mn, + struct mm_struct *mm, + unsigned long start, + unsigned long end, + bool clear) +{ + gfn_handler_t handler; + bool failed = false, young; + + trace_kvm_fast_test_age_hva(start, end, clear); + + handler = clear ? kvm_fast_age_gfn : kvm_fast_test_age_gfn; + + young = kvm_handle_hva_range_no_flush(mn, start, end, handler, &failed); + + if (failed) + return MMU_NOTIFIER_FAST_FAILED; + + return young ? MMU_NOTIFIER_FAST_YOUNG : 0; +} +#endif + static void kvm_mmu_notifier_release(struct mmu_notifier *mn, struct mm_struct *mm) { @@ -926,12 +951,16 @@ static void kvm_mmu_notifier_release(struct mmu_notifier *mn, } static const struct mmu_notifier_ops kvm_mmu_notifier_ops = { - .invalidate_range_start = kvm_mmu_notifier_invalidate_range_start, - .invalidate_range_end = kvm_mmu_notifier_invalidate_range_end, - .clear_flush_young = kvm_mmu_notifier_clear_flush_young, - .clear_young = kvm_mmu_notifier_clear_young, - .test_young = kvm_mmu_notifier_test_young, - .release = kvm_mmu_notifier_release, + .invalidate_range_start = kvm_mmu_notifier_invalidate_range_start, + .invalidate_range_end = kvm_mmu_notifier_invalidate_range_end, + .clear_flush_young = kvm_mmu_notifier_clear_flush_young, + .clear_young = kvm_mmu_notifier_clear_young, + .test_young = kvm_mmu_notifier_test_young, +#ifdef CONFIG_HAVE_KVM_YOUNG_FAST_ONLY_NOTIFIER + .test_clear_young_fast_only = + kvm_mmu_notifier_test_clear_young_fast_only, +#endif + .release = kvm_mmu_notifier_release, }; static int kvm_init_mmu_notifier(struct kvm *kvm) -- 2.45.2.505.gda0bf45e8d-goog