From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBF30EE0212 for ; Thu, 14 Sep 2023 01:55:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7C57C6B027A; Wed, 13 Sep 2023 21:55:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 727136B027B; Wed, 13 Sep 2023 21:55:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C8B96B027F; Wed, 13 Sep 2023 21:55:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4E4C96B027A for ; Wed, 13 Sep 2023 21:55:43 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 1B55240E78 for ; Thu, 14 Sep 2023 01:55:43 +0000 (UTC) X-FDA: 81233536566.28.BF1E48F Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) by imf03.hostedemail.com (Postfix) with ESMTP id 42D4620008 for ; Thu, 14 Sep 2023 01:55:41 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=HXIy2lnu; spf=pass (imf03.hostedemail.com: domain of 3G2gCZQYKCCQSEANJCGOOGLE.COMLINUX-MMKVACK.ORG@flex--seanjc.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3G2gCZQYKCCQSEANJCGOOGLE.COMLINUX-MMKVACK.ORG@flex--seanjc.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1694656541; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/PWByveWahNDDpwQn3dTY8oCHEhRVqmq8s8F8KhgOhw=; b=CbYZKJQWwzG4ETHuBbIGHPdi/mlkaIJZfEwgDLfkL+v3bX/22Fc/VMdlTmL+BpHqdNHyMo CQERlcSEbMt8kIVI6TVB7HXiB89QhmPbtiuZMF3ZSaPlqT+XJDziNKkrqqFU6O1cf3cKTx y+/hIcQW42GBkgUSp4WDBg6HYpIVUDw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1694656541; a=rsa-sha256; cv=none; b=StJI37m7Klkzs4FEYIgUpgqdmNd7+yt+6UgpWruq3vBx7/OStt62E8E0rLcLQpCZExs6QQ xIcD/FJfyqB2eg6GCtIh0XQN3xkm0k1+v6tuEqi6WEwVXNKYtWu32t1noz0NXwnvecfMmB um0qylEVzSAZAsGUzRTFzkw6f8VshWM= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=HXIy2lnu; spf=pass (imf03.hostedemail.com: domain of 3G2gCZQYKCCQSEANJCGOOGLE.COMLINUX-MMKVACK.ORG@flex--seanjc.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3G2gCZQYKCCQSEANJCGOOGLE.COMLINUX-MMKVACK.ORG@flex--seanjc.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-57787e29037so342127a12.2 for ; Wed, 13 Sep 2023 18:55:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1694656540; x=1695261340; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:from:to:cc:subject:date:message-id:reply-to; bh=/PWByveWahNDDpwQn3dTY8oCHEhRVqmq8s8F8KhgOhw=; b=HXIy2lnut6XyQkW5Xo2VBegD9RFsEeJ0REXF8hUqooDK90DM8GJA3y8b+ZYzy2x3x4 CIQoWuUYZxIZltSaX+g2tO0Qsygr/X0uKhLxYjeLYWABye4uw3ol/Q5BwdY+pYGCyNj/ ESBcB99Uto0G6RT8rO95ZsTDzNm4cAtBznzuOe8Gu+l38+WiualtZWx1CqvCFedAVZph 0pfnEkfRVzyHnSdBbT9Aje4uXQNjQi0nQJhewMyLSwIQPLz5Nty+739cDeoCVZTvg48j lFFjET0am6z3JNYMBYDRmySpQRoouvOKtqg62gwUmyFn4mknVO0665ZXUa2a0vfyQ3Wl VGcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1694656540; x=1695261340; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:reply-to:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=/PWByveWahNDDpwQn3dTY8oCHEhRVqmq8s8F8KhgOhw=; b=B6pFbFKH+PV+7Uqa9rYKe8wQ5bNo+aE4vJLJzGtbq40JzDERSGj8WNROjZ7ySthx5j B2C4yodO3WnHWuJlPvkSWjTGUUG3CEMgoAQg4A2v1ubS2iibZnd4msa9ft7FH1zNR+p5 OzIEbYLW96dKV35jFi8HbTNeLCxmYXAwAk8sYlWjTjWTeN1ckjh1oLhJMsScTcn3A1G+ OsKMY80W/7ftYJsis5cmUJkHmeAlE/Pg0FpZ/1fNNhPYOW366VswAHyOat5gU83sTaxX QwpiWlHFx5doNSDZwgbmxvYGVoKlkYZN7kVeXJA2stHj+PQZGwubpYldcR9onqfT6zhc gNsQ== X-Gm-Message-State: AOJu0YxxIYy68sa+EE68npPuHsIrN3Drh5U630iUZGq291TXlH5KFpnn 8xUcy2dOXScPa+/0bp7oJDTNN39Bpi8= X-Google-Smtp-Source: AGHT+IFH7Y8a1vBNnSACzhOpKycbEt9pew6IzwhiL/gw/jXJMO2Ad5b3/QDP/rYw9JwfwFTkI8vz1sHeU4s= X-Received: from zagreus.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:5c37]) (user=seanjc job=sendgmr) by 2002:a17:903:230b:b0:1bf:cc5:7b53 with SMTP id d11-20020a170903230b00b001bf0cc57b53mr198908plh.1.1694656539979; Wed, 13 Sep 2023 18:55:39 -0700 (PDT) Reply-To: Sean Christopherson Date: Wed, 13 Sep 2023 18:55:00 -0700 In-Reply-To: <20230914015531.1419405-1-seanjc@google.com> Mime-Version: 1.0 References: <20230914015531.1419405-1-seanjc@google.com> X-Mailer: git-send-email 2.42.0.283.g2d96d420d3-goog Message-ID: <20230914015531.1419405-3-seanjc@google.com> Subject: [RFC PATCH v12 02/33] KVM: Use gfn instead of hva for mmu_notifier_retry From: Sean Christopherson To: Paolo Bonzini , Marc Zyngier , Oliver Upton , Huacai Chen , Michael Ellerman , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Sean Christopherson , "Matthew Wilcox (Oracle)" , Andrew Morton , Paul Moore , James Morris , "Serge E. Hallyn" Cc: kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-security-module@vger.kernel.org, linux-kernel@vger.kernel.org, Chao Peng , Fuad Tabba , Jarkko Sakkinen , Anish Moorthy , Yu Zhang , Isaku Yamahata , Xu Yilun , Vlastimil Babka , Vishal Annapurve , Ackerley Tng , Maciej Szmigiero , David Hildenbrand , Quentin Perret , Michael Roth , Wang , Liam Merwick , Isaku Yamahata , "Kirill A . Shutemov" Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 42D4620008 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: fgb1paom711qkrczq4c8f4epuep6395n X-HE-Tag: 1694656541-542858 X-HE-Meta: U2FsdGVkX1+UzyW78UPflj2wgYmCXhFD0B9H8WGSFiOFY09MQeUsGXEi0MihDYwvbBO0Kve5XsWrn7XkdiqW+jl36QTZWWjHrUmUAB+FGLwKy9eu59sOBtkE490xDXBmQRpmw0taNY/tQ6y2G0CBDOIZ/6YwpucF879FZc3PiYBLymZthFPi8FRV86l4UHXn62KAU1du9ks8O7+VZp2zSt3K54pAPiNQ8ymP5+4oSt6lFvzzTIe7LEaDVHCgrl2saoBnG/KkRyCgwwMokPDcpybzVBTB+H5y2Z6YvMP7t5hob7511Qb09bm1jizYXrRXn6Xsai0UArbLbKFAEnZJKdyTXmWWmz8lQBx0THoa3RkZosXRoeEcSJgR2wUIvqBLzHcfLCWyiZt6EMkCVMaBUt8xWomqSn6JTDUIRTUH7gSQCT6PoJm9qJcFMtg62SvgQoxGrd1WqG4OhLM8cupGBuiG3ZOUo0UsSaBt9pU+NKuvVG6IYr7I3BU3bHuaf6X1OlidgW7os0XVpv/Fxx5Dbnh0ZhdWVoZ6sSqt8OESm1DS/doLamH+ZIbF33pos4jx3j8LmGUOtCeorGglNiwUzl2abGw/EdBoeOW2bXBMjsr28JEvl6CeKJRsKPHN5SRQMxtUIK0XQuanCGN0TQpwZa/PHYxw240Oz2QmjEsaHZ4ZwU18ns5wK+O4vvuVuCKpbd/ZNDDAljzxYZcamwu4rfwTlenEgY0yZl6uQ8//VYEPwEwGoWTNGV1qaQJhox/xOg+Rk+8IDW+WvS0GJiXSH6MEzTvJ9BND9F4Zx5IeFJOEKkr8gQgBsFbpFffoc5S6K/E87X8/Uoho4wfKZ3bUOBJ22QbXAzyKU7NCU8d2MKvBdCH8cqY+sJMem9G6WHgdYYpY4S3a/YYKK26a69F9Xo6r8CjFWI4EA2gmveXpg2L1/rgyHGGUcU/u8G90qgHdQt5KCujPtYuWzey0v4b BDRA+Ad+ TtVSCmDb5CvocUPaOzV5EuaH6Xku6jACK1fYZfEG9y4AeiO/0YZyudJu4xE0leey4v09zufhApGKmYfa/eVYOP11+65eqnUEFTwHLf5uDOYvtgaJR2jyCTcaEfnoudzhXAZz8TR7ndoCM95MJ28Y3AsLPHazPS7HNw9jL8a3sCIDgwAp2IpvB+E322gQ6tK0LkBT81bcq+Vn2aYOgGcApYUR0lRS88oXc4Yi7lpeG8WLXCFVzEAVYQm8CdnCmQeppnArYF4XC37UkA5ud4WUKL+quYdSJAK2FmaE/NhIeCqhHP/eExtVICiil7OAcgNBhkZxjr9jEs11Z2lRYDN4jnWt3pLDq8YH/LyDqD7DXvL2N28iy1pAnJEuxTGXY5X3rc8IeiXPILA5KFMZ5FAZxR2A0GuIvbBluUgPma0XjvDtlixhBuDtTE7DctgOV1x8VhwCMoYdcAAnhIqV/g4y7WVpb8CZbkZyFG26tFumsKn9E7KnygJK0MbzgN4jNrVK9TEye9fP0tJczs8ozYNJakXX7vIuKip6UA8HchPCwEXd7GuVl11wB0moWk5sTajA/eOrXQ5MolUGskTPC8OcISPciNF4NeDKB//g1KjOGbuKPjzdhFkilxdc6PyieItTTSHdaVK4SBMeRj5qEsL7N/pxSURXduj8G5kYlDh3dZ2bqF2zCa5dM+EQ+2r/QzoPaCMuQmKJUV9dtfNrL0QHDYqTFV5dmivPzrF6PwJa8/aN867NL8GORoJzmZQ1+2kqfEmLQxtNSCl+WqdCrHR0HmISGTs/rPQ1wtKr9p3MNIiO3+cFIQpgC99gY0hEQeOQcO5yRsbZmDOCiFr3u/pe4lZz/Ow== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Chao Peng Currently in mmu_notifier invalidate path, hva range is recorded and then checked against by mmu_notifier_retry_hva() in the page fault handling path. However, for the to be introduced private memory, a page fault may not have a hva associated, checking gfn(gpa) makes more sense. For existing hva based shared memory, gfn is expected to also work. The only downside is when aliasing multiple gfns to a single hva, the current algorithm of checking multiple ranges could result in a much larger range being rejected. Such aliasing should be uncommon, so the impact is expected small. Suggested-by: Sean Christopherson Signed-off-by: Chao Peng Reviewed-by: Fuad Tabba Tested-by: Fuad Tabba [sean: convert vmx_set_apic_access_page_addr() to gfn-based API] Signed-off-by: Sean Christopherson --- arch/x86/kvm/mmu/mmu.c | 10 ++++++---- arch/x86/kvm/vmx/vmx.c | 11 +++++------ include/linux/kvm_host.h | 33 +++++++++++++++++++++------------ virt/kvm/kvm_main.c | 40 +++++++++++++++++++++++++++++++--------- 4 files changed, 63 insertions(+), 31 deletions(-) diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index e1d011c67cc6..0f0231d2b74f 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -3056,7 +3056,7 @@ static void direct_pte_prefetch(struct kvm_vcpu *vcpu, u64 *sptep) * * There are several ways to safely use this helper: * - * - Check mmu_invalidate_retry_hva() after grabbing the mapping level, before + * - Check mmu_invalidate_retry_gfn() after grabbing the mapping level, before * consuming it. In this case, mmu_lock doesn't need to be held during the * lookup, but it does need to be held while checking the MMU notifier. * @@ -4358,7 +4358,7 @@ static bool is_page_fault_stale(struct kvm_vcpu *vcpu, return true; return fault->slot && - mmu_invalidate_retry_hva(vcpu->kvm, fault->mmu_seq, fault->hva); + mmu_invalidate_retry_gfn(vcpu->kvm, fault->mmu_seq, fault->gfn); } static int direct_page_fault(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault) @@ -6253,7 +6253,9 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end) write_lock(&kvm->mmu_lock); - kvm_mmu_invalidate_begin(kvm, 0, -1ul); + kvm_mmu_invalidate_begin(kvm); + + kvm_mmu_invalidate_range_add(kvm, gfn_start, gfn_end); flush = kvm_rmap_zap_gfn_range(kvm, gfn_start, gfn_end); @@ -6266,7 +6268,7 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end) if (flush) kvm_flush_remote_tlbs_range(kvm, gfn_start, gfn_end - gfn_start); - kvm_mmu_invalidate_end(kvm, 0, -1ul); + kvm_mmu_invalidate_end(kvm); write_unlock(&kvm->mmu_lock); } diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 72e3943f3693..6e502ba93141 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -6757,10 +6757,10 @@ static void vmx_set_apic_access_page_addr(struct kvm_vcpu *vcpu) return; /* - * Grab the memslot so that the hva lookup for the mmu_notifier retry - * is guaranteed to use the same memslot as the pfn lookup, i.e. rely - * on the pfn lookup's validation of the memslot to ensure a valid hva - * is used for the retry check. + * Explicitly grab the memslot using KVM's internal slot ID to ensure + * KVM doesn't unintentionally grab a userspace memslot. It _should_ + * be impossible for userspace to create a memslot for the APIC when + * APICv is enabled, but paranoia won't hurt in this case. */ slot = id_to_memslot(slots, APIC_ACCESS_PAGE_PRIVATE_MEMSLOT); if (!slot || slot->flags & KVM_MEMSLOT_INVALID) @@ -6785,8 +6785,7 @@ static void vmx_set_apic_access_page_addr(struct kvm_vcpu *vcpu) return; read_lock(&vcpu->kvm->mmu_lock); - if (mmu_invalidate_retry_hva(kvm, mmu_seq, - gfn_to_hva_memslot(slot, gfn))) { + if (mmu_invalidate_retry_gfn(kvm, mmu_seq, gfn)) { kvm_make_request(KVM_REQ_APIC_PAGE_RELOAD, vcpu); read_unlock(&vcpu->kvm->mmu_lock); goto out; diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index fb6c6109fdca..11d091688346 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -787,8 +787,8 @@ struct kvm { struct mmu_notifier mmu_notifier; unsigned long mmu_invalidate_seq; long mmu_invalidate_in_progress; - unsigned long mmu_invalidate_range_start; - unsigned long mmu_invalidate_range_end; + gfn_t mmu_invalidate_range_start; + gfn_t mmu_invalidate_range_end; #endif struct list_head devices; u64 manual_dirty_log_protect; @@ -1392,10 +1392,9 @@ void kvm_mmu_free_memory_cache(struct kvm_mmu_memory_cache *mc); void *kvm_mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc); #endif -void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start, - unsigned long end); -void kvm_mmu_invalidate_end(struct kvm *kvm, unsigned long start, - unsigned long end); +void kvm_mmu_invalidate_begin(struct kvm *kvm); +void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end); +void kvm_mmu_invalidate_end(struct kvm *kvm); long kvm_arch_dev_ioctl(struct file *filp, unsigned int ioctl, unsigned long arg); @@ -1970,9 +1969,9 @@ static inline int mmu_invalidate_retry(struct kvm *kvm, unsigned long mmu_seq) return 0; } -static inline int mmu_invalidate_retry_hva(struct kvm *kvm, +static inline int mmu_invalidate_retry_gfn(struct kvm *kvm, unsigned long mmu_seq, - unsigned long hva) + gfn_t gfn) { lockdep_assert_held(&kvm->mmu_lock); /* @@ -1981,10 +1980,20 @@ static inline int mmu_invalidate_retry_hva(struct kvm *kvm, * that might be being invalidated. Note that it may include some false * positives, due to shortcuts when handing concurrent invalidations. */ - if (unlikely(kvm->mmu_invalidate_in_progress) && - hva >= kvm->mmu_invalidate_range_start && - hva < kvm->mmu_invalidate_range_end) - return 1; + if (unlikely(kvm->mmu_invalidate_in_progress)) { + /* + * Dropping mmu_lock after bumping mmu_invalidate_in_progress + * but before updating the range is a KVM bug. + */ + if (WARN_ON_ONCE(kvm->mmu_invalidate_range_start == INVALID_GPA || + kvm->mmu_invalidate_range_end == INVALID_GPA)) + return 1; + + if (gfn >= kvm->mmu_invalidate_range_start && + gfn < kvm->mmu_invalidate_range_end) + return 1; + } + if (kvm->mmu_invalidate_seq != mmu_seq) return 1; return 0; diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 0524933856d4..4fad3b01dc1f 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -543,9 +543,7 @@ static inline struct kvm *mmu_notifier_to_kvm(struct mmu_notifier *mn) typedef bool (*gfn_handler_t)(struct kvm *kvm, struct kvm_gfn_range *range); -typedef void (*on_lock_fn_t)(struct kvm *kvm, unsigned long start, - unsigned long end); - +typedef void (*on_lock_fn_t)(struct kvm *kvm); typedef void (*on_unlock_fn_t)(struct kvm *kvm); struct kvm_mmu_notifier_range { @@ -637,7 +635,8 @@ static __always_inline int __kvm_handle_hva_range(struct kvm *kvm, locked = true; KVM_MMU_LOCK(kvm); if (!IS_KVM_NULL_FN(range->on_lock)) - range->on_lock(kvm, range->start, range->end); + range->on_lock(kvm); + if (IS_KVM_NULL_FN(range->handler)) break; } @@ -742,15 +741,26 @@ static void kvm_mmu_notifier_change_pte(struct mmu_notifier *mn, kvm_handle_hva_range(mn, address, address + 1, arg, kvm_change_spte_gfn); } -void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start, - unsigned long end) +void kvm_mmu_invalidate_begin(struct kvm *kvm) { + lockdep_assert_held_write(&kvm->mmu_lock); /* * The count increase must become visible at unlock time as no * spte can be established without taking the mmu_lock and * count is also read inside the mmu_lock critical section. */ kvm->mmu_invalidate_in_progress++; + + if (likely(kvm->mmu_invalidate_in_progress == 1)) + kvm->mmu_invalidate_range_start = INVALID_GPA; +} + +void kvm_mmu_invalidate_range_add(struct kvm *kvm, gfn_t start, gfn_t end) +{ + lockdep_assert_held_write(&kvm->mmu_lock); + + WARN_ON_ONCE(!kvm->mmu_invalidate_in_progress); + if (likely(kvm->mmu_invalidate_in_progress == 1)) { kvm->mmu_invalidate_range_start = start; kvm->mmu_invalidate_range_end = end; @@ -771,6 +781,12 @@ void kvm_mmu_invalidate_begin(struct kvm *kvm, unsigned long start, } } +static bool kvm_mmu_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) +{ + kvm_mmu_invalidate_range_add(kvm, range->start, range->end); + return kvm_unmap_gfn_range(kvm, range); +} + static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn, const struct mmu_notifier_range *range) { @@ -778,7 +794,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn, const struct kvm_mmu_notifier_range hva_range = { .start = range->start, .end = range->end, - .handler = kvm_unmap_gfn_range, + .handler = kvm_mmu_unmap_gfn_range, .on_lock = kvm_mmu_invalidate_begin, .on_unlock = kvm_arch_guest_memory_reclaimed, .flush_on_ret = true, @@ -817,8 +833,7 @@ static int kvm_mmu_notifier_invalidate_range_start(struct mmu_notifier *mn, return 0; } -void kvm_mmu_invalidate_end(struct kvm *kvm, unsigned long start, - unsigned long end) +void kvm_mmu_invalidate_end(struct kvm *kvm) { /* * This sequence increase will notify the kvm page fault that @@ -833,6 +848,13 @@ void kvm_mmu_invalidate_end(struct kvm *kvm, unsigned long start, * in conjunction with the smp_rmb in mmu_invalidate_retry(). */ kvm->mmu_invalidate_in_progress--; + + /* + * Assert that at least one range must be added between start() and + * end(). Not adding a range isn't fatal, but it is a KVM bug. + */ + WARN_ON_ONCE(kvm->mmu_invalidate_in_progress && + kvm->mmu_invalidate_range_start == INVALID_GPA); } static void kvm_mmu_notifier_invalidate_range_end(struct mmu_notifier *mn, -- 2.42.0.283.g2d96d420d3-goog