From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA8F1C27C55 for ; Tue, 11 Jun 2024 00:22:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C45EF6B0089; Mon, 10 Jun 2024 20:22:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BA5516B0092; Mon, 10 Jun 2024 20:22:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9D0F26B0093; Mon, 10 Jun 2024 20:22:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 746266B0089 for ; Mon, 10 Jun 2024 20:22:04 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 390CB1C1E75 for ; Tue, 11 Jun 2024 00:22:04 +0000 (UTC) X-FDA: 82216705368.05.93C6AD1 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf29.hostedemail.com (Postfix) with ESMTP id 6E06A120012 for ; Tue, 11 Jun 2024 00:22:02 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=l6b9mXFu; spf=pass (imf29.hostedemail.com: domain of 3qZhnZgoKCLIblZgmYZlgfYggYdW.Ugedafmp-eecnSUc.gjY@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3qZhnZgoKCLIblZgmYZlgfYggYdW.Ugedafmp-eecnSUc.gjY@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718065322; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2WPKTEITGM8+K86UxPq3iqbgD4SGKZWY5JBo64AhYsQ=; b=wcYdmY0BhD+9q42h40lVQtcABu3QEPkyR6MKDLPZnuixrE1ib0N2KVbQwUnMHtsTa0g0I/ huyFMqA/lzeSbTnmBn6GMFHakhElCi/WRCC4YDsxfbXezsURkF/a/ZMP7KWap4nGov9dVV nWMQryvFPGeJe/HGH/+4jnrIrFP+JRI= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=l6b9mXFu; spf=pass (imf29.hostedemail.com: domain of 3qZhnZgoKCLIblZgmYZlgfYggYdW.Ugedafmp-eecnSUc.gjY@flex--jthoughton.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3qZhnZgoKCLIblZgmYZlgfYggYdW.Ugedafmp-eecnSUc.gjY@flex--jthoughton.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718065322; a=rsa-sha256; cv=none; b=cSM+CqkloPUvNAFz+LhJzl4ldfqN8fbspukYkLhak5irOkNd14Kb7Nh+NpZ78oajGQGdw7 QIYznV5jcrXZP7NlesK313EJfzvcOdrtpCcq44cnmMN24+8xeYh1nk2a7TLTjVk59ujWqJ /IJuVuC34YyqCHwUOIZoiwqcH0P9JHE= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dfaf7ead784so7082400276.0 for ; Mon, 10 Jun 2024 17:22:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1718065321; x=1718670121; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2WPKTEITGM8+K86UxPq3iqbgD4SGKZWY5JBo64AhYsQ=; b=l6b9mXFunW1dllgpY1V0ssgxjuwK+ogHYAQOlK1dij6uXv02EbhE+v+wucT0nlPzbq cgEG2m+dwQXEaFosztPSGF3t5BzUyF74NOWdqGv73FgrubPA+aZ81HJaMTTzm0zO89T3 Br8IOtWOvOKzj6+zcC2RQStonjDYJAgSIRXj5iSEGqGECzB6gxZPugDKD+JMjPVHPZ6T JYpCbhwQyqRJswMA69xkS7iMWpdKCUAJC83zjgk1sJi0OZrqiAPkjpFJA+1ULSO829Ij OiHCutN05Ch8Pt0XtdIq82GYujDm4Y7kPilQj8N/QqaZHpvNVVufNePj/HXUX4L5Ykkb WaVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718065321; x=1718670121; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2WPKTEITGM8+K86UxPq3iqbgD4SGKZWY5JBo64AhYsQ=; b=iadbyCNOsw3E8ct80OQDCADXq8QVUb7IXkL0pTtsR9zb7eRg28+KKc9b2pDyRts3A6 mw3mx0so9EtGWZsCBhNSRGtHrsFjXDKbkdebGbiGbDy+qzVJ0RpfRZuGEz8xaRKsp10R Ea++lEcJk8gTb97llxFqllubmo6qCRz1AGrV34jJFciD6rab4og06H06jJ5P1/V4UTB4 llPegKJx/hZ0e0YM6wN0GhfvxQSM7Lz4nxCvsmD8xrYzEmT8n5mHTRhRCDmvk6gEN39C cUm+9OIxOOpCI4WcJAsb8fwFJKzrbGM9K4Cex9cxvSKbfVr/CQPmJmJNU3809gcexyN2 l7Ng== X-Forwarded-Encrypted: i=1; AJvYcCWNJCZECPmNBjVWU7+ar+6zbGwwAF/bPudUaKTniH6liZ7MOUT8BrbWZquvZr4Py7AXv7NG8EtCE1P7xxlyRI5eGh8= X-Gm-Message-State: AOJu0Ywt9/CdZNUyb3fGZk1fzd5fQlWvisHHCS4Q0sTzp5pNRCRfn3np j6sJoe3JF5udifp+asi8xQdjvyIycL6yROm4gZwfLKyFX2/s/RZyX7ju/VZb2FFaFg09taBrX3J hyFRG/deaZ4/FaoL7eA== X-Google-Smtp-Source: AGHT+IEkfpXJhDpYSXj2zWTmc5g4eYevyTxa6JBBNjQgHAzXAToudTEtQid77N/s8K4jHcfyWfR4EDBUwm2H5zD1 X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6902:100a:b0:df4:ece5:2720 with SMTP id 3f1490d57ef6-dfaf6608d72mr3353252276.13.1718065321541; Mon, 10 Jun 2024 17:22:01 -0700 (PDT) Date: Tue, 11 Jun 2024 00:21:39 +0000 In-Reply-To: <20240611002145.2078921-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240611002145.2078921-1-jthoughton@google.com> X-Mailer: git-send-email 2.45.2.505.gda0bf45e8d-goog Message-ID: <20240611002145.2078921-4-jthoughton@google.com> Subject: [PATCH v5 3/9] KVM: arm64: Relax locking for kvm_test_age_gfn and kvm_age_gfn From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Ankit Agrawal , Axel Rasmussen , Catalin Marinas , David Matlack , David Rientjes , James Houghton , James Morse , Jonathan Corbet , Marc Zyngier , Oliver Upton , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Suzuki K Poulose , Wei Xu , Will Deacon , Yu Zhao , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" X-Rspam-User: X-Rspamd-Queue-Id: 6E06A120012 X-Rspamd-Server: rspam01 X-Stat-Signature: jgz6nzuy6ktmu5x79y8or9qsd5zrrskk X-HE-Tag: 1718065322-676257 X-HE-Meta: U2FsdGVkX197b/nP/nkYrGOeyXT7ka7VlTjjIgJRpeiKv9a3BuINcQmPeyHjPmDjyv/cmjZnamAf9khLVZoVI4/j9Sa8zEBP1DFS/uv3GWUGGx74Wx4MgSfUhqY5yaZCfUmCQAdF+IPP0N8Gm+1bsOjATgGGw/KGUSj6B7ySHJRJmG8LzFcuwNDYF76jPSNEWzn4uKh19YYYCShvdIe8hOteJnV8a/djS5YS92syGp1yW+78jzCBDx0o719V3FmWfJBG2H1wdq/1xpa0Ee8V+rJTT3COnzGlH327ZgGPE422z8+C4BkmoAYCCXBYHeOpjPwEXa47DGdeO32/NMn2S9jsFZCHkGqZgELG5T84euhlr5hAGovnpfLO+lW9s6gDzUBKzzY433L3kIbBcQEhPF90rxiAZeLpRUcLS42X2Uyc9DWZhZduBDslRezHB9kcIKsb7IXwk1ihrsYBKbFhWpDNZFssnWUuiFo13+QOeP7cs8lvNMkPw9ubdL0daY4vcB+6zQuZ/sqmV0oaZZZgkahwJ6eysHcp/RCg6lOWp4wOssWQgcffouJrGh518clSh/s8Zdu2Xwd3EBHg+fmAh1wcq11YGpe/JOsWJQoAJvBXjcPK/LHIlndGCmTytXKkZSxtO8GgNLhC7dMnjRmdE6ynxucSiv3ZRGISoZCxuhK8mc8L6QSWROyOHja05RWH7N3oRc+7Ntxeu9zdRW+GOxKJcODfocsSjaWVkbWaSedNV7GpRaLRYc3jrGZUrbM51UR+8cyCE7XyyTBWW5kaFTAYGSbuvfYkQA38oM0y7eGQScYMsExDCeQrz37/c+Z4rURVcNzji30EJZKrrPDekTLuRk2buKSMCgydULwC9LMSIwGXIroB80Zec17+n3iiVD8LKcQl40IgTx1GWcaZmjIf5R9WhfonQ6H9HsseSjHkIj0bofeUV/vvrccUiOGWGcqUPL6oHmswPDu3gbr 9WghqXsI h6VxsnDrwjePAL8M3Sk9usYl42+5ofYBPbjLc/Xsk5afesbtrAduYRGwAfpo7U+sv1vZDQIsr5jquRhJzExxLYi23DC838mb/AL0l9c3qaXe7yheEaD+kwlOFI3i4ZM+E4CH21cbekb5vOB1Hd9aANcQPVkT7hIeR5CjgNTBkQV6zDzSdvRRCvq/3VWK/uRuE5wvvb8mi70EAytEg4578CPO6KfGQ5//8VlxUsg7vUI+zSM3HJwlWcuDZVXC7YoeVOnkR7SxqxFeu1a4kYqWKeONSEM3oaad5HTa1s9/e6LSMZMIFd9WhzuORzny6vBLop66JK3Ry2Sn4ut12sMcvVsCGxZJB/r956pNS4uufN29psIBApUErp4Ke+MfqDgWPgczb19DU4QAST4q06FtahLDK6N5Y1tMvbR7gtKqxY3X6N++4KIVbkNs3fiGdH+4XbIHpfx02YjC8Eek8DwH4ojCmEgxts3Y+GguY/RlXPlqy3nkZg4VifVjoFOLfg2rSqr6235ScNtjt5ZOSFVGVEp54JAFD3yYK6rhIZWSNmWY8L6NlLQsrlJo4CDWt7YWU7M7ugojzy2NNA8pYOzzf8g53mJzbG7+zLp4C X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Replace the MMU write locks (taken in the memslot iteration loop) for read locks. Grabbing the read lock instead of the write lock is safe because the only requirement we have is that the stage-2 page tables do not get deallocated while we are walking them. The stage2_age_walker() callback is safe to race with itself; update the comment to reflect the synchronization change. Signed-off-by: James Houghton --- arch/arm64/kvm/Kconfig | 1 + arch/arm64/kvm/hyp/pgtable.c | 15 +++++++++------ arch/arm64/kvm/mmu.c | 26 ++++++++++++++++++++------ 3 files changed, 30 insertions(+), 12 deletions(-) diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig index 58f09370d17e..7a1af8141c0e 100644 --- a/arch/arm64/kvm/Kconfig +++ b/arch/arm64/kvm/Kconfig @@ -22,6 +22,7 @@ menuconfig KVM select KVM_COMMON select KVM_GENERIC_HARDWARE_ENABLING select KVM_GENERIC_MMU_NOTIFIER + select KVM_MMU_NOTIFIER_YOUNG_LOCKLESS select HAVE_KVM_CPU_RELAX_INTERCEPT select KVM_MMIO select KVM_GENERIC_DIRTYLOG_READ_PROTECT diff --git a/arch/arm64/kvm/hyp/pgtable.c b/arch/arm64/kvm/hyp/pgtable.c index 9e2bbee77491..b1b0f7148cff 100644 --- a/arch/arm64/kvm/hyp/pgtable.c +++ b/arch/arm64/kvm/hyp/pgtable.c @@ -1319,10 +1319,10 @@ static int stage2_age_walker(const struct kvm_pgtable_visit_ctx *ctx, data->young = true; /* - * stage2_age_walker() is always called while holding the MMU lock for - * write, so this will always succeed. Nonetheless, this deliberately - * follows the race detection pattern of the other stage-2 walkers in - * case the locking mechanics of the MMU notifiers is ever changed. + * This walk may not be exclusive; the PTE is permitted to change + * from under us. If there is a race to update this PTE, then the + * GFN is most likely young, so failing to clear the AF is likely + * to be inconsequential. */ if (data->mkold && !stage2_try_set_pte(ctx, new)) return -EAGAIN; @@ -1345,10 +1345,13 @@ bool kvm_pgtable_stage2_test_clear_young(struct kvm_pgtable *pgt, u64 addr, struct kvm_pgtable_walker walker = { .cb = stage2_age_walker, .arg = &data, - .flags = KVM_PGTABLE_WALK_LEAF, + .flags = KVM_PGTABLE_WALK_LEAF | + KVM_PGTABLE_WALK_SHARED, }; + int r; - WARN_ON(kvm_pgtable_walk(pgt, addr, size, &walker)); + r = kvm_pgtable_walk(pgt, addr, size, &walker); + WARN_ON(r && r != -EAGAIN); return data.young; } diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 8bcab0cc3fe9..a62c27a347ed 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1773,25 +1773,39 @@ bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { u64 size = (range->end - range->start) << PAGE_SHIFT; + bool young = false; + + read_lock(&kvm->mmu_lock); if (!kvm->arch.mmu.pgt) return false; - return kvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt, - range->start << PAGE_SHIFT, - size, true); + young = kvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt, + range->start << PAGE_SHIFT, + size, true); + +out: + read_unlock(&kvm->mmu_lock); + return young; } bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { u64 size = (range->end - range->start) << PAGE_SHIFT; + bool young = false; + + read_lock(&kvm->mmu_lock); if (!kvm->arch.mmu.pgt) return false; - return kvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt, - range->start << PAGE_SHIFT, - size, false); + young = kvm_pgtable_stage2_test_clear_young(kvm->arch.mmu.pgt, + range->start << PAGE_SHIFT, + size, false); + +out: + read_unlock(&kvm->mmu_lock); + return young; } phys_addr_t kvm_mmu_get_httbr(void) -- 2.45.2.505.gda0bf45e8d-goog