From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B802C27C5F for ; Tue, 11 Jun 2024 00:22:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0F6D46B0085; Mon, 10 Jun 2024 20:22:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0A6E56B0088; Mon, 10 Jun 2024 20:22:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E88BC6B0089; Mon, 10 Jun 2024 20:22:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CDCF16B0085 for ; Mon, 10 Jun 2024 20:22:02 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 7CDE9A0542 for ; Tue, 11 Jun 2024 00:22:02 +0000 (UTC) X-FDA: 82216705284.09.5EC440B Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf29.hostedemail.com (Postfix) with ESMTP id B3249120004 for ; Tue, 11 Jun 2024 00:22:00 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zxUJiZfG; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 3p5hnZgoKCLAZjXekWXjedWeeWbU.SecbYdkn-ccalQSa.ehW@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3p5hnZgoKCLAZjXekWXjedWeeWbU.SecbYdkn-ccalQSa.ehW@flex--jthoughton.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718065320; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UYg8xBfSHgODLvTlV3mxh03y8s3TyTlow0j+6oXRZyE=; b=okPj+eR4zFXhpMvVoxZexmeCggVBk5mi3nSf06oLP89vrpJb5pUFWFsW3fElqKILbsQcLk 5G+ijgzf9hgKs8bx5n1hUr4OCjC2WTyF+BT0JMVWeFZYtbg7+DmM0lL66uUCTM8qxLVNuK hpc3wmqhKpk2aI86ogn5B4B7bcdeV08= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=zxUJiZfG; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of 3p5hnZgoKCLAZjXekWXjedWeeWbU.SecbYdkn-ccalQSa.ehW@flex--jthoughton.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3p5hnZgoKCLAZjXekWXjedWeeWbU.SecbYdkn-ccalQSa.ehW@flex--jthoughton.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718065320; a=rsa-sha256; cv=none; b=sfZHL4ofvYMoLas0Bo7+3UfAJQouUGs3Vmzx6NrEp7Vx4mIzpePD9oahS2TJdAdkZrzfIi 6/MEWGTz8OX9kNIytJaesvaYvUUHXWFuRIQWFt4EjwfWnvxUGhHdHpumoSaQcs8ab4p2vV PaNyIKVrB0QmZf3s2kazX9IxsdhK4bk= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-df7a6530373so9383600276.0 for ; Mon, 10 Jun 2024 17:22:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1718065320; x=1718670120; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UYg8xBfSHgODLvTlV3mxh03y8s3TyTlow0j+6oXRZyE=; b=zxUJiZfGCvI8kt+E7p5GlxUmf4/rd70Sa/FTG6pqT4I36qUaKXK0Jof1TRpDrku4n4 h3XXDJE/Z9R4LN017LkM3NGDDFPsWt4Ph7LLbQ4OtkwqbCslMMNrGa5rFLTUsGqwb1Wq d5SdTT88cXYvopiCpxxjiTUbRTlqfc3EhohU8Ty+m706qd0FyhRuMEaDzN1LIXK4RXlH ZxfyglTtqYsw3MxRJTlKMzTgcJRp1JTExphQS92Adu2tfo+rn9doQXIi7aMyOVp0FJiV YcgfIWRYZzLU4kGwJuKpNLRHyjuG3EHoiPt34JkjVAPh1XmFgQRG4xcHA3T+S2SIODv9 8ZIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1718065320; x=1718670120; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UYg8xBfSHgODLvTlV3mxh03y8s3TyTlow0j+6oXRZyE=; b=HiuYjypZ0s3yZXEf1YRSTTOttlYYyh1XG+v56aXxjE9Pxwo2QSxarhIe3zgvA2gOQs KOPa0LNU1E7Oo6P9vdUJsWnhtNZozajb79YdfDI5tRUtQlfh51/h1HbOXCVfpGk9fAmt +v86zmFXTV77+vbZIKwearhVOho48rC4iLiyEyC9bqC//URbaYhbPQHrN5RjA4nR31ZS XlhUffBQ3Y8+Dec65sHtAwBOMoCF8Y/x4nR/E7KUmWU1S5PHOtUrFDu2BUU1I+GjB9DG C3IhSqAPdwwr6S0glnHvWxy0ANGfSjoCGtlbbfmAKpBDxUo1l8JGUG2C8B74Wm7FRudq 1k4w== X-Forwarded-Encrypted: i=1; AJvYcCWu8JAPEbcv3zPqnBS2oZZJRKd/eumGpdVknlzo3qwelo8qLnZpPL3qGx4capa0+/2ZRCZl0CEHbEK+p80yUpknnIc= X-Gm-Message-State: AOJu0YwPfmGoixA2oDFlmR4DDBV4E9gZsK+dxX7yCQq4sJWz/jhM18m1 VQmvabMapcTjStaKybHUGaVmjxNcbv5hHW0n9u2e5Ry2ZFC78ur9QQhad6DtoxdxeQhIpgJYHkg fBwZ0Tse05e8sjYjn+A== X-Google-Smtp-Source: AGHT+IEGsmdXMyxgJ1Cj/3hdc6/ErG88swxZnK/rdFuCwoqmhzkdtuuICPRcxc2axyqsks7ulwYmlzACcQfHnSLC X-Received: from jthoughton.c.googlers.com ([fda3:e722:ac3:cc00:14:4d90:c0a8:2a4f]) (user=jthoughton job=sendgmr) by 2002:a05:6902:1003:b0:dd9:20c1:85b6 with SMTP id 3f1490d57ef6-dfd9fcb617bmr288182276.2.1718065319751; Mon, 10 Jun 2024 17:21:59 -0700 (PDT) Date: Tue, 11 Jun 2024 00:21:37 +0000 In-Reply-To: <20240611002145.2078921-1-jthoughton@google.com> Mime-Version: 1.0 References: <20240611002145.2078921-1-jthoughton@google.com> X-Mailer: git-send-email 2.45.2.505.gda0bf45e8d-goog Message-ID: <20240611002145.2078921-2-jthoughton@google.com> Subject: [PATCH v5 1/9] KVM: Add lockless memslot walk to KVM From: James Houghton To: Andrew Morton , Paolo Bonzini Cc: Ankit Agrawal , Axel Rasmussen , Catalin Marinas , David Matlack , David Rientjes , James Houghton , James Morse , Jonathan Corbet , Marc Zyngier , Oliver Upton , Raghavendra Rao Ananta , Ryan Roberts , Sean Christopherson , Shaoqin Huang , Suzuki K Poulose , Wei Xu , Will Deacon , Yu Zhao , Zenghui Yu , kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: B3249120004 X-Stat-Signature: 8hf9n36cpcjs1we5wrq1kn8sw56tom5n X-Rspam-User: X-HE-Tag: 1718065320-991318 X-HE-Meta: U2FsdGVkX1/qh64gOJtRS+paBE5E9E6tPJSi/h7NE1SZ44ivnrE+Wtp7lvWXFsgZONHwTLQoiaYVegGIsoFf8pc7xf77pz/68HjkMjHp1dNOScHpVlCnJu34G558hKHP0Ro6x+4yZKj9ThITFStmDgUGpAeRp4gN7FgsyCKiks4S+gzxBfGaZFNdKf2dPEEkQHdAP+tgZrSdBFzrtSLPDXiJluhO9+Z82VeWRiZ2lQn6SL2nBQvgtVMCl7x9PcL+WrbNVeZO96rLFVMFvwoiT7oj7OnHJJpU2Eteek39VWK3WCnpz1UPHdNrW1yLriUs/wNXChGqjNt5Z6jaDfJPsQ9E62BdNaTY4jaNPDcatnLJWw4S6WHgM4FoJ85OhzOw9V4gHYoVK2iJKVSOxrG8ED/ZPUy0XJLFV45IaGpEi5R8lOTlYcF3VHqTUyIb51o6kUABYYJ1DZSgVVgo11VUKBNPt3f+nsFUwX/oMCrhyGGebWyryENZordkZqcUnNZVMgH9mWQgoZjbALL229Nl8+ov3YHiEmA6RmdTDDqULMF55apHix5kjsUf2Y5v6TJ62QgZ//wVV0b6cUT3R5XBITKM1dBoEoLKPgCjqeWHuyKLNaMIYM+9dsP8zQkuClFa1YPobrqaAhJeHv2kXXvDnq3UYBRnFWIWdsSMbuOlUA26riPumUtK5BATiX9TBbevgZcFRujTXTWxqT3spHAotAAiLmDhDitDRaReojZ9iC/Wu5CS2nAIfRWwOUG2SPaZngL7TD6VqDo49Pj6k0iKbHhD4W4VeQtnNuVUhRVxEipfpnxJseE56Zf2DZFwy7cuO9YhMjf6EQC3ZFJCAUg+UR/2S23sIHsuO+xGw4ihh40yIaqc6g3gKJblqRuiSaoJ2by2g8f24w4nXTqEUwS2pxy1MpbxscVMQI86t0zhKpum2bEMyK6RktB0JDVYsuys6YREfMbBXJkPJWwMCL2 bteZgd9o JYoMXmB4IXbIFRbwzJrycnG6YstDDtayRt/e/of/+ep9XC6sT2HwU3GWXSO5ZCWKFL0oAqKhR9u4horYy0tOuvAVyUHNJKcf1mT6N2Axh5gbKUXp0HpFGeLgK0mCa5iCPM7kAqnmtDjw4Ye6k8oR5cZquptzu+5FIo+xh2qFDy8ntAk/XNzw+usJMX74SSrnbAu+HRsRgrD3tTw5Jl+zqa/5UrMZcNzxzi9VPERYRibK+yBZXQnAEp4FTeOfsqt82Zy6NKe4SjoIxyTkrRoTtJaeguzO6egDRnydLqJNnlz+QunRCLkxDLNp1uPvBvOpA+x4kYW4BViipDMrLGJykpMQPYM3NPyOAWrA1fi4++FIZrHb6xBUhuWzHquY8Gr8uklh79I8GuYkZrKZVdQETs7PS7uV/TRjmb6OCasFKsCkIYIn9n//6R9vdzu6/ilao7ZSaXunhsXCEu0Me30wYMJO5yfQ7h+KBBsBh/IYhM6HvWCfcg8RIu+sze+Xpq8wEb8SAziLxfUlWLjWDAOI/qNf3daIhNEq8nlcX X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Provide flexibility to the architecture to synchronize as optimally as they can instead of always taking the MMU lock for writing. Architectures that do their own locking must select CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS. The immediate application is to allow architectures to implement the test/clear_young MMU notifiers more cheaply. Suggested-by: Yu Zhao Signed-off-by: James Houghton --- include/linux/kvm_host.h | 1 + virt/kvm/Kconfig | 3 +++ virt/kvm/kvm_main.c | 26 +++++++++++++++++++------- 3 files changed, 23 insertions(+), 7 deletions(-) diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index 692c01e41a18..4d7c3e8632e6 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -266,6 +266,7 @@ struct kvm_gfn_range { gfn_t end; union kvm_mmu_notifier_arg arg; bool may_block; + bool lockless; }; bool kvm_unmap_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range); bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range); diff --git a/virt/kvm/Kconfig b/virt/kvm/Kconfig index 29b73eedfe74..0404857c1702 100644 --- a/virt/kvm/Kconfig +++ b/virt/kvm/Kconfig @@ -97,6 +97,9 @@ config KVM_GENERIC_MMU_NOTIFIER select MMU_NOTIFIER bool +config KVM_MMU_NOTIFIER_YOUNG_LOCKLESS + bool + config KVM_GENERIC_MEMORY_ATTRIBUTES depends on KVM_GENERIC_MMU_NOTIFIER bool diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 14841acb8b95..d8fa0d617f12 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -558,6 +558,7 @@ struct kvm_mmu_notifier_range { on_lock_fn_t on_lock; bool flush_on_ret; bool may_block; + bool lockless; }; /* @@ -612,6 +613,10 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, IS_KVM_NULL_FN(range->handler))) return r; + /* on_lock will never be called for lockless walks */ + if (WARN_ON_ONCE(range->lockless && !IS_KVM_NULL_FN(range->on_lock))) + return r; + idx = srcu_read_lock(&kvm->srcu); for (i = 0; i < kvm_arch_nr_memslot_as_ids(kvm); i++) { @@ -643,15 +648,18 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, gfn_range.start = hva_to_gfn_memslot(hva_start, slot); gfn_range.end = hva_to_gfn_memslot(hva_end + PAGE_SIZE - 1, slot); gfn_range.slot = slot; + gfn_range.lockless = range->lockless; if (!r.found_memslot) { r.found_memslot = true; - KVM_MMU_LOCK(kvm); - if (!IS_KVM_NULL_FN(range->on_lock)) - range->on_lock(kvm); - - if (IS_KVM_NULL_FN(range->handler)) - break; + if (!range->lockless) { + KVM_MMU_LOCK(kvm); + if (!IS_KVM_NULL_FN(range->on_lock)) + range->on_lock(kvm); + + if (IS_KVM_NULL_FN(range->handler)) + break; + } } r.ret |= range->handler(kvm, &gfn_range); } @@ -660,7 +668,7 @@ static __always_inline kvm_mn_ret_t __kvm_handle_hva_range(struct kvm *kvm, if (range->flush_on_ret && r.ret) kvm_flush_remote_tlbs(kvm); - if (r.found_memslot) + if (r.found_memslot && !range->lockless) KVM_MMU_UNLOCK(kvm); srcu_read_unlock(&kvm->srcu, idx); @@ -681,6 +689,8 @@ static __always_inline int kvm_handle_hva_range(struct mmu_notifier *mn, .on_lock = (void *)kvm_null_fn, .flush_on_ret = true, .may_block = false, + .lockless = + IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), }; return __kvm_handle_hva_range(kvm, &range).ret; @@ -699,6 +709,8 @@ static __always_inline int kvm_handle_hva_range_no_flush(struct mmu_notifier *mn .on_lock = (void *)kvm_null_fn, .flush_on_ret = false, .may_block = false, + .lockless = + IS_ENABLED(CONFIG_KVM_MMU_NOTIFIER_YOUNG_LOCKLESS), }; return __kvm_handle_hva_range(kvm, &range).ret; -- 2.45.2.505.gda0bf45e8d-goog