From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4D1A1C71131 for ; Wed, 11 Jun 2025 13:34:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 47AF26B00A9; Wed, 11 Jun 2025 09:34:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 42B8D6B00AA; Wed, 11 Jun 2025 09:34:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2F4EE6B00AB; Wed, 11 Jun 2025 09:34:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 0DD296B00A9 for ; Wed, 11 Jun 2025 09:34:03 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B42F0BE283 for ; Wed, 11 Jun 2025 13:34:02 +0000 (UTC) X-FDA: 83543213124.05.BF68DA5 Received: from mail-wm1-f73.google.com (mail-wm1-f73.google.com [209.85.128.73]) by imf17.hostedemail.com (Postfix) with ESMTP id C9C2C4000C for ; Wed, 11 Jun 2025 13:34:00 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=OxynOcNS; spf=pass (imf17.hostedemail.com: domain of 3x4VJaAUKCEIxeffeksskpi.gsqpmry1-qqozego.svk@flex--tabba.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=3x4VJaAUKCEIxeffeksskpi.gsqpmry1-qqozego.svk@flex--tabba.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1749648840; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Xs7DJ5+xZJN+yjOANfS++zfiFVzFqjriEWdIEfE1s+w=; b=ZiRO11agUirtok3zVli4U3SNW0lRKM6lqaNMXmdLqLFRWkBMsni8zB2fpiwGcCff/FFtWu DROOzcRrLA11R565cTqcokKDfsLQYd4RqBsCSqJKX2MNmsG3+Qxot8m2geOudUx9QkxymQ VdqVgzJEu4+swsN4PrGWdwxnRN/Oa/8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1749648840; a=rsa-sha256; cv=none; b=xAfbzZ3Z/DcClRCjEKGNWJcBq9VjQyUlhXoTta7xc9RC0+yHxan4Tt9bYeHglrATG3ZlMX Nc/a1cbxRDBX1ydDASJhHpNnXBKr3x68PfYVIRQ7VbUtir38C1C9PoG7fpOBx4oPQ8HsOF I9WioyP7xhaaim4ORGNFGgLVucGHq4c= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=OxynOcNS; spf=pass (imf17.hostedemail.com: domain of 3x4VJaAUKCEIxeffeksskpi.gsqpmry1-qqozego.svk@flex--tabba.bounces.google.com designates 209.85.128.73 as permitted sender) smtp.mailfrom=3x4VJaAUKCEIxeffeksskpi.gsqpmry1-qqozego.svk@flex--tabba.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-wm1-f73.google.com with SMTP id 5b1f17b1804b1-450cf229025so21795915e9.1 for ; Wed, 11 Jun 2025 06:34:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1749648839; x=1750253639; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Xs7DJ5+xZJN+yjOANfS++zfiFVzFqjriEWdIEfE1s+w=; b=OxynOcNS4uRgyUjoCpPBznXZJgi80WGjtH0k4Z6CmNj9kzoiVsBtLhJKFmYNKNKLh5 5BhZF8XecssFbzpDCU34/2KyXhNKhEDryRO6vbsw31hhd0zn83rjOpWiZZCFWZvPVbU8 7ugUaEDIWWW3L8kRVSCv1a+GCFBe3jcZCOqhzD7WXogtnTjN9gQVPz/R2E1RI8LE2Yx9 /WOPU9mxX0BwVHY/yKOpD1cgKB0MYpRMTWlSwEAe+r1Lsk/MhXv0c4z4j1XfULoVqtMD 2/XwKkr0shjwvZdz8zkmQ1+sh/pbnXVxfAgvO32mTq5mMNrUNYWQG4L3SR6BAGXG9lIp eJ0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749648839; x=1750253639; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Xs7DJ5+xZJN+yjOANfS++zfiFVzFqjriEWdIEfE1s+w=; b=Lww+4k7QtSgf3l0jsIAC2jeEuBj2z54fKPkptYfiHN5DDq3fg3YnQDZR+43aGg99hW kKfOnBA3do2fPsLmsqDWfaVa/2Am6yMEgI0PfwcXAPjWATqnflocV3ikxAiQlY+/tYDe zrRvUAedlASRiEoGBWjFd/led7/YHY23EHoQ29eA8eOqaUzWXbS+JZ8F2dnDvPRmB30Z oSqF4YQaCJfO2RICFvxyiu5m3am5wdeGYZnrKVZ4Upj+9DTMBf02fRAzTVxtx4gFyJ7P wnsZ80Whr0ycrLmwipcG3sFYc13YpZXjpmpltT8Dm9+GC9HHcm1m+O8br3Dc3lbz79f3 K4Ug== X-Forwarded-Encrypted: i=1; AJvYcCWUHw1FVBXjiKMxNbDl9fv5oIAPu1cq7bbbElRIu7kQXnqu2cn+QjD0Ho6v4x9ryNEUfiaZ1SnGNw==@kvack.org X-Gm-Message-State: AOJu0YwPG6E+gZzUVDhsBfqPn1elRlVJvmkPeeeT/6b9EF8YjU0ftY6f lIMRd4nva1pt01iwy8qc89CzZSYaY8tGwMBqQTO0vmsNrnvGQ42meCJvcxHpOYKPetK5cMxjl3z 1Wg== X-Google-Smtp-Source: AGHT+IFDevYzogmvQMnwc6T7ca/b/tL8625Ge3voTKyD0LsZPA1p6DBJpQhu/7NjaftdNHQ2VpbCh3tzTA== X-Received: from wmbji2.prod.google.com ([2002:a05:600c:a342:b0:453:910:1a18]) (user=tabba job=prod-delivery.src-stubby-dispatcher) by 2002:a05:600c:5295:b0:43c:fcbc:9680 with SMTP id 5b1f17b1804b1-453248d2509mr26027135e9.25.1749648839430; Wed, 11 Jun 2025 06:33:59 -0700 (PDT) Date: Wed, 11 Jun 2025 14:33:25 +0100 In-Reply-To: <20250611133330.1514028-1-tabba@google.com> Mime-Version: 1.0 References: <20250611133330.1514028-1-tabba@google.com> X-Mailer: git-send-email 2.50.0.rc0.642.g800a2b2222-goog Message-ID: <20250611133330.1514028-14-tabba@google.com> Subject: [PATCH v12 13/18] KVM: arm64: Refactor user_mem_abort() From: Fuad Tabba To: kvm@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-mm@kvack.org, kvmarm@lists.linux.dev Cc: pbonzini@redhat.com, chenhuacai@kernel.org, mpe@ellerman.id.au, anup@brainfault.org, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, seanjc@google.com, viro@zeniv.linux.org.uk, brauner@kernel.org, willy@infradead.org, akpm@linux-foundation.org, xiaoyao.li@intel.com, yilun.xu@intel.com, chao.p.peng@linux.intel.com, jarkko@kernel.org, amoorthy@google.com, dmatlack@google.com, isaku.yamahata@intel.com, mic@digikod.net, vbabka@suse.cz, vannapurve@google.com, ackerleytng@google.com, mail@maciej.szmigiero.name, david@redhat.com, michael.roth@amd.com, wei.w.wang@intel.com, liam.merwick@oracle.com, isaku.yamahata@gmail.com, kirill.shutemov@linux.intel.com, suzuki.poulose@arm.com, steven.price@arm.com, quic_eberman@quicinc.com, quic_mnalajal@quicinc.com, quic_tsoni@quicinc.com, quic_svaddagi@quicinc.com, quic_cvanscha@quicinc.com, quic_pderrin@quicinc.com, quic_pheragu@quicinc.com, catalin.marinas@arm.com, james.morse@arm.com, yuzenghui@huawei.com, oliver.upton@linux.dev, maz@kernel.org, will@kernel.org, qperret@google.com, keirf@google.com, roypat@amazon.co.uk, shuah@kernel.org, hch@infradead.org, jgg@nvidia.com, rientjes@google.com, jhubbard@nvidia.com, fvdl@google.com, hughd@google.com, jthoughton@google.com, peterx@redhat.com, pankaj.gupta@amd.com, ira.weiny@intel.com, tabba@google.com Content-Type: text/plain; charset="UTF-8" X-Rspam-User: X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: C9C2C4000C X-Stat-Signature: xwzpunc9c1xuueecbkicoo6u9pd31x8c X-HE-Tag: 1749648840-389260 X-HE-Meta: U2FsdGVkX1/u2UbbOwX+3DOpcrNLh/cQpNF6cYOCErB67dpYLg2TzkJ4Dr992dU+aAp2lfYB7kvvIIDUxnRya2kmZWSCGxWvSHwLb0fOKur/Fk4W35STaQ55kVgpWdxVqvHPC02pgYf4kkBvWp2W6qefowE63oaBbXjJzYK5FXJPdCRplRZbHamKQGknNXGJfb6tCcM4JVkrKGgftU9eq+SPwfLsxPLMhERhKQEJD6sleM1T7TYuCnaxH/mINr5a6HzpJh2mL856hBpC07g01HF8wFsBbrDmD7Yem/QgMRhVgxrHUkbO27Rd9+RyKRn/zjIWlOrAZ80+sTSKSA4EEK0t5SJkWI2xfPBx3WkoBsv6/7fsc7Gfehhec+WjnQxfhUk42TEH5+91tmv34hd3zGLNk2RWz8bQezyM4uKz61Pe3DuLpKl+fUvykKyAvItdSasi3bQKIFT19BUeSwyBG1Sf7LgyD9a8z8TPPYvXESnJWu9wcHtLpgMPOvtrAaWoKizN/EhOny7/heAT/MEIL2joBALBynwKTnkunpWNoq+ZdmN7Ibxt0BFLru8Jcyz8KQj4Yc31zUoA6E+1/3G9c3Q+M2CEtlhomWWtc6Dhq/vNAJNCs6/5/2hb7Gk+d75QowOztLDybaO2ic3xrCkD8l1NWahH97XGW+YHJJdO+Rm5uHuDQA7fd3y22kgDVFo4Kq1HcvgEmZIOrTcYRHsBGWfMbwtIXC/COfv09wzaeTFbrZZTU4wz8nqESBOfWcaCQDaXSHu1mgvWsGUXVidKorUm8MCqlr4xyUPdo709vQ7HXlpryokg2OuwfwbtKb6XNEdqGSNI9EQ2AtbImQRHM/Rg+nyeDjU1ah4Q9mM0mj2yn9taSmmgBOCkW/dSeaA6Soyc1UIgckcKX4v2ZTeqeNwfLUNp/k0HcI5tV6mVjmRkgturgFkpjFzOxdZXdCW2IRoZP2eczVvydwEGvb0 M7ru5tUF b0o6JIenV7irIHJGc+zMxRMCDHzEPRv6ZJyGgwVPH97nG3C/SIAxRWCXtrsgM/ZdsRc3HuyaweJHlYu7C3ikju35Og1ISIUK3shyZ/CEetcm86g8xcjDdn90VnBG6WcMZsNSdwpxs2mqQTUOkGuUDeaP0EWhbGXSgQr13W2QKceQuNXf9ITFktNm5wILNaEDQQbxNkqA0c0Nl29V2hXsi1bOTtnFlZJAItDQ4lks6yaguNjSsQzcaiIV1IKEn/1Lqp9HHf1E4ZeTpN7dHkIuEC+BKsbFosAjKTdi684KEOI+XU+D9jdQvFJblRtsYXlhqm9h5D6SiECurByFOewpCUBL9MXc1GDmq49pzK0WLmlzAgrpdaPgKtQvmGcyHLcG56SrgJUHal4fG6F873eq3TbJQEbYVBcCXj038ekhlQSwURv03sCfobZoys/+nPdSc7P6hroe5tlFo0TD1DxiSOr4oHYnUXfL7G4EG4HxikxI+SjD+ONrw55q70Y2eDuWDw9e4NoBHN9bOUxFaTPMmrb1iDeq+qEox/onBHGGLexgHWOgyQPYv0k4LyaJ+hlOKseajALKA1DELMaPj0myxRmJC5Fi7oK5VrmzWyGY77NZec5Iv9Fv7+rTScaYIHfbJQ9WIyBJIclOCvHHwvqNLnMRNk5kwKk5cZLfsJVyA4qMjXvA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To simplify the code and to make the assumptions clearer, refactor user_mem_abort() by immediately setting force_pte to true if the conditions are met. Remove the comment about logging_active being guaranteed to never be true for VM_PFNMAP memslots, since it's not actually correct. Move code that will be reused in the following patch into separate functions. Other small instances of tidying up. No functional change intended. Reviewed-by: Gavin Shan Signed-off-by: Fuad Tabba --- arch/arm64/kvm/mmu.c | 100 ++++++++++++++++++++++++------------------- 1 file changed, 55 insertions(+), 45 deletions(-) diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 2942ec92c5a4..58662e0ef13e 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1470,13 +1470,56 @@ static bool kvm_vma_mte_allowed(struct vm_area_struct *vma) return vma->vm_flags & VM_MTE_ALLOWED; } +static int prepare_mmu_memcache(struct kvm_vcpu *vcpu, bool topup_memcache, + void **memcache) +{ + int min_pages; + + if (!is_protected_kvm_enabled()) + *memcache = &vcpu->arch.mmu_page_cache; + else + *memcache = &vcpu->arch.pkvm_memcache; + + if (!topup_memcache) + return 0; + + min_pages = kvm_mmu_cache_min_pages(vcpu->arch.hw_mmu); + + if (!is_protected_kvm_enabled()) + return kvm_mmu_topup_memory_cache(*memcache, min_pages); + + return topup_hyp_memcache(*memcache, min_pages); +} + +/* + * Potentially reduce shadow S2 permissions to match the guest's own S2. For + * exec faults, we'd only reach this point if the guest actually allowed it (see + * kvm_s2_handle_perm_fault). + * + * Also encode the level of the original translation in the SW bits of the leaf + * entry as a proxy for the span of that translation. This will be retrieved on + * TLB invalidation from the guest and used to limit the invalidation scope if a + * TTL hint or a range isn't provided. + */ +static void adjust_nested_fault_perms(struct kvm_s2_trans *nested, + enum kvm_pgtable_prot *prot, + bool *writable) +{ + *writable &= kvm_s2_trans_writable(nested); + if (!kvm_s2_trans_readable(nested)) + *prot &= ~KVM_PGTABLE_PROT_R; + + *prot |= kvm_encode_nested_level(nested); +} + static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, struct kvm_s2_trans *nested, struct kvm_memory_slot *memslot, unsigned long hva, bool fault_is_perm) { int ret = 0; - bool write_fault, writable, force_pte = false; + bool topup_memcache; + bool write_fault, writable; bool exec_fault, mte_allowed; bool device = false, vfio_allow_any_uc = false; unsigned long mmu_seq; @@ -1488,6 +1531,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, gfn_t gfn; kvm_pfn_t pfn; bool logging_active = memslot_is_logging(memslot); + bool force_pte = logging_active; long vma_pagesize, fault_granule; enum kvm_pgtable_prot prot = KVM_PGTABLE_PROT_R; struct kvm_pgtable *pgt; @@ -1505,28 +1549,16 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, return -EFAULT; } - if (!is_protected_kvm_enabled()) - memcache = &vcpu->arch.mmu_page_cache; - else - memcache = &vcpu->arch.pkvm_memcache; - /* * Permission faults just need to update the existing leaf entry, * and so normally don't require allocations from the memcache. The * only exception to this is when dirty logging is enabled at runtime * and a write fault needs to collapse a block entry into a table. */ - if (!fault_is_perm || (logging_active && write_fault)) { - int min_pages = kvm_mmu_cache_min_pages(vcpu->arch.hw_mmu); - - if (!is_protected_kvm_enabled()) - ret = kvm_mmu_topup_memory_cache(memcache, min_pages); - else - ret = topup_hyp_memcache(memcache, min_pages); - - if (ret) - return ret; - } + topup_memcache = !fault_is_perm || (logging_active && write_fault); + ret = prepare_mmu_memcache(vcpu, topup_memcache, &memcache); + if (ret) + return ret; /* * Let's check if we will get back a huge page backed by hugetlbfs, or @@ -1540,16 +1572,10 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, return -EFAULT; } - /* - * logging_active is guaranteed to never be true for VM_PFNMAP - * memslots. - */ - if (logging_active) { - force_pte = true; + if (force_pte) vma_shift = PAGE_SHIFT; - } else { + else vma_shift = get_vma_page_shift(vma, hva); - } switch (vma_shift) { #ifndef __PAGETABLE_PMD_FOLDED @@ -1601,7 +1627,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, max_map_size = PAGE_SIZE; force_pte = (max_map_size == PAGE_SIZE); - vma_pagesize = min(vma_pagesize, (long)max_map_size); + vma_pagesize = min_t(long, vma_pagesize, max_map_size); } /* @@ -1630,7 +1656,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, * Rely on mmap_read_unlock() for an implicit smp_rmb(), which pairs * with the smp_wmb() in kvm_mmu_invalidate_end(). */ - mmu_seq = vcpu->kvm->mmu_invalidate_seq; + mmu_seq = kvm->mmu_invalidate_seq; mmap_read_unlock(current->mm); pfn = __kvm_faultin_pfn(memslot, gfn, write_fault ? FOLL_WRITE : 0, @@ -1665,24 +1691,8 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, if (exec_fault && device) return -ENOEXEC; - /* - * Potentially reduce shadow S2 permissions to match the guest's own - * S2. For exec faults, we'd only reach this point if the guest - * actually allowed it (see kvm_s2_handle_perm_fault). - * - * Also encode the level of the original translation in the SW bits - * of the leaf entry as a proxy for the span of that translation. - * This will be retrieved on TLB invalidation from the guest and - * used to limit the invalidation scope if a TTL hint or a range - * isn't provided. - */ - if (nested) { - writable &= kvm_s2_trans_writable(nested); - if (!kvm_s2_trans_readable(nested)) - prot &= ~KVM_PGTABLE_PROT_R; - - prot |= kvm_encode_nested_level(nested); - } + if (nested) + adjust_nested_fault_perms(nested, &prot, &writable); kvm_fault_lock(kvm); pgt = vcpu->arch.hw_mmu->pgt; -- 2.50.0.rc0.642.g800a2b2222-goog