From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB662C83F07 for ; Mon, 7 Jul 2025 01:01:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4DA958D0003; Sun, 6 Jul 2025 21:01:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 48B088D0002; Sun, 6 Jul 2025 21:01:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3A0938D0003; Sun, 6 Jul 2025 21:01:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 285A38D0002 for ; Sun, 6 Jul 2025 21:01:07 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B5F2D1D875A for ; Mon, 7 Jul 2025 01:01:06 +0000 (UTC) X-FDA: 83635664532.25.E3B4389 Received: from tor.source.kernel.org (tor.source.kernel.org [172.105.4.254]) by imf02.hostedemail.com (Postfix) with ESMTP id 405AB80004 for ; Mon, 7 Jul 2025 01:01:05 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of cmarinas@kernel.org designates 172.105.4.254 as permitted sender) smtp.mailfrom=cmarinas@kernel.org; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=arm.com (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1751850065; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9QUVNe7L9edGqcKaeMqOMgtNSGUzYbwGQwHgagEyTzg=; b=d7HdV9nIA+GUpzBexw8SVn/6iWvPSQgJiLoBr/fxaUSCVW8f44R1+htzCbVkXSkD7ecaQQ GilBFpHy/fIERO68gju1hH2UjeiY3E+NdH2bo9OJU0Xf9KaIZDpvCFGCiiUuZYX+6IYrvB 7qL3J43FpIDRgGpyCe8p0voSzC5RzVw= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of cmarinas@kernel.org designates 172.105.4.254 as permitted sender) smtp.mailfrom=cmarinas@kernel.org; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=arm.com (policy=none) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1751850065; a=rsa-sha256; cv=none; b=xC9aUUqnCW1dnioq5DUkxWXaaJE/NSIbzxTbWOjNCORJqlOEN/3UHpYRdZf3vO1tCLGahH P2PyhSpjuRijCFBHjEaHiJ8SKujiLB6iCNd5AEDfZO01XAattv+4fUsjvk1Bpo3uVbbals U+lNZkz5/wTU652qgN4uCgDKJ92ulME= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by tor.source.kernel.org (Postfix) with ESMTP id 9846E6112A; Mon, 7 Jul 2025 01:01:04 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A5C44C4CEED; Mon, 7 Jul 2025 01:00:28 +0000 (UTC) Date: Sun, 6 Jul 2025 20:00:27 -0500 From: Catalin Marinas To: ankita@nvidia.com Cc: jgg@nvidia.com, maz@kernel.org, oliver.upton@linux.dev, joey.gouly@arm.com, suzuki.poulose@arm.com, yuzenghui@huawei.com, will@kernel.org, ryan.roberts@arm.com, shahuang@redhat.com, lpieralisi@kernel.org, david@redhat.com, ddutile@redhat.com, seanjc@google.com, aniketa@nvidia.com, cjia@nvidia.com, kwankhede@nvidia.com, kjaju@nvidia.com, targupta@nvidia.com, vsethi@nvidia.com, acurrid@nvidia.com, apopple@nvidia.com, jhubbard@nvidia.com, danw@nvidia.com, zhiw@nvidia.com, mochs@nvidia.com, udhoke@nvidia.com, dnigam@nvidia.com, alex.williamson@redhat.com, sebastianene@google.com, coltonlewis@google.com, kevin.tian@intel.com, yi.l.liu@intel.com, ardb@kernel.org, akpm@linux-foundation.org, gshan@redhat.com, linux-mm@kvack.org, tabba@google.com, qperret@google.com, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, maobibo@loongson.cn Subject: Re: [PATCH v10 5/6] KVM: arm64: Allow cacheable stage 2 mapping using VMA flags Message-ID: References: <20250705071717.5062-1-ankita@nvidia.com> <20250705071717.5062-6-ankita@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20250705071717.5062-6-ankita@nvidia.com> X-Rspam-User: X-Rspamd-Queue-Id: 405AB80004 X-Rspamd-Server: rspam09 X-Stat-Signature: 7sbzd81u1xpjy74btf1zjk7t1i514t16 X-HE-Tag: 1751850065-918050 X-HE-Meta: U2FsdGVkX1+8HiKlffCI3GzxDQU4LXp0xeTga2mfOXok6o7xYN2J3DQNfdutn/goW2w60/mn24ATIm2+omAGKmmVcc0MIYK7E8Iihi3reMjV675weBOSMyg28Rby7tDqjO12+BblaZxjB4FjnVqsqZOfMiNIggTuJCm/fGlsXvni/u474EIkUprguoDy/2SChrR54Y21vFPuEbvfCnN056FnZhCcC6S1eObtGrKa3dPz7irVzynQ3wMDni2YqbHB/jRbDDT2GYE0NGbrGV+LmZIETIBUkOP0MLNta7N0TU8Myh4kF3YNfH1gtA30ui+JhsXheeUIemMZe5yQFHUlAONoqonwh4/ksZkYc4/8V/EEQ66mWmf8biWNYTqQGaGiZCswaSayjmBuhb3V8cAzsKzy6MdqbueuFpJIodO07TZLLTdQ9J4LZ4mCSN2uao3LZvHyMF4sLOqlATV1s6ZlwESBOchM0Cgu9meN4eaCHMBAzeSR/Kh+yJr+n61sosjyQXKzcIO31DxD6SfsWvgEH20M8Qr3+UhvBkTtWiMnbe7OvCiv37m8ZB/ok0PVEpKGXwa2lJcCX61jYj/tNj/TTkOiKrzoZfP5dHDMFVKskwE7AV4M0WmjTsM3LlAxJRIQEzWgtAfVeSTCn4v6fIpVpsIHhSdXy9+s+rfkau3zv/uFHd0YqZnnsNwheYlKIuPnpBR6j6sRHpWupAD01n5vQJM5vDN9kl9wrnkVaakgCpCy5riwt6b/Zcf5L7mnDgUAMFjCWxA+m+30VfQTdU/XGj4RNYfd9ZbY4wMmi1TJ1AfWLAlE3J+xHZb4ZUo06Lj/i848TT0sHy7NTilhnq7d/Wf9uw22xM9M9V3ngZq0hhkvDhCkvYj7A0zhxdTJUZgZ9b3QYRjCynirdK/zVFLNSNZSQivKOOpp7RZqNdZb+wz7NZ9N67YsDhFYGS5PcaJe96lZRxtrD8Vo6wPj/RR O81Ov5nN bwtUBX3aTDwq6h7ASa3cCgo81+rR2mwYty/uDt8/4tQ5vGZ9Toy1M2hj873mqTaE7KX+cmyMwi2xe4qbUf6JO3aqxc8l6HSWPB9SLc9RBC3BViWx+cNl83XBlWrBplsJrE6Sp5j+6mrCTwCxoBZA2G6VeEOyOjofEvEMI0Xo5DPPKa8nCNxV0QykhDNSWsDWg+64mCaqQdXHe1Qty3nK/Y7pSEeex4BwzSlf2bXwBV2L0iqWNn2fofQSGJaowdqnUBEf5RhoihMvrVQedjcfRXKTe+FT+D/YtE7QuAaRhAg3Oqnt/lDE7N0+4Wl6yvXXcuv6RnbOt1FqxfKD3lnarvMKUMVIYgyZ2IPHmj9EQssHCyVKqSH89dVQrR088c0x0fQc6Dxb3aGHTaGHt2KQqw7qRc3LtOcfQzi+Xa1UYJDBwF7adjg0krpSqXg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Sat, Jul 05, 2025 at 07:17:16AM +0000, ankita@nvidia.com wrote: > From: Ankit Agrawal > > Today KVM forces the memory to either NORMAL or DEVICE_nGnRE > based on pfn_is_map_memory (which tracks whether the device memory > is in the kernel map) and ignores the per-VMA flags that indicates the > memory attributes. The KVM code is thus restrictive and allows only for > the memory that is added to the kernel to be marked as cacheable. > > The device memory such as on the Grace Hopper/Blackwell systems > is interchangeable with DDR memory and retains properties such as > cacheability, unaligned accesses, atomics and handling of executable > faults. This requires the device memory to be mapped as NORMAL in > stage-2. > > Given that the GPU device memory is not added to the kernel (but is rather > VMA mapped through remap_pfn_range() in nvgrace-gpu module which sets > VM_PFNMAP), pfn_is_map_memory() is false and thus KVM prevents such memory > to be mapped Normal cacheable. The patch aims to solve this use case. > > Note when FWB is not enabled, the kernel expects to trivially do > cache management by flushing the memory by linearly converting a > kvm_pte to phys_addr to a KVA, see kvm_flush_dcache_to_poc(). The > cache management thus relies on memory being mapped. Moreover > ARM64_HAS_CACHE_DIC CPU cap allows KVM to avoid flushing the icache > and turns icache_inval_pou() into a NOP. These two capabilities > are thus a requirement of the cacheable PFNMAP feature. Make use of > kvm_arch_supports_cacheable_pfnmap() to check them. > > A cachebility check is made by consulting the VMA pgprot value. > If the pgprot mapping type is cacheable, it is safe to be mapped S2 > cacheable as the KVM S2 will have the same Normal memory type as the > VMA has in the S1 and KVM has no additional responsibility for safety. > Checking pgprot as NORMAL is thus a KVM sanity check. > > No additional checks for MTE are needed as kvm_arch_prepare_memory_region() > already tests it at an early stage during memslot creation. There would > not even be a fault if the memslot is not created. > > CC: Oliver Upton > CC: Sean Christopherson > Suggested-by: Jason Gunthorpe > Suggested-by: Catalin Marinas > Suggested-by: David Hildenbrand > Tested-by: Donald Dutile > Signed-off-by: Ankit Agrawal Reviewed-by: Catalin Marinas