From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D303C3DA6E for ; Mon, 8 Jan 2024 11:05:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DEBEC6B0072; Mon, 8 Jan 2024 06:05:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D9B7A6B0074; Mon, 8 Jan 2024 06:05:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C8B756B0075; Mon, 8 Jan 2024 06:05:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id BA2F26B0072 for ; Mon, 8 Jan 2024 06:05:04 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 90DAFA1261 for ; Mon, 8 Jan 2024 11:05:04 +0000 (UTC) X-FDA: 81655861728.13.755A448 Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by imf02.hostedemail.com (Postfix) with ESMTP id 84D4780013 for ; Mon, 8 Jan 2024 11:05:01 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of cmarinas@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=cmarinas@kernel.org; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=arm.com (policy=none) ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704711903; a=rsa-sha256; cv=none; b=74/KlObuU0kZ9iZlgBbRbxRWC2p/Wyfdh7ly4AT6sg96CpcAcGqC+MlNB99oTJNvs3V4VJ rt4aKZ3lt8k+AE1QnPrhTI8pxklFn/XU965qdUhaC4IYQXlnp1ZEU933pkqwCYpMbXj+nT MuGMm0UZLaoosIhXVnceNDjDXErNo5g= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; spf=pass (imf02.hostedemail.com: domain of cmarinas@kernel.org designates 145.40.73.55 as permitted sender) smtp.mailfrom=cmarinas@kernel.org; dmarc=fail reason="SPF not aligned (relaxed), No valid DKIM" header.from=arm.com (policy=none) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704711903; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ntNDtBQCHqA01Pn70YcZIVpF2U85mmzEeS6W4avtfYk=; b=d76QKmSv/Oc+15zt4aj5c15hbymtTn+nuhFyRxVRnBUKnvsTd+tk08ko0fquR+k+qYvW9Y MB2HhMHHWtHigYGyOtGiK8stUU+t9iJPtdGniMX8KUKaao3JiS+2yXsZ7SMPaT5kvkmNq5 1bEIm0UtJuNjt41SIjuByWfrQlITTZI= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sin.source.kernel.org (Postfix) with ESMTP id 4206ACE1011; Mon, 8 Jan 2024 11:04:57 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id CA6B7C433C7; Mon, 8 Jan 2024 11:04:49 +0000 (UTC) Date: Mon, 8 Jan 2024 11:04:47 +0000 From: Catalin Marinas To: Oliver Upton Cc: Lorenzo Pieralisi , Jason Gunthorpe , ankita@nvidia.com, maz@kernel.org, suzuki.poulose@arm.com, yuzenghui@huawei.com, will@kernel.org, alex.williamson@redhat.com, kevin.tian@intel.com, yi.l.liu@intel.com, ardb@kernel.org, akpm@linux-foundation.org, gshan@redhat.com, linux-mm@kvack.org, aniketa@nvidia.com, cjia@nvidia.com, kwankhede@nvidia.com, targupta@nvidia.com, vsethi@nvidia.com, acurrid@nvidia.com, apopple@nvidia.com, jhubbard@nvidia.com, danw@nvidia.com, mochs@nvidia.com, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com Subject: Re: [PATCH v3 2/2] kvm: arm64: set io memory s2 pte as normalnc for vfio pci devices Message-ID: References: <20231208164709.23101-1-ankita@nvidia.com> <20231208164709.23101-3-ankita@nvidia.com> <20231212181156.GO3014157@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 84D4780013 X-Stat-Signature: 9dbfsk9eg6hxk8d95z9sfor1qbuouab3 X-HE-Tag: 1704711901-418793 X-HE-Meta: U2FsdGVkX19Y+DNYuKDkq0bJCH+zlpu0VPPaUzEsuN9ODk2d3JOhPiCLpgK8DwEHG1ziLUTvoA9KHS67yHrkW29Bw6OMV1E00sP+6+Uu2rHWYX9JgWmnoy+O7n04Im1BitWU0Ku6LmNMHateE+ESRJvf4uRDWuhm5hR7dd35n7ES49kSLY10orZoUrei7EukjpnWh5ocKY8d+Fhq2dxuL9uCpudTsKD/QFwsSdZRecAayIPMwnIaJECC4/9YZ9XEUUPq9o9KKiziYBbJJ8AZw8137iyGb7/O3Sqnq0wXgISgKq2VSL6Y89CE16EhGZnzZiVv+Etb+Cpl/P98S43BBEAKRPF8vhxNPLgOI9AOxRz2gLBCJIvf0DBgnQCRaQkAv4L2THNYPYz6ivDiZO70C/CZCuueBzzr5APpKBpQ+VTdHLK3BH8Xc7oklkUVn+Mut1xm2/ADk2cbszDsjWvwSGRSSj8BCkxFATdZQAndOyUbXdT1iq9BOof1OB+5ZTQvpTj5QdUmVuSKdNnLKi5pw15AKWP9cdenZsrmEYPA0K1xgjMiDOpqsC1ERe7WY30pghHmBHa8rHLuphoU/zbK8l5w7xzCtdX3ovwC/W6uXopBSTOmDGFcZCZXwJeDvuVJNlk4BDOEQiXEtz8UUa1/TSmOC51dg38JGwU6dFvH27erbpVwBk1G2MFEX9Q1EvDAWpBqgAmTr66A9BZSSPoTpNSv1F7ChYzDwuN66JxAiyG1eTnspXMXFI8xZ5eXJRW7lX6Gi7v6069yfRHwUElA26Iiv0Qi+H4AG9RxoyFNIWnxfXo4n97jsYt47HIB4ZQ6SLPlL8KSgBnT7QQinY3HhgxBVSD1OJKRBzlfFyfZweyfKZQawVHZDDd4VLBT8dsdzJiBM7DiB8rMv27tGJmqB9RYaKmHo8M3zcttzS9MFOEyRKDA8CadBlfGNN/8e1Qz2qZj7LjH531a4SNQoY5 TQln+J0U EjWvO/62Jy9Nubp/VXUGhdyK5irHLSy6Mta1BrkolOVn6adk0qQMnPupqWF3YSKFQB8BSP+Bzeb+B8tUc9EbxkbwQgEQ2zQ/syGW2lbemnqRxQbVoEdjP2Nluqp/AMEa4PANXR8t5PYrSTQBcwthqa8lc2kPuk4x4xIVOMZzeYN2A/fcp/vB8OkyNnynmxZm9Esub5jWIqvuCAtNsMWiZdtvq4vrfxjzDirt8c3FsqnD2BceZg3VA+dIb+4NmKaGqeJ4/v7oGedG8baY/jHzeo2gQv0t/ru5gJj1p+YFDE9MLjxwoF8UkriKuihw5+LgT7Mf5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Jan 05, 2024 at 08:42:31PM +0000, Oliver Upton wrote: > On Thu, Dec 21, 2023 at 01:19:18PM +0000, Catalin Marinas wrote: > > > Apologies, I didn't mean to question what's going on here from the > > > hardware POV. My concern was more from the kernel + user interfaces POV, > > > this all seems to work (specifically for PCI) by maintaining an > > > intentional mismatch between the VFIO stage-1 and KVM stage-2 mappings. > > > > If you stare at it long enough, the mismatch starts to look fine ;). > > Even if you have the VFIO stage 1 Normal NC, KVM stage 2 Normal NC, you > > can still have the guest setting stage 1 to Device and introduce an > > architectural mismatch. These aliases have some bad reputation but the > > behaviour is constrained architecturally. > > > > IMHO we should move on from this attribute mismatch since we can't fully > > solve it anyway and focus instead on what the device, system can > > tolerate, who's responsible for deciding which MMIO ranges can be mapped > > as Normal NC. > > Fair enough :) The other slightly unsavory part is that we're baking > the mapping policy into KVM. I'd prefer it if this policy were kept in > userspace somehow, but there's no actual usecase for userspace selecting > memory attributes at this point. If by policy you mean who's deciding the write-combining relaxation, this series moved it to the vfio-pci host driver. KVM only picks the appropriate memory type for stage 2 based on the vma flags. That's Normal NC in the absence of anything better on arm64 and it does more than just write-combining but we can describe what this new VM_* flag allows. If we want to keep this decision strictly in user space, we can do it with some ioctl(). The downside is that the host kernel now puts more trust in the user VMM, so my preference would be to keep this in the vfio driver. Or we can do both, vfio-pci allows the relaxation, the VMM tells KVM to go for a more relaxed stage 2 via an ioctl(). -- Catalin