From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65C5BC77B7A for ; Tue, 30 May 2023 13:44:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F17A86B0072; Tue, 30 May 2023 09:44:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EC81E900002; Tue, 30 May 2023 09:44:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DB72D6B0075; Tue, 30 May 2023 09:44:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id CD5D66B0072 for ; Tue, 30 May 2023 09:44:21 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 52B8AA02BF for ; Tue, 30 May 2023 13:44:21 +0000 (UTC) X-FDA: 80847040722.22.20987B5 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf20.hostedemail.com (Postfix) with ESMTP id 2663A1C0005 for ; Tue, 30 May 2023 13:44:18 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=none; spf=pass (imf20.hostedemail.com: domain of robin.murphy@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=robin.murphy@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685454259; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4CbK60LB2V5J70d4bXMY5wTRA6z84anIQ9Lr+cEPIrs=; b=cMzEEydNS6te8THjUfOb5ggKUtOu7WjbasHxvYmny51bb/S9v2E9Ba0eTQgwlNzuFuZ5Uy YWwuWkdhGWE25KcCOvg8wFQRoe2l+m5Sy7aSAd7EvZEJ9wpxG/qUPxXPJ6iWBOzf4nr7yK FniG24HTEB78/8Q0k/UOAAuBOwUONE0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685454259; a=rsa-sha256; cv=none; b=c+KfCaub40OPRitfUCUOiVtWBvYZhYIwf8gjDgl5N7jNGHUELGlVv8sGnUhWy6fPoVW+Ay 1PHMm7TH8dHS8EkkPVyHH5lgWvf9AEffcEnhxthaAAnDME6tvE+t1EWHFWtTQ97qd40M2z qJfEw2+tnskduis7MZnrJYRdyCXAV5Q= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=none; spf=pass (imf20.hostedemail.com: domain of robin.murphy@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=robin.murphy@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 42B4616F2; Tue, 30 May 2023 06:45:03 -0700 (PDT) Received: from [10.1.196.40] (e121345-lin.cambridge.arm.com [10.1.196.40]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6636A3F67D; Tue, 30 May 2023 06:44:16 -0700 (PDT) Message-ID: <89dba89c-cb49-f917-31e4-3eafd484f4b2@arm.com> Date: Tue, 30 May 2023 14:44:11 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux aarch64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Subject: Re: [PATCH 2/2] arm64: Notify on pte permission upgrades Content-Language: en-GB To: Jason Gunthorpe Cc: Alistair Popple , Andrew Morton , will@kernel.org, catalin.marinas@arm.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, nicolinc@nvidia.com, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, John Hubbard , zhi.wang.linux@gmail.com, Sean Christopherson References: <3cece716fc09724793aa832e755abfc9d70a8bb3.1684892404.git-series.apopple@nvidia.com> <5d8e1f752051173d2d1b5c3e14b54eb3506ed3ef.1684892404.git-series.apopple@nvidia.com> <87pm6ii6qi.fsf@nvidia.com> From: Robin Murphy In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 2663A1C0005 X-Rspam-User: X-Stat-Signature: t6t11dmtkxdq6856cnesowss5utxackg X-Rspamd-Server: rspam03 X-HE-Tag: 1685454258-797646 X-HE-Meta: U2FsdGVkX1/GpsKFXxDAA6OmKVaePo5D+fXrzEtEnlL9965jrLtHh1z1gAfqhKHy0ETWoJnzC2ps3kiE1ZCapLHWN8XmR8lrsfYDNi9IKdakMBYTf2tKfGtF1ucW7BE+0PXB8fqt/OL4LFSUCf0lAAKeQxqr7jIruX3v9XjGXy/RK4ugyBKCi5Q6CsBTVh14zA3IcqT6KedKhdLkTNVAo5NAcApVq0a5q0HHqxj82GVuP5JiNfTI5nnx2HgWM7w0hcFHiMUYIZqMfuwCu+8e0t97D+i0kA21AYiGH/s+HCT9hM/q4laYnnzIiP/44CNvTp2y6zQy1PH8qDyhknUWsAdwzBbXu/5E1fRUkWvyqidJzJq/wi0Prt6U8bPU1U8Iud/yOViO65AA72jJZRKtWv4i2MkmSOUwMqmhpLfR93d8JOGCTMpqymU5FgxsWVrRugvQYVhLcMPFyqityrJ0AhOhQGSSmE1vZUKnOxVf6uKb5w8xpqaOLeZTgdFgvxJQtTVbGJ805p6JGWjGq46pSA2xXMs/VPbKounXV0W+ZcJw+KJ3wwShUd2s2K9x2hE21QXCm2exGXRdwQnd8TzhzMv0wx8xCch1GCgUG5FG8y7tVUBO0mA69t4sJvMHRk32HI3DQxXKXicQ3RRk311Gj8q+IflMwGX93lcBwaeFloNDuq4fkFTeM2+0psUGmh+vmH9NWyKbF0TlWukmbKF8/zZGu5mfyttDFcGfD5zeKlEVPG+Hl4Bhm4FZbLd5/WjaWpu26ZPh05BD4EW7IbPJ1jRi15ofEwskk7ASrR1n/FImI8HBINAPKRjETGGwDjQ7REJYzTqmZx1TE+CraLUQyRfacJ9iBAn8gYyhgSECUnsPJKPjYH0f2uyilKPDRhI9cqRqSIz64QdlR8ILAIIr6rZ7sb8+YKj30ZdxPwmtGJGQm4uyq0I9TiimcGXWjmv5vXREVX5PXrg3e59AHnf QFFcYIKO HN2I4XgSZMmG14XxXDMK7SjZjLPlBjMUlXqf8tD2F8RjxhDA5Iz9i0FLtmnSzCOQzAs2NkwqcDSirgbRn3KhuYTedu2DywSQcoCHypXv7ol/SGU8Ctslt+loKrpxpoL0E8eavhdqbCadp1bW46bNy3+aRR02sH0IY8WZvIatjN/fDaMlYsPSxwgCZyQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 30/05/2023 1:52 pm, Jason Gunthorpe wrote: > On Tue, May 30, 2023 at 01:14:41PM +0100, Robin Murphy wrote: >> On 2023-05-30 12:54, Jason Gunthorpe wrote: >>> On Tue, May 30, 2023 at 06:05:41PM +1000, Alistair Popple wrote: >>>> >>>>>> As no notification is sent and the SMMU does not snoop TLB invalidates >>>>>> it will continue to return read-only entries to a device even though >>>>>> the CPU page table contains a writable entry. This leads to a >>>>>> continually faulting device and no way of handling the fault. >>>>> >>>>> Doesn't the fault generate a PRI/etc? If we get a PRI maybe we should >>>>> just have the iommu driver push an iotlb invalidation command before >>>>> it acks it? PRI is already really slow so I'm not sure a pipelined >>>>> invalidation is going to be a problem? Does the SMMU architecture >>>>> permit negative caching which would suggest we need it anyhow? >>>> >>>> Yes, SMMU architecture (which matches the ARM architecture in regards to >>>> TLB maintenance requirements) permits negative caching of some mapping >>>> attributes including the read-only attribute. Hence without the flushing >>>> we fault continuously. >>> >>> Sounds like a straight up SMMU bug, invalidate the cache after >>> resolving the PRI event. >> >> No, if the IOPF handler calls back into the mm layer to resolve the fault, >> and the mm layer issues an invalidation in the process of that which isn't >> propagated back to the SMMU (as it would be if BTM were in use), logically >> that's the mm layer's failing. The SMMU driver shouldn't have to issue extra >> mostly-redundant invalidations just because different CPU architectures have >> different idiosyncracies around caching of permissions. > > The mm has a definition for invalidate_range that does not include all > the invalidation points SMMU needs. This is difficult to sort out > because this is general purpose cross arch stuff. > > You are right that this is worth optimizing, but right now we have a > -rc bug that needs fixing and adding and extra SMMU invalidation is a > straightforward -rc friendly way to address it. Sure; to clarify, I'm not against the overall idea of putting a hack in the SMMU driver with a big comment that it is a hack to work around missing notifications under SVA, but it would not constitute an "SMMU bug" to not do that. SMMU is just another VMSAv8-compatible MMU - if, say, KVM or some other arm64 hypervisor driver wanted to do something funky with notifiers to shadow stage 1 permissions for some reason, it would presumably be equally affected. FWIW, the VT-d spec seems to suggest that invalidation on RO->RW is only optional if the requester supports recoverable page faults, so although there's no use-case for non-PRI-based SVA at the moment, there is some potential argument that the notifier issue generalises even to x86. Thanks, Robin.