From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0808C77B75 for ; Tue, 23 May 2023 00:47:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 904C76B0072; Mon, 22 May 2023 20:47:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8B44D900003; Mon, 22 May 2023 20:47:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 72E0A900002; Mon, 22 May 2023 20:47:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 63EF66B0072 for ; Mon, 22 May 2023 20:47:04 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 3B45F140562 for ; Tue, 23 May 2023 00:47:04 +0000 (UTC) X-FDA: 80819680368.04.4F604B7 Received: from NAM10-BN7-obe.outbound.protection.outlook.com (mail-bn7nam10on2068.outbound.protection.outlook.com [40.107.92.68]) by imf13.hostedemail.com (Postfix) with ESMTP id 6CFF520007 for ; Tue, 23 May 2023 00:47:01 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=XUjeer1o; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com; spf=pass (imf13.hostedemail.com: domain of apopple@nvidia.com designates 40.107.92.68 as permitted sender) smtp.mailfrom=apopple@nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1684802821; a=rsa-sha256; cv=pass; b=D0ixxYCGTdkaQtepAhPqC8R8dObWESAMj1jGATwnjEoPdIJeo70KLfYnsUvBv/ogNLMHnp Uozq3XAnipYZ3xGLiYr/K7+SNMQdF/DGqjnR1y4CdU+muEZpjrKGwsd+9s7QrEmsn9uF0M z/ao2ZFaWtlKb6ekP3hpVjv69fRnpLc= ARC-Authentication-Results: i=2; imf13.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=XUjeer1o; arc=pass ("microsoft.com:s=arcselector9901:i=1"); dmarc=pass (policy=reject) header.from=nvidia.com; spf=pass (imf13.hostedemail.com: domain of apopple@nvidia.com designates 40.107.92.68 as permitted sender) smtp.mailfrom=apopple@nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684802821; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AllstN60M7xRCDeVJkT6HoMgkly4ZZP75yVXjI32Qw0=; b=P6bp0bYWMXtIk6HuQeVM7f0ZmqPzvfNaF5GFKo9+/qPfG/OE7LD0KT5GPqI28iUp96tGOR UAxG8z06RREd2xmdgP4s7MmCDAWcufZ9P5taXz7lzGXXtLXdF3kQ8jigBAzxI6J2jlMfLh +5/ExuefGDOyS/WOYo3o39fgpQZFxuk= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=C98pppIbG0Q+JJ1FEjXeP9qVZR1cPYCj4HjrU8n3f0BU3rP/dLmQ1bP7Z1w2J1DEfRGI2pfY0jcsmOqQsmV1ILlefFo8AAZbYHSElPwItNuEQIC+wypt4OTb8PCgoNO9F6mekWG9AAupBlLZvFYX9jxPhh4id6Yl7hWtiFewT8Yz5BJbsM10K8Hw2NLnDJNUpLfi2Xr7GNNj4Nt5qctAm19KtrY7cFdBK8n/ePipNaSCczgrvTwWNdkS6WvnuIrX044PrphJNbtWOjurKuoAGVF9NPsAa2ovwrJuXCJhX1F1KyY9+fqDtxfrOnuFOMM/cFl8pdM7lFIgLoEdRhW7nA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=AllstN60M7xRCDeVJkT6HoMgkly4ZZP75yVXjI32Qw0=; b=jap69jOSI7L1YGkYtdYoYYC0fxN+cFEM3zqOd9xJr3frcbZ0OtjZbA4RlTYnMwNZx6uRskYBMRPKCVQrqwDCavA9K11Xdoiyjq+7upQUOxy8Au2RSXGtGgo8ZlGvtO+WJXv3wm+WxzV5yY7bXLHeCMhbhcyaOkzQ1ifwxtqLcfqGei82pA4S5qAd/4m4CK7ALqralYItAX5/LnDE0ygr5fCkJpKmpcQ2Fj7Lb5kIDwqQZGbaqW3ReKj7pqpSqohg2HfH+x2j7SwnyWgzJFSh7wj2r3nwPTebxmqHplaL0yRKvgU3WxlkDzONEpV5qgJZqJtN8LYtclD3KbRJKCZZlw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AllstN60M7xRCDeVJkT6HoMgkly4ZZP75yVXjI32Qw0=; b=XUjeer1omxaMkg/8wP3ABtl2HNV4FomOLhcCWtRRgf0wnGRHx14dPX3AP9zEWOgaxb0lfZqazlCezmhqJQvXeAs+2Z6lhJ8aapajEa8AQWa6vfRBxdmz0rA8FZU2mNnHtDPh1/jHg9602XVzUUwuOMs13SrSvUEvzLwtM4cavgDl7UZeLAEes0noJ0fJGwWHPbbVSAlxP2mMltcQevdYV2uBRlRsd55SJao6oVBxC7/DXVVAIgXLEZnYD3gxMSgjBxsu0PMeFn0c2TcIg13ecYapV26AevQU7cjpKCOtepmduphW4yHYkLwO4XT1JipJKt6W31y3KLXJHao3t6/yPQ== Received: from BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) by DM6PR12MB4315.namprd12.prod.outlook.com (2603:10b6:5:223::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6411.28; Tue, 23 May 2023 00:46:58 +0000 Received: from BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::f6e4:71a5:4998:e6b2]) by BYAPR12MB3176.namprd12.prod.outlook.com ([fe80::f6e4:71a5:4998:e6b2%5]) with mapi id 15.20.6411.028; Tue, 23 May 2023 00:46:58 +0000 References: <20230522063725.284686-1-apopple@nvidia.com> <875y8jappz.fsf@nvidia.com> User-agent: mu4e 1.8.10; emacs 28.2 From: Alistair Popple To: Sean Christopherson Cc: Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, robin.murphy@arm.com, will@kernel.org, nicolinc@nvidia.com, linux-arm-kernel@lists.infradead.org, kvm@vger.kernel.org, jgg@nvidia.com, John Hubbard Subject: Re: [PATCH] mmu_notifiers: Notify on pte permission upgrades Date: Tue, 23 May 2023 10:43:17 +1000 In-reply-to: Message-ID: <87wn0z98n7.fsf@nvidia.com> Content-Type: text/plain X-ClientProxiedBy: SYXPR01CA0143.ausprd01.prod.outlook.com (2603:10c6:0:30::28) To BYAPR12MB3176.namprd12.prod.outlook.com (2603:10b6:a03:134::26) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: BYAPR12MB3176:EE_|DM6PR12MB4315:EE_ X-MS-Office365-Filtering-Correlation-Id: adc92530-b127-482b-e3cd-08db5b2736bb X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Km1COjutpSICGrn1oQM5/e1e78utryBl+xvlD8sUslh0MhYHjrJ8Kp3HKRovwF/CoWLEMFZinzwFezpqCrjPCAsSuxX/au1luxtNsTGQ4jbUBYfzEkVz22xLsc7Fq1weribvJaRgzOO94gJ0CnqYT/YPlfcxqz0kCNY71EPv5kiLlMJqLnfCdpC6qRasYPmRO3Qwx1hvwW/baOHaK51YwJA8N0e+9MZcQTENOEQn2t7LxbtA5PRe2WRYpLRk8jsnnBuJs6Ti/1UMdL1YRuY1RVyEclfwAbA5LvDWBQA32mKUydzOagYNIo1ajudtig36kYuPerEy1rHEWCuDIQi+KONU+K+tqakZlhKwrY5QUdvD21ZamSlfC9ztQSjIprm5K7Er+BW0P9wsXMf6NAg4pFEpCoCo8t0NaE0ILxRuwgG+dlcEpH21QBECgYxaI6SO+4v2qwBEg3eHZuYtRgSBTl1JSpUtqPNUUbewYyRcLzdYyhpqCDWUjrQyj5ZmoXoZZjD6ISEad6t3rAVyer9pP9QQhb18RDyEDkUVbNygKmv5qizPw6z5s7uovvh2oZwsVgI2sy3Wq00MMDosAdtk5JYofG/f/agVJxHxOx3sOtY= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR12MB3176.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230028)(4636009)(396003)(376002)(136003)(39860400002)(346002)(366004)(451199021)(8936002)(8676002)(5660300002)(83380400001)(26005)(6506007)(6512007)(107886003)(86362001)(2616005)(186003)(38100700002)(966005)(478600001)(6666004)(6486002)(66946007)(316002)(6916009)(4326008)(36756003)(41300700001)(66476007)(66556008)(54906003)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?a72ko43rOAQRKlaMX4JoSWjHYz3e2NmF5rmEkiLmDToi82XjjjPYCQidqtQ3?= =?us-ascii?Q?vKXhPvj6W9sboj0hS9tcCY+Z9gjIS/l9p8DQ+7Bh62kTRYQckHnH5RjmLFnl?= =?us-ascii?Q?S1UyLq+hrPLYp3UQeqePwKeYZ1V/lJrAwLvKimNYDMdxCg75yj0xjTWOkRmw?= =?us-ascii?Q?oeqD9tlxOywxrecMdUTtatgTmQm23OU9saOI2U/t39nwcpbBSCaFicnxL0fv?= =?us-ascii?Q?2UOGtlUHkK2wo8suqMlUdL/eGM/dNJXqIil8Z8htYOBhOLUs+Gi8VympLnNe?= =?us-ascii?Q?NnA1nwBEJzXFcnpCEQJ7EOEIPpPR5rNJxCYbJciZN2iyii7XZqyhXMi15f/B?= =?us-ascii?Q?RcnjYWlqkNCEwioC+7iaQsMvqSmVxknnKtMrBi6wXzGFXe3xkRiW6PLV9JRY?= =?us-ascii?Q?+g7NcfCp1J/FuguF/iAyaKtG1PZ4QiHYDBTb4+KCqxM/ul2klL4Uneo2pkfG?= =?us-ascii?Q?XECDQGL2vvmjRXDgXDOtSN+B8bnBrGxjG4UICMW2vSo9qZ1Q6d0dsGyUxdwa?= =?us-ascii?Q?sAWWIxDwW/CFzO+/jD3sZBNMD7ySi9etjQ1+N/HkQlR+HIqBdMGZsAxuJ7LY?= =?us-ascii?Q?HiGmOarqoTcjGDYkkDBlO4b095PW3SuLJpp/F2xh08bqBuGLUkczPRZx+B/F?= =?us-ascii?Q?OBk9wsj8FxFz8Nol/hNvmzXeP4xp8uXTCGM7A0jcdQ1t0kfRg9DWvlEUUILM?= =?us-ascii?Q?bn09zeOCYH0vERXQnrFa3oQLpSUHHJPucuOgBufZCOYemiaTrCgDd2WVWVSa?= =?us-ascii?Q?j6Qrh33vMAFOIpUMtPETA/2wBOYetPqdJEXLPYIDYWaKkBF9tiEG7BM0FGmX?= =?us-ascii?Q?XVV4tW274n0opZodT4Ky70G9CBDToI3YtIetreT0Q0N/Ut9lDwcZWd35eyqe?= =?us-ascii?Q?LPKaRN1iUuGr7UAlKVqiLZRnSjRh7d8ZD01hnTlyzYd6Ck83HTO9JaJW2l8N?= =?us-ascii?Q?w4WmpEmDXevBCzNSl3FyaxPsVbdX2QykKSaYRS7CGJLQCsl1lRx3gWEuCY8e?= =?us-ascii?Q?HcFfaalTCfDYkR6I7Yr1+vVj3I+jvrjWRi2WbRNGB9bHyQXaPvktyNDFVj87?= =?us-ascii?Q?uEExGbqb0lnditAgqe60feeTghT5inO4rTjVskCOKo5QhYn9Ba96O/oxazHh?= =?us-ascii?Q?a3DojUpi3IfRUb4Ftv7eKsrVl/n7qsZ7GmgqUeeTvkjPsrI0yXsaHUGJ14Lt?= =?us-ascii?Q?qXkbCGodWLmorADMjG780Gd93lsfLCVVSVc7ZK/kmMf8uSPdM9hkm/JuNeHc?= =?us-ascii?Q?/LOGUO1bsW177wFxzUWqY5qjiaO9ioY1dHiJjbcIpNCGboqE5Vq3F53gTzFl?= =?us-ascii?Q?c6Z18G4aHhfxm11iidkj+OXzGC3N0rDPj1qSPF7Wp5u2JK8lHhQzTfZigdIS?= =?us-ascii?Q?u/Xn5DJXo6q157l6wSxKOcpmtfvjosoJb8vDyXzW+3d0bfq+qC1A/WPcx4oH?= =?us-ascii?Q?rG0LMGCeu1tpXyYw/usYAEsnsAyA4Ais5JJ6v2fxSKyD/5m6w9L2bjn0UmJf?= =?us-ascii?Q?R1SKLRO/IrnieuiApER8GAXfIqCKCnTi5mrpdELR8HNnW/veYHl0Qw/VyNjl?= =?us-ascii?Q?92WXKxJydgq1IjqZG9bfNb038+j3/HVXfC0wJOKL?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: adc92530-b127-482b-e3cd-08db5b2736bb X-MS-Exchange-CrossTenant-AuthSource: BYAPR12MB3176.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 23 May 2023 00:46:58.4789 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: QKdVTTII3pk7X71LNbu7uEAIRVAyjCbk7m5pMjdVqOe/G+aOhITSpDF4Pxxois1CepTyuT89DIxbprlSEOCWDg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB4315 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 6CFF520007 X-Stat-Signature: 7pfkcxb1pexbp9p4tdj17huj94zz6it4 X-HE-Tag: 1684802821-423526 X-HE-Meta: U2FsdGVkX1+GvbbdulBZZdJpbuYq6iwTipVMxg6xGiU1DwWbzPbRFGPZIeLOwSOjeUKs6N6snxhPlB0mSxL7C4V1AF2H5v7g6Vyng/GEdct74FGDzne70RyqQAFQ9rZkAJu0xjQL4dFuCupQedQR6hGjzxchESR788/ORFQR2BIKXgPmQXNdZOSlOfhvuc9CL7U9JM1kEtuPsA9RdyVv6jd4G1txVSg14pT51j+jVV4M7FXxdUfvq4rwNA0H7KYgAa7ni1ueEEiP1Ygp7GLu+3cIrLPnL9BfsG+8a+duVldjSNhtbV4oLT0bc2sNzOqvDZ+Un32PDbyokrDyoAiCaetYINwoIIqnSCXYpfMGTeo7EBikpSCZpkGRQl7X7JZMIALeEkYQDfPmeOExafuDaShej/IuhmD9ORIzcRLcq76efz+sh2pJ56YqJ+V/8JUMrX0ips+YMwbtyVynWTyrKHbrIz2PKD/m+EcXfGZWlmFSjs1OxReumipAy/Yvy3QLI1JNpQ/xRtJLzD7ccXWrTQJJpXxljTIV3bAQ87/0BZ9xu3XppVr4ExtK9eWnzGT+wPTXvUHw9WFSFtuFiZBUHCxqeH5PUnP4Eh87wxwjrG9nCoB5fHGRS0O84VEqACWXKApdsxzdt/7Bw8S/ccGesLoOal4MT/kdFBhE8HEwhkng0LzK51p3mUFiNDyi6JU/kZUEqTUN1tv6VbdMGyMBTegrF9reaz72xy0fKK/exUsH/eLAA+uLIAZSjW9+0Ac/h2Q2kCDmal5GHjsf6rfnmT9aZR9eMuy6KpM49lG0CFHs4JQpIfedLtL0gdY+KZJJWdOjUtdm/fmfg/pFAs7KhKmyD4xB7YLV9vOICWEzhPcXcCBsepIFUJe4MXdqWjRc/Om/ivAAK3t0pu2U1NXnGn4pKB6FQRMEzsjAYu2inRXjncakM5hh9omwvq6gK4y6LRRUUQ1Lx0x2SnRiX/z lyfNEWJ4 irGCSUCnftoa4z00VwpWvo/N3KDGftLXJEacmV0z+kDZPhdcS10uLFpMhm6NZEPGddcgDP0hJJxbhe5ifZaVegccptXekO/F6ueXBenzBeuk25C2mzW2ylBCIOi3dvfnWa7LUN4vkI3PutMRMtVBUQvauxBO/dolJm+a2/Tw6KY6dxB0n1p6Pp8E36RXx3UwwEznOxlP1pG17z8wQO7I6PNomOxyRVwUPE34mn5U8h9SAyB0nPyY4GzjfaXj2ad5qfpto0/QSr1jkicJI0GU8sYdpoDMTlVvibuuv X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Sean Christopherson writes: > On Tue, May 23, 2023, Alistair Popple wrote: >> >> Sean Christopherson writes: [...] >> > * If invalidate_range() is used to manage a non-CPU TLB with >> > * shared page-tables, it not necessary to implement the >> > * invalidate_range_start()/end() notifiers, as >> > * invalidate_range() already catches the points in time when an >> > * external TLB range needs to be flushed. For more in depth >> > * discussion on this see Documentation/mm/mmu_notifier.rst >> > >> > Even worse, this change may silently regress performance for secondary MMUs that >> > haven't yet taken advantage of the event type, e.g. KVM will zap all of KVM's PTEs >> > in response to the upgrade, instead of waiting until the guest actually tries to >> > utilize the new protections. >> >> Yeah, I like the idea of introducing a >> ptep_set_access_flags_notify(). That way this won't regress performance >> on platforms that don't need it. Note this isn't a new feature but >> rather a bugfix. It's unclear to me why KVM on ARM hasn't already run >> into this issue, but I'm no KVM expert. Thanks for the feedback. > > KVM manages its own page tables and so does its own TLB invalidations as needed, > e.g. KVM can and does change KVM's stage-2 PTEs from read-only to read/write > irrespective of mmu_notifiers. I assume the SMMU issue arises only because the > SMMU is reusing the host kernel's (stage-1?) page tables. Argh, thanks. That makes sense. The SMMU issue arises because it is not snooping CPU TLB invalidations and therefore relies entirely on notifier callbacks to invalidate it's TLB. If it was snooping invalidations it would observe the TLB invalidation ARM64 does in ptep_set_access_flags()[1]. Now that I've figured out we can call invalidate_range() under the PTL I think I can just add the notifier call there. [1] - https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/arch/arm64/mm/fault.c?id=ae8373a5add4ea39f032563cf12a02946d1e3546#n229