From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E8C4C46CD2 for ; Tue, 2 Jan 2024 17:09:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 24D716B017D; Tue, 2 Jan 2024 12:09:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1D6CE6B017F; Tue, 2 Jan 2024 12:09:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 028686B0180; Tue, 2 Jan 2024 12:09:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E1E4D6B017D for ; Tue, 2 Jan 2024 12:09:15 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id B907DA193E for ; Tue, 2 Jan 2024 17:09:15 +0000 (UTC) X-FDA: 81635006670.20.39541C5 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2053.outbound.protection.outlook.com [40.107.100.53]) by imf25.hostedemail.com (Postfix) with ESMTP id EFAF6A0026 for ; Tue, 2 Jan 2024 17:09:12 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=WJJEiEDU; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf25.hostedemail.com: domain of jgg@nvidia.com designates 40.107.100.53 as permitted sender) smtp.mailfrom=jgg@nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704215353; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SaeMPThzjsXQhyQhHTjT5LOmTU89XdTuetaU7ll8RpI=; b=2Q7wGuP7HV7heLWLVIFpl31HXuPlTb7cTJRmfLqrIQwzyaWubWsqwc/+wHkv6O1/hZ4vhN 894jfLvcdDs7EqlnxPk3YRU5CRO+Ywlh3RwUofE2xD8HU/zxDnNHrwGWVfhEDP7vkncdw7 pYdOzfIMOLUw7BNZ3DxfoJGKGRGAphM= ARC-Authentication-Results: i=2; imf25.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=WJJEiEDU; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (imf25.hostedemail.com: domain of jgg@nvidia.com designates 40.107.100.53 as permitted sender) smtp.mailfrom=jgg@nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1704215353; a=rsa-sha256; cv=pass; b=j6tkKy98iyGPurdayAxEIK2sVSiYaUFpgOJ3k8Z6Vh3rZqbECe8+jp5dXZwzqh0D+TO9Ky 3A4Ckgt3bzI/TECp3Bb7P6H238n2GVfr+6QtO9G7RQtdOj3XdqV8ZJ+MOncBt0Ad54OnXD MCDxMvtU5Bd16pBsaH3dzFaG7qudMM4= ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GGDmDpWdgFD2KzlgkcvBrwzvr1Euuc4hu+R2CbfsH+HlY6ZxkUuuXyMwXKBtlf0pdU8qBcHN1GtphwZsHG4waESzlg5JPjndg3DS1T27SdYt7i5kKBdh2F1isUODJOCkVSNC1griSvA+h6yDs375PKrEpvoI0XCSQKMhLgRlo1EENbQuC4qFzheQSNP9tanGgk8IqsawX3uBhGAUoN6EPJRKdBwvstgxJx+WwzJq4VqYMhE0f7qbMsrtGTDI7IYNceUTUyQqMbIzaMm9zXkAl/mdoda99EOEdCNXQ1laIbPI1hdVXifrtL6/1irP0YpwsJHIV2rBBNt0Ypr/CaQHgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=SaeMPThzjsXQhyQhHTjT5LOmTU89XdTuetaU7ll8RpI=; b=BbSDSHY5tIjCrS0qFuLqeax2+lPx4hLkWFgD6C+2Bx+IaBqfwBl8OsQ0FO4YlC4/GsM39GzT9I/SlDEmF7YBuPbb1KWLvJtId+EQipbfsZMnx6CPd/clUP9iaoRq7misvfdi6Q6sOLNo3Hkfbs2AnS++CNiaCT3J/UDWft1IL9UAymhjqB6Gm8TtNzntQVSmcRnzD36Dw/7dINVaQo2P0wOLrKxRZ5rrDBHrQnqrLdobu9Nms4Jozh5qCMmdCdzhmsUwo4YUyn52k6lFGMYPOFmo631zmn/Hcai1tfU5qK4MrQdfo+Bvu6Xlzw50c3wVbJ74Ct/BJKAHVj+9FtV2vQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=SaeMPThzjsXQhyQhHTjT5LOmTU89XdTuetaU7ll8RpI=; b=WJJEiEDUK9E7Jmkx516Zd9AsXpdmHgN6biKcgm4trgR9ogEnUESFpSWo5p45mod0dTbhtp0M0pi1kDBTIZy3IcQXnSrsI3u/vB5Bvj1chjTVBYEo8d7PQCNkbJy/5w8N4MFAyZa3qADjV3+aKZ5ICNL9+zTgGUPsHBNAP4bKOvR6dR1Bi+jWZOtjVGFdlfZ79TrJ3vM/fmudHubvkorJvEVKXNVQNGT+z7EYA7b0mWwMtsduZp5lq3aZsgJZ2VhiaJ8dYBHhPe7s4jPXqh018QLiMEgRNiFKSf9nkmC6qsoSslVf/6Za9/r3xdwRBamQD7ZgbPlZn82AZDSx9kI91w== Received: from LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) by SA0PR12MB4512.namprd12.prod.outlook.com (2603:10b6:806:71::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7135.25; Tue, 2 Jan 2024 17:09:09 +0000 Received: from LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93]) by LV2PR12MB5869.namprd12.prod.outlook.com ([fe80::60d4:c1e3:e1aa:8f93%4]) with mapi id 15.20.7135.023; Tue, 2 Jan 2024 17:09:09 +0000 Date: Tue, 2 Jan 2024 13:09:08 -0400 From: Jason Gunthorpe To: Catalin Marinas Cc: Oliver Upton , Lorenzo Pieralisi , ankita@nvidia.com, maz@kernel.org, suzuki.poulose@arm.com, yuzenghui@huawei.com, will@kernel.org, alex.williamson@redhat.com, kevin.tian@intel.com, yi.l.liu@intel.com, ardb@kernel.org, akpm@linux-foundation.org, gshan@redhat.com, linux-mm@kvack.org, aniketa@nvidia.com, cjia@nvidia.com, kwankhede@nvidia.com, targupta@nvidia.com, vsethi@nvidia.com, acurrid@nvidia.com, apopple@nvidia.com, jhubbard@nvidia.com, danw@nvidia.com, mochs@nvidia.com, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, james.morse@arm.com Subject: Re: [PATCH v3 2/2] kvm: arm64: set io memory s2 pte as normalnc for vfio pci devices Message-ID: <20240102170908.GG50406@nvidia.com> References: <20231208164709.23101-1-ankita@nvidia.com> <20231208164709.23101-3-ankita@nvidia.com> <20231212181156.GO3014157@nvidia.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-ClientProxiedBy: MN2PR16CA0039.namprd16.prod.outlook.com (2603:10b6:208:234::8) To LV2PR12MB5869.namprd12.prod.outlook.com (2603:10b6:408:176::16) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: LV2PR12MB5869:EE_|SA0PR12MB4512:EE_ X-MS-Office365-Filtering-Correlation-Id: 6de4bb8a-1139-4cd6-a05a-08dc0bb588d3 X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: efzWhFQtMa+oMUpN0ruK2+54sA2yUI2AkVzpw1qDLAcfjLP7RyozdBm3xn4myDN2JtiGX0Iw6Oik9Zs02rbCPRuKHVDDKBj8fQcJ6VkTz+dgMraf6kPB8NkRMvNuJOYEPESd6oA7yqbSKGCUW583j3w2Rzyd7sKhfwlsG3pre5QUskLXjM0wKeMU6A83b3CiajVtEKkyDgp3Bb0IpPNdQ3bhdCc5tiwKGvhePsN4A1ejpzA7L0O00uhSnPrZkmf4LpNJF6aBnBlGs4yAnt6B5cAFQ7s5qx+PQeQJdPfaExytR0bFUO7eTyLKNgO3ZgIScSYxASC7YHbXiqZa+8PHBZMENaXoIAY99f9FUzjNK+wf4h3kztZ3Jphw2XLNJw3m1gZiL3m+lOLNDsZB4+my2EVwA6AXQ41gaJq3KqqEq/QH6L82Bk4P4rAhXihnJM3ijSF5j6Xh7ahRQuCZ51mh8FybNIM1sgdEEyiaHbrUgb54y8ZPcj4GrUdMd/4WB6VuxcX7j5akiboHzjiLpfvoIKbgXJk0S4Lr2RvpjjzzND+OrDjwGwgOjzOnO3APP85P X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:LV2PR12MB5869.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230031)(136003)(346002)(396003)(39860400002)(376002)(366004)(230922051799003)(1800799012)(451199024)(64100799003)(186009)(2616005)(83380400001)(1076003)(26005)(38100700002)(41300700001)(8936002)(8676002)(316002)(54906003)(5660300002)(2906002)(7416002)(4326008)(478600001)(66476007)(66556008)(66946007)(6506007)(6512007)(6916009)(6486002)(86362001)(33656002)(36756003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?TtBvKvohDDZNz0Vwtl3KxYxOSkw+Q7ZOm8HRc/PzlAqkgVkFMwjQE/VJ7tGz?= =?us-ascii?Q?QswMtmjNq1ai1rCMsTqQV03MkXqAcwMUMVRNijruYEDOYEm0pfD4YJLXUqxk?= =?us-ascii?Q?5KH/59vwRfvdDzM3gMU7BEG/vsAfDgX/RxhfFXa/wshGO1jqwmQnWoTVm3nl?= =?us-ascii?Q?QV0XAv4R9UMYdAtaaeCmKJXR9HC31bjEDKMttvezIoojYkqbqd4uQytbXJL0?= =?us-ascii?Q?sUVhkT+Lq4OLwsIRcrcsCc5Whjwk/RiFZ9doEXVj4jwdicAP8wEcEc6/Mfli?= =?us-ascii?Q?DH9qcinereDOiBuuN3FUJQ2izCO1PQQkFRvN4JmY6zFUvE61oKHdDpxfyPGx?= =?us-ascii?Q?4Rl76lKdcPfbsq9a9Pf0D3FrqW2gj9v81CJx608bEcRvxde70Sv5Y6s6Uaod?= =?us-ascii?Q?9JbjRTczBuppdYpbiCuROTsp7Bdwco5HLRFKdTReFsxzCnNGsJaSFx5SyMYy?= =?us-ascii?Q?bRSnW4+Uw5xWEnmB2cEn+DOiO0VyRWAzJ9EFpZLVFwPumGaABeh3ISVMnlUO?= =?us-ascii?Q?aIfFkIdYQcKwuss354ehL9ASulYX1EQXbWjE3VN15vw1nXXY30ApGDbWzk54?= =?us-ascii?Q?LuAH0GQVjtWiKhFm+ydBSwfI7g+L65TCUUYCCxWDfISJUnsrbn+tFTKamUvA?= =?us-ascii?Q?rJry/juZ1bwrGAIAsFWkNYEinPB/eXsRZxp2iYlFMu4c/vlA9kARSiRKt9jC?= =?us-ascii?Q?OAtGnUhGDlnhlFtEZ1k8oY4ltnaaHmaGVMMBQAoDec1S5MFK9JbdEp07jbLI?= =?us-ascii?Q?TqY8D/TslL+/16yKYFiwevK4hhR7my4uU7f7+L51Bx1gWdMTvTsakCseKIdG?= =?us-ascii?Q?8RJuIPnrI1GKp+NPN4aUsyLtISRtZKOLe84tMpPUFdsgrKXUdVkoRpfUZT7P?= =?us-ascii?Q?LNBwxFVX6dukyoUI1PPlJXo7rBK+YS9GftE8m5FS2SviC4BMVRkM6qFMyT7T?= =?us-ascii?Q?kO6f+d5vVJKgqcNsgosRHB/KK6ln/VdcH2epqAmurFiTMWBri32JieNhO9rB?= =?us-ascii?Q?JU2E+upnRt6lhfcQLeu1FvS91GzE+LAIXaTWbSHQHreaSpj8MEHq6gDiK4Ow?= =?us-ascii?Q?3Zn4u4uHuE6tyrSS/0D7ymOG1aaxxqv2BF7kHLq4I7nfdKuM//9JB0cjBNnV?= =?us-ascii?Q?5Z58BUdijP3freysxvrg0vgGRimwuXEEu6PPyNqFX2DFDzYLLut3bM9IIxgZ?= =?us-ascii?Q?iHnMCEGnG0J19gnrMDfEwyINAOlaNmrwgqoVLIyA0Tzjdrz9VO0WFWshwbcT?= =?us-ascii?Q?wTgk8xMzolSCmJNCHRZZYR8g4Dbah1xnouHYDPXet9Npcu+qnGcHpm68DTid?= =?us-ascii?Q?p3n44A9IZBCdjEtTOrrZErK3Bww0+5GsltXAPlIJSYXTb0ZABCiNiOT2ukJQ?= =?us-ascii?Q?zolN07osXhZoG2Ux6k+7z9BudPK/eicTEVHuDSOqxgS3oI1Zh1I7wnGLKpK7?= =?us-ascii?Q?DXEZ8sSY7Cj24m2+HpRm09/6qlzJccQ6gATd3yRMF81nnh/akbefrjN9j8Gv?= =?us-ascii?Q?XHe2UGh7JJ+jMr9omcs+tI0EOWMwQGjKIjm6TvgP0LAgYE2GYt+2HcHuTKmg?= =?us-ascii?Q?VPJihSB3WihC3MmR9zU=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 6de4bb8a-1139-4cd6-a05a-08dc0bb588d3 X-MS-Exchange-CrossTenant-AuthSource: LV2PR12MB5869.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Jan 2024 17:09:09.4941 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: jORWB1Olaraj4ndpLWDuiT3ZQC+XSvFD92Wa/Ta9VO5aFzBs4zR4GIk+9i7oTwzq X-MS-Exchange-Transport-CrossTenantHeadersStamped: SA0PR12MB4512 X-Rspam-User: X-Stat-Signature: ygtojxzp1opqjftapgbgiqojrw57ouea X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: EFAF6A0026 X-HE-Tag: 1704215352-462389 X-HE-Meta: U2FsdGVkX195SarIuIf40ENxViDIUdq5rOrb8Qsx2mo+X2Jlm+LHeJ2KmItXzCnf8pOZ+C+Bwrg/fGE1VgZcPnMTkEt9aSMtVbWQSf1QW3XJLhyS4VM7rixS+KIMySMIrddf6Yjt8V0L7r84J8f3WC0KXLfo3GiUgwSjG/Ufo3yo6zg6ptAK2OgOG1mVplDbkXvGsZzbLKk3H+JquzWMDmx5H2wEWQCsRv7xfSbG561Td2anyuASbhkYmmBqAZRor7ZmniLT2TteyqvNGyIH7SeY84IAo+yALphCxq0fkEHp1rqY5ljoRCfClCWM1gvoQz+KkuOxX1CnxASVE+9YYedXEDXN5wYKgF4nO1RBhz3bwGR2KcCLUaOvvfvTrCn//7fycc8nmudw15udyPiqcNSckxA0RBHSjokzenWG+mW4LRz8NubQBc89ahMrofHLrzu/WrHdRQV84CMDt9RIZK3iJwXYgWfQIFqEcUKITmyeciJ/D2uPa8X44+pPOBqdIvgnMRsKoG88wmKtArekwJy0UswYvzWEn7fJ8OJsMJeTOi/6A4hqBkGCmhc3ss8JZ+mfDTDU7QklBslQgkBL5h1ZYqtAc8k4OJriVx9l6OKbhjpN4jld7Fg/cqlDvxm3+wE8XLTA1dXoGqeRPOauk1FBeZc0PrrG8ZmoV7JLalbYrJuzeWf1FDC35ugVA+3zeJSgq4efslRYlVJMlzc6K4L1/pgOvRUYLSWjpSTltbVzgCMGuLY+GhbGuOVxlM/QijLica19gpoKZa0PsxMdzhVP4Dha40Fa9nqFAWl8PAzooFj3AR+Y2fLHcJFYL2+K6In88CuUjEd7q0d0tASs14euWukrX2Tv6K8rTKNAx+A9A0e/gvl/WMPMq+3CaOwb4SAwrLE6E0MPoQZW4fvI+g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Dec 21, 2023 at 01:19:18PM +0000, Catalin Marinas wrote: > If we really want to avoid any aliases (though I think we are spending > too many cycles on something that's not a real issue), the only way is > to have fd-based mappings in KVM so that there's no VMM alias. After > that we need to choose between (2) and (3) since the VMM may no longer > be able to probe the device and figure out which ranges need what > attributes. If we use a FD then KVM will be invoking some API on the FD to get the physical memory addreses and we can have that API also return information on the allowed memory types. > > Kinda stinks to make the VMM aware of the device, but IMO it is a > > fundamental limitation of the way we back memslots right now. > > As I mentioned above, the limitation may be more complex if the > intra-BAR attributes are not something readily available in the device > documentation. Maybe Jason or Ankit can shed some light here: are those > intra-BAR ranges configurable by the (guest) driver or they are already > pre-configured by firmware and the driver only needs to probe them? Configured by the guest on the fly, on a page by page basis. There is no way for the VMM to pre-predict what memory type the VM will need. The VM must be in control of this. Jason