From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80E78D1D878 for ; Tue, 15 Oct 2024 15:24:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1AC966B0089; Tue, 15 Oct 2024 11:24:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 15CD16B008A; Tue, 15 Oct 2024 11:24:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F18466B008C; Tue, 15 Oct 2024 11:24:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CB07B6B0089 for ; Tue, 15 Oct 2024 11:24:22 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 8C3AB1A16D7 for ; Tue, 15 Oct 2024 15:24:06 +0000 (UTC) X-FDA: 82676207586.09.573CD82 Received: from NAM04-BN8-obe.outbound.protection.outlook.com (mail-bn8nam04on2063.outbound.protection.outlook.com [40.107.100.63]) by imf27.hostedemail.com (Postfix) with ESMTP id 1955740006 for ; Tue, 15 Oct 2024 15:24:11 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=TeMzfZgG; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf27.hostedemail.com: domain of ymaman@nvidia.com designates 40.107.100.63 as permitted sender) smtp.mailfrom=ymaman@nvidia.com ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1729005827; a=rsa-sha256; cv=pass; b=T2U8h/3+CkJ0qxLWkEdp1s/uTNj5Z1ZRskhQSjAT8gCvVnRiEs0sBmBqbT8qTM+sho2NFs kGY+Dxc2fJMIc1WiXOsVSRH+JVch4wwRTJNz6raJzG7H6r8Oo4m5z8h1puJYZFHqn/1gb7 9auFWlesBGucRiKcFukAKrlJtKN2Lxw= ARC-Authentication-Results: i=2; imf27.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b=TeMzfZgG; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1"); spf=pass (imf27.hostedemail.com: domain of ymaman@nvidia.com designates 40.107.100.63 as permitted sender) smtp.mailfrom=ymaman@nvidia.com ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729005827; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vWcn9NVt1hK7DI5Z8sasmo2JCUcg6jAxYQ0go1j3Uyk=; b=MV/Pza0P0K0nlbWDYnV7Dc7FXJ7B0i4Khea2m/mOGoi33wOE2Z4rQotRo08X8yIvUTN4mA AUCD7geiK+LLSs033ME3cFb5OLnkD6pbpdUiifGLaUYv9fti+qyHTeh45KiTR1jHCRpuaX P1aXJkn+8b1JZWfwuZD+Y5aApN31+08= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=OOIa4n2GeEtQdQ5mIrObPHNq2tbUyQH0yoRmpptZnFJqba4Lg563lSBhCaM067gFS3dS2mOCWCjJfgsagyXK9Fe3DXntcJQC56t0SzQLSwHx3PaCJmxNj5lwKca7KvHOW+/7t+nYPLUkjjlBp6LDetrKq6KJIZdjfU/P4Tk8KdJsay06aZlxBWTUwq5su6vm7dlyUk2OdAYaoffrDZQWXX0SHFgp6sZjMFQVYg66SsWyoPcM3dJWs61Sz1JX14ThxhwkZ8jVl4CLY7Cm5eIR3yndR6nG5qQNYEItaagbXGfXn4j8GTWm5fR1FWt0dPdo/6ZSf/G+OmiiixbgDQhhIA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=vWcn9NVt1hK7DI5Z8sasmo2JCUcg6jAxYQ0go1j3Uyk=; b=s/hUjmd3GWdOfX2YYvEHuOhin80QYC919HLnOyPbtqKHdEaXd9DPRZBLZDKEckfG02M/9GhzEp7Os/OcFdgQSU/ISFz4p76Qt/Ye4wOGJMxvcAc264M2+NlPwN0wvMWM/GvH2flKfltrtSCrVK7en8d4H09F3EpnsudQorgckr5vszit89oPBk6yszjiLe1gLuyvjrrxQkJ9AlFbTsAZrCuHJswWVci2w96PK3o72lBZfw2WwuI8sWH3G+0tkkNkm4GzszIFGEoOYwCfdAGK85O5SrVNB4mGq+7ty2tapoWUDEGitsANWazvgU9MsNgp6ysjJGFAgR0Eki5srDl/Qg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.118.233) smtp.rcpttodomain=lists.freedesktop.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none (0) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vWcn9NVt1hK7DI5Z8sasmo2JCUcg6jAxYQ0go1j3Uyk=; b=TeMzfZgGXRyEnUbvup3rg4dxNu88kGxtycFUWNDUnHWE0YoLMIdRoezFFfEhcch7toMybYecn87buCZtIXR3gyXf2XRltjTIkMaSNETTbkLXuJDaJnxzNuiw6wmJjvNFMA1AEWOq3tN9T0TdVccNzR6EHzUp2qg/G95jwQqgaAYjx/NCRRod6OGhlUrwa8NlT+Eu99nc27LPTJxq7WmYjCATJx/weMsVoXBEA4y7RvZwYiVZwOH51d9iJFNtHf58BUC4L97WrQDzKw0FdFtmuQP5IxderL0Ei5RgTWIi5T7I9RAf1RbYTbQU8NdDsywY2LRIOhphPYKQVjr8E39XVg== Received: from SJ0PR03CA0135.namprd03.prod.outlook.com (2603:10b6:a03:33c::20) by DS0PR12MB7996.namprd12.prod.outlook.com (2603:10b6:8:14f::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8048.16; Tue, 15 Oct 2024 15:24:15 +0000 Received: from SJ5PEPF000001D1.namprd05.prod.outlook.com (2603:10b6:a03:33c:cafe::60) by SJ0PR03CA0135.outlook.office365.com (2603:10b6:a03:33c::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8069.17 via Frontend Transport; Tue, 15 Oct 2024 15:24:14 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.118.233) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.118.233 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.118.233; helo=mail.nvidia.com; pr=C Received: from mail.nvidia.com (216.228.118.233) by SJ5PEPF000001D1.mail.protection.outlook.com (10.167.242.53) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8069.17 via Frontend Transport; Tue, 15 Oct 2024 15:24:14 +0000 Received: from drhqmail202.nvidia.com (10.126.190.181) by mail.nvidia.com (10.127.129.6) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 15 Oct 2024 08:24:04 -0700 Received: from drhqmail202.nvidia.com (10.126.190.181) by drhqmail202.nvidia.com (10.126.190.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1544.4; Tue, 15 Oct 2024 08:24:03 -0700 Received: from vdi.nvidia.com (10.127.8.9) by mail.nvidia.com (10.126.190.181) with Microsoft SMTP Server id 15.2.1544.4 via Frontend Transport; Tue, 15 Oct 2024 08:23:59 -0700 From: Yonatan Maman To: , , , , , , , , , , , , , , , CC: Yonatan Maman , Gal Shalom Subject: [PATCH v1 1/4] mm/hmm: HMM API for P2P DMA to device zone pages Date: Tue, 15 Oct 2024 18:23:45 +0300 Message-ID: <20241015152348.3055360-2-ymaman@nvidia.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20241015152348.3055360-1-ymaman@nvidia.com> References: <20241015152348.3055360-1-ymaman@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-NV-OnPremToCloud: ExternallySecured X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: SJ5PEPF000001D1:EE_|DS0PR12MB7996:EE_ X-MS-Office365-Filtering-Correlation-Id: 7c904050-d6f0-4e81-1d1c-08dced2d6daf X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|1800799024|376014|36860700013|82310400026|921020; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?DfnJDOvTskSHOrWolZIi/Fe9XUtlZpVqH6rLAkZNFCSfnSOYyTNOUgx+XtrI?= =?us-ascii?Q?SmwUY89Gc+zL21Vh7T3xjcR3ChSlJmAsg4zOKB02SEA77WSbtmw//RKQWyH8?= =?us-ascii?Q?srpPFT3op2aJdwEEs2G4Ifs/VLnWDaSdACMsZMFOqtP50pPdQBy0r5BwdyrI?= =?us-ascii?Q?t8VgOxyIag1VGiJPg+K6MiVKJAr70mErbkIjIZcEpNi/nPboO/yuxIizy/rF?= =?us-ascii?Q?iDXAJXaF4silKSn7ApovMzRo5lZoYuZIKAkWLPMU6fB9owzP2wKs7QvSww2W?= =?us-ascii?Q?CVvZBXD9n9l+NdLxUgWf5LNuPg/iVD4vwnyrVauJm1ayMlVI+v4c3Lod6vhG?= =?us-ascii?Q?vi8vapM1cX+rodzD1WohhaPJ+t81xqj/9OJp6N7fvzVBd+hx+c+p4Mnaqy+y?= =?us-ascii?Q?hEsNRI7LXkS7xWnbt0H1lV9Je2nkUA/dhrYFZ8FxJ0eSPKWl9WiAo7VEUow+?= =?us-ascii?Q?05Sc7F5t4pC+janrlXDs7KCe14PuJyiRUCyYGizKN04K1XDSk6zfujiAKdX0?= =?us-ascii?Q?4Gq+xKhyzjAiMc1SyMtCsyNdsus17NqC6sWV1XeknlVOSybhZx4IGlZWaR1I?= =?us-ascii?Q?lIW6qNVRYdgOcCX8sgNGmqEF2fzyGnfO26RwzAWP1gi/KD9PopE+9aFR4Ftr?= =?us-ascii?Q?h1sfCKegC4PX09km0G8KQ1of5ELn1N4c08MgAk0zUCEkjFDnYzf4jmuK9WZC?= =?us-ascii?Q?3DLLresibMLu3xZVBcwcvhxDDI/H/qMBUjsUxLt4BkgXjgUPI7CnX+cuKpTP?= =?us-ascii?Q?UK7Ym7hzXqvFSr+5+nf1VJ60SG2cvTSIHOp746IPt/CywM8v5Q8z935cSBls?= =?us-ascii?Q?To03yNdQkQgpQxlpDu6EdKUzvj0TFnU+nlaDBhV1sdFCYPkcH44opP7b49q/?= =?us-ascii?Q?O1Zn8uAjCAmLnNiPhWVaWsGy7X97cNXMLrV6B5Lzdv0T8bF9N3Tx9n5w2g16?= =?us-ascii?Q?aBlJRJP7vWWuDsJBa955lq6hm8hRKLgfsAoICFiFA+fztx/L+ACuAAe00UZU?= =?us-ascii?Q?vee1yNk1SMbhzNLom8Qvl9Q0sE1SwDMz7Gn9eJ8PPN6jSkfss5RrWw0c5kBm?= =?us-ascii?Q?Rru6RISKCBw7P37ond3gLSXlJX7wSRlUu8X8+yaIaoaRiOgV8a0Lqy8b3rJq?= =?us-ascii?Q?w4cyHcjFeCfk6r6aAwhGhFBkNVVY0Vqv/DfRipK1YWNoIAiXc0z/d7unK0Xm?= =?us-ascii?Q?TLhrGq7+0CJdl1FGrLTI4hEYAJbMw+Dqa9Q+3cLL/uIWDcqb0w6+xArwMVZ+?= =?us-ascii?Q?Z0l28ANL0k4HdOjgW6MILiaUeYCt/u3R2wSP690X9yVeBqVmI3T4LpIGWQoZ?= =?us-ascii?Q?07HN3nA0T9stDf1FaY++67Ucb4BK6fpmms9HSb07yC2R3ZlGt10pISSXgjob?= =?us-ascii?Q?JLhn8ZwrGntjAVuK3VybTKNWWFbn?= X-Forefront-Antispam-Report: CIP:216.228.118.233;CTRY:US;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:mail.nvidia.com;PTR:dc7edge2.nvidia.com;CAT:NONE;SFS:(13230040)(7416014)(1800799024)(376014)(36860700013)(82310400026)(921020);DIR:OUT;SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 15 Oct 2024 15:24:14.9115 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 7c904050-d6f0-4e81-1d1c-08dced2d6daf X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a;Ip=[216.228.118.233];Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: SJ5PEPF000001D1.namprd05.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DS0PR12MB7996 X-Rspam-User: X-Stat-Signature: gnr8jfyeqtbdb3s8ngpn916itiusweop X-Rspamd-Queue-Id: 1955740006 X-Rspamd-Server: rspam02 X-HE-Tag: 1729005851-249677 X-HE-Meta: U2FsdGVkX1+oNghe4jF0suxtHl3lnuwRxuAqCVZqu87wfSpp/H5orhD3WkUEqZMMxUZKTTClf+qiD1kCc5gwx0jLrwtxHD7gxmUNDSh98XHwDxZLE0oSXDnLkZhrRe0SacJBSPyuzmikcXYloOmCVnTdTx6yFkGOoWRYOVf/3t8mZsN6FG7d87HN1My4N6XdHxpbuGBnC+iasoUIaqx5E8W5w0i73mZgfyJZXig94B6PlgvGfsjhLpOZgY0GSYMgzwo5WIN3lH2kl7PuAsYpM+cRjY1lm/RiEkPO2kctCDq5pgTWgfO3ntHvw/9JIbhUE/y06Bo0zANAzMMXUpx1x1z+VTtk08v96lmyszzh/eITvYTrRGMgv/SckeTrkT6KErVnMcxE0orP5N1oiEBjmPp4hJ5tcEOtunuz1JmD6Ax6yBCVjcmPkn/xKy3AuTuYvJVsn5XoKEijPpxKYb+c0AojGx1DnZp2S6Edih7h2cfhgvJ+ufXvQl7cIB8PLASiQk+6b/9An6BH9C9aA2tGuhJG6nu1FaNplp0EVYbJk+w8XLuE1ihNHWJoQf5puvhVleSXL1fpHdQla6gkl0eLeAuXD+voTd+8fNgVz5Du2AtSzi59pfD8zQZtEnITVad85MXAIs0BBP9ZtrtwITEGTuMH6UwU3/T6KGPZUg4A0qVY7AskL3dWuen2pvIAuSTWO8RWNxVP2/l+0uAh4S7KBfIWmP97rH2Do62zb0l1FR9di9eSYauvfm99overGY/b0b2XPNkuojp73W7WpDx4d5h8Et4rqd8TrSfGTkre0zIOVFOPFP0rNw0ek/DJiEF95X+1+qp1jYTXAYiHZSx25CPmLGizsnhzyLTdy9PwCQDzKOLd4k0VrPgtMOjXZx+E6LbdtAwJR4cMPfR6nAo62KLFtShwU/JRERmHA06u2+GLQapYDEqzqlQslz4Ok/xHwPYd+uXReoQcR6eM4n4 VfcnYD59 cwQURPxtSLoT8RTW1DJM6GbXod/zVOx1OaduTqOv6jI7dSMTZxNkS0LXB9mQbXBJOAAkYT2m+YAtFcbm5aprRo7iJNzQuyy0/gJhON45JJ8AK4dzAxDDQL/M30JcNPHKb8tdvvZjqJDuXG+y7yuIKUysJJZIKcIQI8PzDbiBJeYHVkrjA9I8HeP5L/u8JHEkpqmf6+kq93XVrQoRlqAjz8AMKnPcxmVhON68FHWr1zPc4BOOBPUE33IBOKGQ9o5V8Whj5QWWqDfqjLH7VReeAOYwkbYZ7DkVENfhK5ESyU71D7DQv552qxGyLfwahWv3SPo+nU9n7XtkKv0dA5XmE68HS0g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Yonatan Maman hmm_range_fault() natively triggers a page fault on device private pages, migrating them to RAM. In some cases, such as with RDMA devices, the migration overhead between the device (e.g., GPU) and the CPU, and vice-versa, significantly damages performance. Thus, enabling Peer-to- Peer (P2P) DMA access for device private page might be crucial for minimizing data transfer overhead. This change introduces an API to support P2P connections for device private pages by implementing the following: - Leveraging the struct pagemap_ops for P2P Page Callbacks. This callback involves mapping the page to MMIO and returning the corresponding PCI_P2P page. - Utilizing hmm_range_fault for Initializing P2P Connections. The API also adds the HMM_PFN_REQ_TRY_P2P flag option for the hmm_range_fault caller to initialize P2P. If set, hmm_range_fault attempts initializing the P2P connection first, if the owner device supports P2P, using p2p_page. In case of failure or lack of support, hmm_range_fault will continue with the regular flow of migrating the page to RAM. This change does not affect previous use-cases of hmm_range_fault, because both the caller and the page owner must explicitly request and support it to initialize P2P connection. Signed-off-by: Yonatan Maman Reviewed-by: Gal Shalom --- include/linux/hmm.h | 2 ++ include/linux/memremap.h | 7 +++++++ mm/hmm.c | 28 ++++++++++++++++++++++++++++ 3 files changed, 37 insertions(+) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index 126a36571667..7154f5ed73a1 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -41,6 +41,8 @@ enum hmm_pfn_flags { /* Input flags */ HMM_PFN_REQ_FAULT = HMM_PFN_VALID, HMM_PFN_REQ_WRITE = HMM_PFN_WRITE, + /* allow returning PCI P2PDMA pages */ + HMM_PFN_REQ_ALLOW_P2P = 1, HMM_PFN_FLAGS = 0xFFUL << HMM_PFN_ORDER_SHIFT, }; diff --git a/include/linux/memremap.h b/include/linux/memremap.h index 3f7143ade32c..0ecfd3d191fa 100644 --- a/include/linux/memremap.h +++ b/include/linux/memremap.h @@ -89,6 +89,13 @@ struct dev_pagemap_ops { */ vm_fault_t (*migrate_to_ram)(struct vm_fault *vmf); + /* + * Used for private (un-addressable) device memory only. Return a + * corresponding struct page, that can be mapped to device + * (e.g using dma_map_page) + */ + struct page *(*get_dma_page_for_device)(struct page *private_page); + /* * Handle the memory failure happens on a range of pfns. Notify the * processes who are using these pfns, and try to recover the data on diff --git a/mm/hmm.c b/mm/hmm.c index 7e0229ae4a5a..987dd143d697 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -230,6 +230,8 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, unsigned long cpu_flags; pte_t pte = ptep_get(ptep); uint64_t pfn_req_flags = *hmm_pfn; + struct page *(*get_dma_page_handler)(struct page *private_page); + struct page *dma_page; if (pte_none_mostly(pte)) { required_fault = @@ -257,6 +259,32 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, return 0; } + /* + * P2P for supported pages, and according to caller request + * translate the private page to the match P2P page if it fails + * continue with the regular flow + */ + if (is_device_private_entry(entry)) { + get_dma_page_handler = + pfn_swap_entry_to_page(entry) + ->pgmap->ops->get_dma_page_for_device; + if ((hmm_vma_walk->range->default_flags & + HMM_PFN_REQ_ALLOW_P2P) && + get_dma_page_handler) { + dma_page = get_dma_page_handler( + pfn_swap_entry_to_page(entry)); + if (!IS_ERR(dma_page)) { + cpu_flags = HMM_PFN_VALID; + if (is_writable_device_private_entry( + entry)) + cpu_flags |= HMM_PFN_WRITE; + *hmm_pfn = page_to_pfn(dma_page) | + cpu_flags; + return 0; + } + } + } + required_fault = hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0); if (!required_fault) { -- 2.34.1