From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C641DC77B7C for ; Tue, 24 Jun 2025 12:18:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 65F0E6B0089; Tue, 24 Jun 2025 08:18:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 636E06B00AF; Tue, 24 Jun 2025 08:18:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 525626B00B2; Tue, 24 Jun 2025 08:18:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 38A0C6B00AF for ; Tue, 24 Jun 2025 08:18:54 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id DD7D0140640 for ; Tue, 24 Jun 2025 12:18:53 +0000 (UTC) X-FDA: 83590198146.19.D6180C6 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2040.outbound.protection.outlook.com [40.107.237.40]) by imf24.hostedemail.com (Postfix) with ESMTP id 35CDF180012 for ; Tue, 24 Jun 2025 12:18:51 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="n0U/Rwrx"; spf=pass (imf24.hostedemail.com: domain of jgg@nvidia.com designates 40.107.237.40 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Message-Signature: i=2; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1750767531; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fLha9285DVj7/grm5DMhOghm33y+WqnhAjlZVpKBwaE=; b=bZnrwRe7rA2DuygnhEorX+Ym32O+7x8G6z/9CXotcuuhb7umo0cgYXHiLbxyDQogBrhUaz 47iZkztPmHcBc1aW1oFQDBFfp/NW8ePOVXsHZMX/3vPN9OQ4HSF1SHMlAbArf64P9EaU2F GINucD+nTNRgVXYzx/aTqNlTAOJ2tHw= ARC-Authentication-Results: i=2; imf24.hostedemail.com; dkim=pass header.d=Nvidia.com header.s=selector2 header.b="n0U/Rwrx"; spf=pass (imf24.hostedemail.com: domain of jgg@nvidia.com designates 40.107.237.40 as permitted sender) smtp.mailfrom=jgg@nvidia.com; dmarc=pass (policy=reject) header.from=nvidia.com; arc=pass ("microsoft.com:s=arcselector10001:i=1") ARC-Seal: i=2; s=arc-20220608; d=hostedemail.com; t=1750767531; a=rsa-sha256; cv=pass; b=g1IWWnXLta7h1gohmqsIKlfjAaU4P2UdGG4oL6XjUohi8H12qFQfgGReau466Kqq/O90rj j9+kX+5ZU8H5HtM32EVaZW9hcJvOb9zsSHKhUHitH8LgXmY1xHLJ8wWgaFtPRxSFu0P3hm NqMBel8vN0Fdsvt7SJHs1UEqbhJ6C8E= ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=MK0T7K9ikcJsj+rcAwwUzdJbWSi6a3DJTVu77wONjkf10ZRLfM5Meel7LMM3AvS9IpElvlnMvQNZ33Jp3gI7xClpjt016vy/qCP7xUE/xbu4eyDXHXUcqIG3xtrqMsuk46+Ml7JwZHvD1KhbgU/1JLAFHORxGCooib2xREWmAimnbS1a/XAhps77jzZZtpunyf1j84z+n5gbLJeM7MZaRQ2pvPHUN2B8a3rY531OlwmNm1rJETLJJyG4nb/cZkgCSqUhCHtsF9jgUi8uexD6VEbON5400yffv1CPPp5KTdLBLKktXzmOira0Rs0F1AuPPoWL8X6n4nRP3isRkOgVfw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fLha9285DVj7/grm5DMhOghm33y+WqnhAjlZVpKBwaE=; b=XMtwuT84LIk6n1zPJt1QKzWoQPk9Le1yUWVv0pCkkKwmTqJ9oPDn3WDJ7LF5x/xwSB3lvoODv10+1Y63w/oRW9x2pquSH7vf45yMbPiBJgubkq5HWWS9u5MULtUn+uKfLr1xuerMODR4gWHCwtj4bJJvkkEIZ50zQ6jKWuNxpUJfu7Kw6R24EObvQQgeGYD5Xx20uvUUktUkGWvdsJRWFcDjBhl8USf9Us7C7tBUJDt5CUo3yr1QriAWMFBGS7tbX0+VjOxkLSmXyFodEGWsyB5m1l1ieiXaXyD+thlJGtDpZZyC2onasIVm515eol27MBnGQoOGIKO1jDdxhBF9nQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fLha9285DVj7/grm5DMhOghm33y+WqnhAjlZVpKBwaE=; b=n0U/RwrxGhHyOnOhVhdImoo6yJ1wuoEarn7N6dQFru63kMO8OCpv67IOX79pwh1/JllS3xRAkILyFvYNX6U+w+hbvTcnt3XMgY4cfCm1BMBwPe86D2h/KiCXg7nN4zsCkGVg0VqW3vkZbRcfUwOCSYg/dDyu5Sjt36fk9I4NwJD0PFiGI8B64EAwfizrz9PBnSN7vE9Jy3VrbCGWYaQ5gI2YJFrjcQxRuiJr1/kmGnlWmVSBid5fPIW+s/BpzJ0KZ8WQv7sA+sP8SaqSPoE+SiztodhOIT81Z8AyEik2zrRmPUaCAm8JN0icONtn1NoeJE1gPRu7SSHoObKVExYX3w== Received: from CH3PR12MB8659.namprd12.prod.outlook.com (2603:10b6:610:17c::13) by CH3PR12MB7594.namprd12.prod.outlook.com (2603:10b6:610:140::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8857.29; Tue, 24 Jun 2025 12:18:48 +0000 Received: from CH3PR12MB8659.namprd12.prod.outlook.com ([fe80::6eb6:7d37:7b4b:1732]) by CH3PR12MB8659.namprd12.prod.outlook.com ([fe80::6eb6:7d37:7b4b:1732%7]) with mapi id 15.20.8835.027; Tue, 24 Jun 2025 12:18:48 +0000 Date: Tue, 24 Jun 2025 09:18:46 -0300 From: Jason Gunthorpe To: David Howells Cc: Christoph Hellwig , Andrew Lunn , Eric Dumazet , "David S. Miller" , Jakub Kicinski , David Hildenbrand , John Hubbard , Mina Almasry , willy@infradead.org, Christian Brauner , Al Viro , netdev@vger.kernel.org, linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, Leon Romanovsky , Logan Gunthorpe Subject: Re: How to handle P2P DMA with only {physaddr,len} in bio_vec? Message-ID: <20250624121846.GE17127@nvidia.com> References: <2135907.1747061490@warthog.procyon.org.uk> <1069540.1746202908@warthog.procyon.org.uk> <165f5d5b-34f2-40de-b0ec-8c1ca36babe8@lunn.ch> <0aa1b4a2-47b2-40a4-ae14-ce2dd457a1f7@lunn.ch> <1015189.1746187621@warthog.procyon.org.uk> <1021352.1746193306@warthog.procyon.org.uk> <1098395.1750675858@warthog.procyon.org.uk> <1143687.1750755725@warthog.procyon.org.uk> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1143687.1750755725@warthog.procyon.org.uk> X-ClientProxiedBy: YT2PR01CA0002.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:38::7) To CH3PR12MB8659.namprd12.prod.outlook.com (2603:10b6:610:17c::13) MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: CH3PR12MB8659:EE_|CH3PR12MB7594:EE_ X-MS-Office365-Filtering-Correlation-Id: 18e0fdf2-809e-4873-d7a0-08ddb319457e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0;ARA:13230040|7416014|376014|366016|1800799024; X-Microsoft-Antispam-Message-Info: =?us-ascii?Q?6AgfjJusFXxPQoqD+WyHjkifvm3L12grtITrjA3I4n4rTpe5Z0Htpm67G181?= =?us-ascii?Q?oaYwYj17D5X6Tl0/dtQ5KyrJNafOZJ4wIjS7rgFvGVQoh3Aw4FewRimLJqNR?= =?us-ascii?Q?DZRvLzkUEdCB5RDED/qjpCx8mXiudru7QKf2jtOZWOmqihbRBqeZjs+TrSJD?= =?us-ascii?Q?RgNGIAKp74g9t9+OA2habq/o0rp4pz85NsMn/uzBHNUUoNmRvcBWFDgJ+DJT?= =?us-ascii?Q?BDA+OMP/ZHkABL8xMm1v7+aGd1iQO4EY+3ejzNXGK8xADK6uFCO+LTwvdymc?= =?us-ascii?Q?bnRzQzA4sAF3TwFQitkfSsidpCSZUirOi15a5xUMAF5uTryVS9KtqvM0bl8u?= =?us-ascii?Q?qeKVo8OCumM4W6zldNo0VXyJ0o3oT5Z7Zj02XXYS3QnvOyHQHB5G/Fh2BByM?= =?us-ascii?Q?1sQr0iLCeljplLYZ5GPDelt1p9Td53Fwr2qRrENNZ0krX94zfqUtyvwHV3L2?= =?us-ascii?Q?LW0DWbxiKi+St4HGaFopToIEbXv74oJECnPs+fazH/n49MAkbNWlebr03fuc?= =?us-ascii?Q?Umy13/y8NaIJOnl+4cTpCWAX83CTIT6C5kwY1WLUCZ3+svdpRgtw+gHOagt0?= =?us-ascii?Q?FxZg55/JbEoG2o1ZEBBhwed/ELXkHGKvgw1MSvOQM303okgt29vQr01cr958?= =?us-ascii?Q?yym9+ibRVK+pOKAH7fODA3qOq/9jHPqBZ6RGLTNFBfcnkSn5xdupbIkGuCcM?= =?us-ascii?Q?svCd6nbaILWpaUwAxTerlxA+itvXpvHe8Q4ox2rTGpqa1zOoBFOsFQ23XaWU?= =?us-ascii?Q?hPUJa8tMourcNBv+sfjMclvYod71gcq8QNIFFYnhO0l4h4pU2PwNkMl3bsZg?= =?us-ascii?Q?VyF/N9ql7vBdIfJ22CRJp3Ox+22EWFlJm4CzFQpBgTk5ZSP+OBnKlxUFH6hC?= =?us-ascii?Q?1BckV62Ehakz6XUYBpggIDVFACIL0VcuLS8eBNmJa292MvBuDi9sbbafyYT7?= =?us-ascii?Q?UuEaXVA36YGO4Jo+du7mz3ecD6kZgUh9JwpAWBqj+aU5wl4PRTXtAW0ZS79W?= =?us-ascii?Q?RAspEeRkWvLt+1/KrDsZR8y+GKWP3iJ+Otq7Qc/2NDsCHnAvS3yTzT3XIouQ?= =?us-ascii?Q?mNNKes+ljkO4naCE3C0lriuPG9TNSX5DBXwzSFC14B1+/kqEpc8orH62KNYH?= =?us-ascii?Q?vmlailsp6GlrUTHKfrRJIiYa0ZTUVB/psxwmdoJcqVRYKpqQpVWPB4zuETSL?= =?us-ascii?Q?6daeeMm/nJVGty5VfpEFPJPz3CKRQfqkp6sa5iLoB9pNZxKVN0pN1PB2MpMZ?= =?us-ascii?Q?5erjlLFowVn14xmZrS39p/3nyCcXU1fDv+8PNQLBaa2s1rN5Go5A/8yrFRc/?= =?us-ascii?Q?TmPpqwGcN8z27pzXWl91F9m7iEhdvnLsSYqfmk8o9JGIUsTIspgAXFeljl4H?= =?us-ascii?Q?ga8amcwNEaWA7hytEkSnGQdzZ18lX3xkrIM5X3CRCprSy5Fx+LkDOh+u3j+X?= =?us-ascii?Q?u3Lt8HStgIY=3D?= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH3PR12MB8659.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(7416014)(376014)(366016)(1800799024);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?us-ascii?Q?lrauL5sTMNfAZKnBxcMSyHdCftAi4mDZ1EyUVKDIKsjO6UANIj8bOl009csP?= =?us-ascii?Q?6ftKvehLzF9zoTB5r3GeojhSfm/unsIVPctdHJ28MYd6VXTSfSVNxrXIgvSP?= =?us-ascii?Q?swyyWKWBocEY5be99RQmbFmhDTLJHBlazXp04CO0zb+Hc7fHuTQG3/Dth1i4?= =?us-ascii?Q?GEsVNG0YVo0Slh8OmmTVvgcjbil0NvnCsqgwS5z6FlEVRHoOUg09Y2yOQ5dV?= =?us-ascii?Q?e4zAAj8/VS1tWAJkseSo8LxolDoBuA+F/1CrimIlvjJ+/+Qoss+5BEJUEI5a?= =?us-ascii?Q?oinu3Y681b0DSjHiYhHyrM8ezfPbC8KdCYgtHtOXLxWjNP6F2PuzPeia1HJZ?= =?us-ascii?Q?2AEYjBrHKEBOPnhHaB/XQHoMR8hn7NNBZgfk2iVQfw4hqpgrjGqJtBYDllUC?= =?us-ascii?Q?+yye1pAycfqaC0KW/TyRr2Mjw2jftw1qjoFHfCZG6GVzVpX5opl7PSB8oL2W?= =?us-ascii?Q?7H7HL/t0yvdxK+EFakzpFEqepObzyxEWIHkjtO8+GpDbkT2v1XvEB4e/Wi8l?= =?us-ascii?Q?alDD5v3HJFJ4U8ZSbjXVRzHXh9IKa3kJtMcpOeWuPtHH6hdaAZxwR+WrH2Vo?= =?us-ascii?Q?vwhQ8tvo3dA7plUAeNei3CyRxVjrGLM55YMjhsGMsiwnfGRxRp4BOArslaDj?= =?us-ascii?Q?ngoLPhXbcBepm8FSVxilKnqkH0i7+T331bwW5Ej0tkI+2J9FM2ikyPr6insE?= =?us-ascii?Q?E9NJCu0Qm0ebBkCNbwr6G8hfnGsvMcqHxOcT4HGQVxWrWI/We7Ge+eM6hZhC?= =?us-ascii?Q?aafjZpjE3e0b60jbsBMaX/weHFTOcycDhMQoZiln4xUumfuidrwuehQLUvox?= =?us-ascii?Q?3yi6AIYGTtcMFqfmJoDGytuyhuB2LcM72R3LtT90WopSv0NdPDBx6gKxPsfU?= =?us-ascii?Q?cK6lHCsJ2Y55rooJITtw+I7Y70xblAGmmoiH990F50ccsgw8EZsZcdtofY61?= =?us-ascii?Q?19mZcDHctAAsiCR2WWCjqFczTUF7u9FjRiDlm7ERUFIZZEg8tVzCtfvXVICj?= =?us-ascii?Q?mR3s9N2J6OAPKA7DHbHWQX5vGeanznClNw8ImcXpZ/OymDS3fK5GZ9tpXYwx?= =?us-ascii?Q?WZndIVso+QOhSUk3l3iEzeeAQVg3AHiLCLzHjT4IQN0iJMAjUF3vmCg6z4Fu?= =?us-ascii?Q?zY83SXKASPytawboaHwK5qYwFHu0Y/W1FGQOvU1dkn3YvdMzzAwvAGBVA+U6?= =?us-ascii?Q?I+1EKCn0jkmkzHp3EBp3KwwrPQ5/2oosl1+bqqGTid+1W7qPj83FpX6IjipB?= =?us-ascii?Q?BJeSSXc65neYq3LhC6t0x+TvdesFw1XkRUJoiRXU/SY66htltJhW2Gq1F2Iq?= =?us-ascii?Q?mAmHOa4F+825EDDksK80cwqcy4APCKk+krFTMa/EcvSltadGg3BQ7/lU/kvy?= =?us-ascii?Q?baExxJtg5BtGUk94qK+f4vxs5ai3jZhyVeHi69LsHy0bsmyn6y5YzeBctAAL?= =?us-ascii?Q?BvM0l155jIhNkvdQMHLfyui/ZeANDehGCZqh9zNi7fs2UQMYo4PPxTOJbU1Q?= =?us-ascii?Q?6L0c2RkjfQzuUi/KfSnCheLONvcemWc+jcttfzFFbpb5SVzqZUA7kXxv/JKF?= =?us-ascii?Q?dRpugoxbNctZMk68CcY=3D?= X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-Network-Message-Id: 18e0fdf2-809e-4873-d7a0-08ddb319457e X-MS-Exchange-CrossTenant-AuthSource: CH3PR12MB8659.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 24 Jun 2025 12:18:48.2290 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: MhlDdHmuo5FZmOtXQ8+vWWKgyJRWFRWpG3oKNrlM3v6AlqhvDJ2KsqecejrqZd28 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CH3PR12MB7594 X-Rspamd-Queue-Id: 35CDF180012 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: 78ya4f7qkx7gzira4n95mx95xnekuc17 X-HE-Tag: 1750767531-913149 X-HE-Meta: U2FsdGVkX1+QZDR7dwqIMRdjOHo+mMRfrhtDv893cwqiZX52euNGXhm4Yrai8nf3WgT23fZPFkpSPvOor4hqyRFbxSJt/oYsbUjxKgVf5Ilrf5ZzIw41VeRfLtodUj73bGRhf1/pa0eAZZSKI7A05vPUEWK50H2tQ4+8MkUhlaOGbUdQBgoIf0L5LWZjc8NW/vzVG7L4hM1V8GAY4CHXKtgKkOM0DQ/0bVXlZKGxMKXhzyNeHe0pd/+Zg34PZMixCwydhKYABsIRR4dk9UlI29Wdy1Wo+duUlt+cg+AQlW/9ltpWWc0/8tr7IjWztvX0U/Xw54jGiQr0k/3LHZFyQEps9YTyaEc/r/kRQGP5aCs4c+G8AXewDAOg5PZuG93k2CfEqd0VfKTeQ6L4nM6ADPqe7B7Qu8BjISiJD6wSNwT0I54d7+hEkiW5KYypDyyFasdR8G383p/9YuOBXkkL0oLsvwiLMRkUEegZwJUKlK5yZs3yR5ovOxHv0dLf8eNGWCbHnAt39XXoTDtzi84Jz1Og1mKAsZULmXhZsJeHOhi8J4PNUSdJvTmD5vtqZzi2rdeh7/5rA6aFrFK8QY2ojwnHWktvs/UD4X/BI2suHy/UqZQibQ5c+MnM8GTGwv+ySdNK4VHCrEbheBO6Fc0RvpTKuxTNZvzAajl+fQWWKNgZ8gXXS35Z7E7y0PuqESm5tQdRU87IfbXTYPGR9ySf29eU5rrqzaGLL0WnuI9wQWM01ADdDKcK64eD+nozMmJruO6PzeCE/QXNG7u5skocst8JfzOLkNb6h/G6RL/8dIXqoNsyr38RR+JiU8iitC0/J3YhicnIFEmpjknhPmuJv7rY/49X5pmmjTi7cJ713aduSxNu0vE4Eu7iLwGneOufmMV/2sNRNg45i9QYZVGM6qTYcLRnYNmC2riZLTeMrl82TgJ1Nku1831EkmK0kPNWWfro5Qm1xNS5ulwaHff OxJbcV55 m9oAcAEE1bpGIM8v19rZ2cSAN1PABwbQWWd0w0SVaqYF407QwImkUPhN7Ag== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Jun 24, 2025 at 10:02:05AM +0100, David Howells wrote: > Christoph Hellwig wrote: > > > On Mon, Jun 23, 2025 at 11:50:58AM +0100, David Howells wrote: > > > What's the best way to manage this without having to go back to the page > > > struct for every DMA mapping we want to make? > > > > There isn't a very easy way. Also because if you actually need to do > > peer to peer transfers, you right now absolutely need the page to find > > the pgmap that has the information on how to perform the peer to peer > > transfer. > > Are you expecting P2P to become particularly common? It is becoming common place in certain kinds of server system types. If half the system's memory is behind PCI on a GPU or something then you need P2P. > Do we actually need 32 bits for bv_len, especially given that MAX_RW_COUNT is > capped at a bit less than 2GiB? Could we, say, do: > > struct bio_vec { > phys_addr_t bv_phys; > u32 bv_len:31; > u32 bv_use_p2p:1; > } __packed; > > And rather than storing the how-to-do-P2P info in the page struct, does it > make sense to hold it separately, keyed on bv_phys? I though we had agreed these sorts of 'mixed transfers' were not desirable and we want things to be uniform at this lowest level. So, I suggest the bio_vec should be entirely uniform, either it is all CPU memory or it is all P2P from the same source. This is what the block stack is doing by holding the P2P flag in the bio and splitting the bios when they are constructed. My intention to make a more general, less performant, API was to copy what bio is doing and have a list of bio_vecs, each bio_vec having the same properties. The struct enclosing the bio_vec (the bio, etc) would have the the flag if it is p2p and some way to get the needed p2p source metadata. The bio_vec itself would just store physical addresses and lengths. No need for complicated bit slicing. I think this is important because the new DMA API really doesn't want to be changing modes on a per-item basis.. Jason