From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E8BD5CDB465 for ; Thu, 19 Oct 2023 16:43:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3F4CE8D01A7; Thu, 19 Oct 2023 12:43:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3A4D58D001A; Thu, 19 Oct 2023 12:43:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2942B8D01A7; Thu, 19 Oct 2023 12:43:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1AB3B8D001A for ; Thu, 19 Oct 2023 12:43:18 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id E1DA440496 for ; Thu, 19 Oct 2023 16:43:17 +0000 (UTC) X-FDA: 81362781234.18.28AB0D4 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf21.hostedemail.com (Postfix) with ESMTP id CCA071C0018 for ; Thu, 19 Oct 2023 16:43:15 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=none; spf=pass (imf21.hostedemail.com: domain of robin.murphy@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=robin.murphy@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697733796; a=rsa-sha256; cv=none; b=h8hDpa+SCRwfPTb6BLWUm0nVSLMeQeS5z6Eu3gqxcwTB2k9g37Agm8lAFRxM2bZnlXfIA8 uLsUBSqQ6IvPXC5/8TZ4AXv27o7Uw8WzEW0fwCSQR/AJDSApmUyc8D7n0YlRm3q/qhcjd/ IjFTTdsrHW1Hpuk62DwdpDxxyTp4M+o= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=none; spf=pass (imf21.hostedemail.com: domain of robin.murphy@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=robin.murphy@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697733796; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Y3lIkX/FowZ3rzaRGGmmq3AoGPXW7t/P7tI15d3ku6A=; b=zxW90HPC+BDsNB3s2HxusyE/oMxL5692rmxG/V1Yu5F770wL4vGXNPwjDRofJijMZz9Ndv H4343slbFjNZKV6/xS1974GUomV1S2PksNNoPdq0Nm+iBe57bFxO67CEEJpZqxVCNCdGDS ddlho4G9wRU5tDUApttZJg6TED1y9q4= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 892062F4; Thu, 19 Oct 2023 09:43:55 -0700 (PDT) Received: from [10.1.196.40] (e121345-lin.cambridge.arm.com [10.1.196.40]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0AD7E3F762; Thu, 19 Oct 2023 09:43:12 -0700 (PDT) Message-ID: <3f5d24f0-5e06-42d5-8e73-d874dd5ffa3d@arm.com> Date: Thu, 19 Oct 2023 17:43:11 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH RFC 0/9] Exploring biovec support in (R)DMA API Content-Language: en-GB To: Chuck Lever Cc: Marek Szyprowski , Chuck Lever , Alexander Potapenko , linux-mm@kvack.org, linux-rdma@vger.kernel.org, Jens Axboe , kasan-dev@googlegroups.com, David Howells , iommu@lists.linux.dev, Christoph Hellwig , Jason Gunthorpe References: <169772852492.5232.17148564580779995849.stgit@klimt.1015granger.net> From: Robin Murphy In-Reply-To: <169772852492.5232.17148564580779995849.stgit@klimt.1015granger.net> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: CCA071C0018 X-Stat-Signature: mcwp9c89re111o3761t33q3rc5tnm83p X-HE-Tag: 1697733795-886811 X-HE-Meta: U2FsdGVkX18zg3IY2SgG1tlDrDol60EYhZfHCJwi1diwh5ZKk+dczzcSTRtpMEi8v6uh+8rYfzQA8i7UDk9UN9aQjTYX5nocnHabOv+Hj+73lkZ5i2BS/IS9ZQqzEiaesMJMwMi2H5vRe/gqj0SnzY8gBuUdnxP6pc/YByHyF8kWoDvUghiTQwJcy2i/gI6bhUfv0dZ34FEQoHiehuDyJYWO9SyGup/sbXP8J9SiprrRddISHLLCtDl6YL7lgckYm6hVeCPunj1boj40Xqgb5G0+8EETws+iC5uCRx7w09xPgXLhrdgBkDHXH/ZP9+yRk7A9yZs6tpLHigfSmfMkL0nO17mdfDfxowNP7V9+PPM7xfDNAA3qsT0sRpC4osHfkVZns0W5RK+PPcalz/eCCKkacP6nph9U4cnwPWuCiCVi65wuqBQj41jHA5cTZAcz3lgzOAYNpE9F1jyI3Q7vz+yAqeEBzNIneiDlE4drtPfHl5XNm4vzGHo+34KZ2bIeWp1SCdKyWDyoEo/nb7k5R55rm+WSPb2t7eDIiSqDQ1vh9b/yatt4j+RgJpc/NBjv+tSP2tRQJo80PeAjZIephEJ6cT1W4g+6rm/wQSaxU4JgcgquR7b+oTQG+EHm6tJ2ooPK6jMlfLKMKOr5E2TifasozrODKgKCj1T5UUVZC1mV3sTWNOTzOVWrlB2Mmd1wO5ZSnD5ysT7mnHMTamWlXvksy3e7y1Gcu+DT9m1x9WA/bbjd2Efx2G2mnRhymlvJrPg0x86NOQ882kdNCCDj+FmZsLqntYvtZU3DnuRzkwDo+bX9+EvG0mD0Uf94tCMI+VtV9hgiDjNkrUDWKJhDku1S0yiMxZFP4kxVG1lF0lpPqfc/h2M2xoLMUfz3TrO3rqhLBF1H9kZE3ZlWw9oxzjKHB/MIZpi17x9wx8ZeMFN2bNqFsX9qrndny927s/cpbZgS4AR4/M15OERNwyG iDRMxUu8 7eXroJSNNdD3KVwKleL63qCSGFB+VZgVhtoLpXWukl36uf0dEyIe3uQWLBaVy2pky43mvgz1Dzv9b3Gzwq75I4Y/EtG3VxSPJzVTEMb+vRxmQzSaH6cERjRV8xjiZrKuEWF1s5NpoBbKYgLlklthIHiRfOw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 19/10/2023 4:25 pm, Chuck Lever wrote: > The SunRPC stack manages pages (and eventually, folios) via an > array of struct biovec items within struct xdr_buf. We have not > fully committed to replacing the struct page array in xdr_buf > because, although the socket API supports biovec arrays, the RDMA > stack uses struct scatterlist rather than struct biovec. > > This (incomplete) series explores what it might look like if the > RDMA core API could support struct biovec array arguments. The > series compiles on x86, but I haven't tested it further. I'm posting > early in hopes of starting further discussion. > > Are there other upper layer API consumers, besides SunRPC, who might > prefer the use of biovec over scatterlist? > > Besides handling folios as well as single pages in bv_page, what > other work might be needed in the DMA layer? Eww, please no. It's already well established that the scatterlist design is horrible and we want to move to something sane and actually suitable for modern DMA scenarios. Something where callers can pass a set of pages/physical address ranges in, and get a (separate) set of DMA ranges out. Without any bonkers packing of different-length lists into the same list structure. IIRC Jason did a bit of prototyping a while back, but it may be looking for someone else to pick up the idea and give it some more attention. What we definitely don't what at this point is a copy-paste of the same bad design with all the same problems. I would have to NAK patch 8 on principle, because the existing iommu_dma_map_sg() stuff has always been utterly mad, but it had to be to work around the limitations of the existing scatterlist design while bridging between two other established APIs; there's no good excuse for having *two* copies of all that to maintain if one doesn't have an existing precedent to fit into. Thanks, Robin. > What RDMA core APIs should be converted? IMO a DMA mapping and > registration API for biovecs would be needed. Maybe RDMA Read and > Write too? > > --- > > Chuck Lever (9): > dma-debug: Fix a typo in a debugging eye-catcher > bvec: Add bio_vec fields to manage DMA mapping > dma-debug: Add dma_debug_ helpers for mapping bio_vec arrays > mm: kmsan: Add support for DMA mapping bio_vec arrays > dma-direct: Support direct mapping bio_vec arrays > DMA-API: Add dma_sync_bvecs_for_cpu() and dma_sync_bvecs_for_device() > DMA: Add dma_map_bvecs_attrs() > iommu/dma: Support DMA-mapping a bio_vec array > RDMA: Add helpers for DMA-mapping an array of bio_vecs > > > drivers/iommu/dma-iommu.c | 368 ++++++++++++++++++++++++++++++++++++ > drivers/iommu/iommu.c | 58 ++++++ > include/linux/bvec.h | 143 ++++++++++++++ > include/linux/dma-map-ops.h | 8 + > include/linux/dma-mapping.h | 9 + > include/linux/iommu.h | 4 + > include/linux/kmsan.h | 20 ++ > include/rdma/ib_verbs.h | 29 +++ > kernel/dma/debug.c | 165 +++++++++++++++- > kernel/dma/debug.h | 38 ++++ > kernel/dma/direct.c | 92 +++++++++ > kernel/dma/direct.h | 17 ++ > kernel/dma/mapping.c | 93 +++++++++ > mm/kmsan/hooks.c | 13 ++ > 14 files changed, 1056 insertions(+), 1 deletion(-) > > -- > Chuck Lever >