From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E099FC369AB for ; Thu, 24 Apr 2025 07:15:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D5D166B0006; Thu, 24 Apr 2025 03:15:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CE4506B0007; Thu, 24 Apr 2025 03:15:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B5E766B0008; Thu, 24 Apr 2025 03:15:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 959C36B0006 for ; Thu, 24 Apr 2025 03:15:53 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 81A7DBE784 for ; Thu, 24 Apr 2025 07:15:54 +0000 (UTC) X-FDA: 83368077828.19.6905B42 Received: from sea.source.kernel.org (sea.source.kernel.org [172.234.252.31]) by imf23.hostedemail.com (Postfix) with ESMTP id CD56514000D for ; Thu, 24 Apr 2025 07:15:52 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=aBTgU71u; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf23.hostedemail.com: domain of leon@kernel.org designates 172.234.252.31 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745478953; a=rsa-sha256; cv=none; b=1gvipnFsY9zd2qYr6ayeLMwomAGiq8pLKHEcZD1Ve+AiRwCFtrA1Z9npXAoD96tvFMrBno vsbVBkJtTLUkVavAWYAlSh+vrWlUX/xay7yycukRo4VpIlD4FaRN1BdgGn7QkIFcfZ8Nrj M4lM+mvGeP0SE284X9nrTcWMeSsPZ5c= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=aBTgU71u; dmarc=pass (policy=quarantine) header.from=kernel.org; spf=pass (imf23.hostedemail.com: domain of leon@kernel.org designates 172.234.252.31 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745478953; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4ZAQxJmOsFsTTi0D+tSRQfOw+J2rVVKEfb31WKljAbA=; b=6ZaTMCfBhXNWe5DGtOnTmTZRU4eYXUbvZxKtpHujk2gh8CARKBokOVM0IQNl94r95boQy/ ks71o8mETX9+YehhLFSjUfqOMd+GyFaxwPxMS88q80D4BztbEbmuo9rOSMjSbn2ijd4ue5 HYFl+7MJlVb/rtqtV4onHkCwAfZsxvg= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by sea.source.kernel.org (Postfix) with ESMTP id CF48E44CBE; Thu, 24 Apr 2025 07:15:49 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8B8E3C4CEE3; Thu, 24 Apr 2025 07:15:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745478951; bh=pTkF/OxNxP3a5nhNTZxPZtH9LiokbdzIPY4lcClh0Q0=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=aBTgU71uDQ76gGQRWfJMAVuYX3A0QbPaLJvEVuFRBRrxPSIlRhqbBqclL9MgN42Mz mOyKpbfujbO9d4jcdRZl8T1Go5m+UqQxIRaX0tVYAWP8gY0N23/sDzG80Ns8/xrjOx yRFGVRmXxsj5fmUE6A3mSSt4ikS5jl7uqbR1Q9jdp7zW3uELJvckJZvlrjbWQLAFMI rlKfp1i/mcS2aG+fPOR4bIoU7h9FpaLggeY2LHT/xOyqKlGS/6YJVHuNgwdRNVVRsH wu0LFFiidEyDynwIeD0OFcSLI8I/quvZGtTqBj02iqPtvdHAUsLiUcS2TnnGA7c1LM 7VSAaFBMPFnUw== Date: Thu, 24 Apr 2025 10:15:45 +0300 From: Leon Romanovsky To: Jason Gunthorpe Cc: Marek Szyprowski , Jens Axboe , Christoph Hellwig , Keith Busch , Jake Edge , Jonathan Corbet , Zhu Yanjun , Robin Murphy , Joerg Roedel , Will Deacon , Sagi Grimberg , Bjorn Helgaas , Logan Gunthorpe , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?iso-8859-1?B?Suly9G1l?= Glisse , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, linux-pci@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, Niklas Schnelle , Chuck Lever , Luis Chamberlain , Matthew Wilcox , Dan Williams , Kanchan Joshi , Chaitanya Kulkarni Subject: Re: [PATCH v9 11/24] mm/hmm: provide generic DMA managing logic Message-ID: <20250424071545.GM48485@unreal> References: <3abc42885831f841dd5dfe78d7c4e56c620670ea.1745394536.git.leon@kernel.org> <20250423172856.GM1213339@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20250423172856.GM1213339@ziepe.ca> X-Rspamd-Queue-Id: CD56514000D X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: ka7dmirdguwzeeudm67hqxrdhqjcmw8u X-HE-Tag: 1745478952-335128 X-HE-Meta: U2FsdGVkX1/aTAkPJ2iykzmEyQ/cxQNjDiO/iN1wbY8nttVAzhUhZvv/lvV0Qf3tzidrGMas4+uoRBRHYBoWuqEHQmKAgrTg5ISO9muHer5K+vaW+7mk74AxcjhxUVhT35W29acbeb+YfVuuKTnU40a97LUudbRdX+DTOGV3kt0w9yStFv21Y7T5AZN4W9HjIdov+WshKkDVyxitGkucVM8uvCokb/DdHcR9xYZH/FR9hEBqUTWT5LFewj8LxxLomQYJTMOIcrPoTYLQEt7PvQvWycnB8oozv56jWyMq4o3Wg26YvkEEzxVvMJN9B1OZq0QP5SAS1tmkd0uohcNhBeY+ED1L5FHUWsiyZ6EnqCaoehe0Yrp43LOFtKSgIpz6SVPLMP+kn8DDckEp7cbiEd4vrh4k/scwf3H3AKlcqj6Qo28EQZ9jafpkeXbn0W0vrHCAembWQ/0nsl1E+J7WNGWHiXBSHdl+uIjit7Gs8RKZm6TOPt5Wp+MWYt/04siRdzfrW/N/mB4Cz3JcgWyUWm+Dvu7wsAloQVyMQPSsX/LPPkOriqgIynyubCD1qtwRZ4vwdzVg3HtZPFnZ3lpoiJ/mLXOMZTvXw7UgcRwOG+NQ5dYOwiXn06SAMkT8KxQVnmFLeoEZaVKGqbYKosk/manL+vNJc1PAx3POCe8sLLbNDjIoHS7wh2sjHnPuoYVE1o4Ae8w+Nrq+no1Qm7/VmXnwMmX97+7P/HkOkA2nsgjW8TBQ1ysODT7ZGI21BgwhWFSvo4xIYoG4X3h7ix0MBv9H4XMN89SYe5PyZtQJOeGrPpUtY2Dyn0l4tMDXEvgO7xrVWgEjTq5QPXwceuH9F5377n15AWYVk0uXiPYKpkre25BrSnNwiMlP2s+yNM+nyA4LrBoW9SS0lUpQav89i2Lq2T23Oe56pPai8A88x11xUN70k5lBIJPIyu/Lld0cz5r690UlgiElS920UxU kIWKL1Bb QBrnSYy33YUhaF7M1Vt4TxxscvO9GlaPgwEVSBaK4dnD/ohvxqBay9d7iabhxqcEGjYoeUljPvoB6emEkdAP0602gpt+MGs/j9H6GbnWke/OrRHKlkcvpMocd7L+uXhseI8uin7/gu0eBLGvFtRttSDkTcl2Utb+y4Fq7JhD1KK/UVLjYXpTcbRFG8R5cGLVPKc5Tl+v8mYwo1EN3f3zsZ2qjbvkQQZhVnvRTbF0m+L2R5R09eWtA++37s4CTTIvACDzindOM2YHFCZ12eTt3Uh7OjA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Apr 23, 2025 at 02:28:56PM -0300, Jason Gunthorpe wrote: > On Wed, Apr 23, 2025 at 11:13:02AM +0300, Leon Romanovsky wrote: > > From: Leon Romanovsky > > > > HMM callers use PFN list to populate range while calling > > to hmm_range_fault(), the conversion from PFN to DMA address > > is done by the callers with help of another DMA list. However, > > it is wasteful on any modern platform and by doing the right > > logic, that DMA list can be avoided. > > > > Provide generic logic to manage these lists and gave an interface > > to map/unmap PFNs to DMA addresses, without requiring from the callers > > to be an experts in DMA core API. > > > > Tested-by: Jens Axboe > > I don't think Jens tested the RDMA and hmm parts :) I know, but he posted his Tested-by tag on cover letter and b4 picked it for whole series. I decided to keep it as is. > > > + /* > > + * The HMM API violates our normal DMA buffer ownership rules and can't > > + * transfer buffer ownership. The dma_addressing_limited() check is a > > + * best approximation to ensure no swiotlb buffering happens. > > + */ > > This is a bit unclear, HMM inherently can't do cache flushing or > swiotlb bounce buffering because its entire purpose is to DMA directly > and coherently to a mm_struct's page tables. There are no sensible > points we could put the required flushing that wouldn't break the > entire model. > > FWIW I view that fact that we now fail back to userspace in these > cases instead of quietly malfunction to be a big improvement. > > > +bool hmm_dma_unmap_pfn(struct device *dev, struct hmm_dma_map *map, size_t idx) > > +{ > > + struct dma_iova_state *state = &map->state; > > + dma_addr_t *dma_addrs = map->dma_list; > > + unsigned long *pfns = map->pfn_list; > > + unsigned long attrs = 0; > > + > > +#define HMM_PFN_VALID_DMA (HMM_PFN_VALID | HMM_PFN_DMA_MAPPED) > > + if ((pfns[idx] & HMM_PFN_VALID_DMA) != HMM_PFN_VALID_DMA) > > + return false; > > +#undef HMM_PFN_VALID_DMA > > If a v10 comes I'd put this in a const function level variable: > > const unsigned int HMM_PFN_VALID_DMA = HMM_PFN_VALID | HMM_PFN_DMA_MAPPED; > > Reviewed-by: Jason Gunthorpe I have no idea if v10 is needed. DMA API is stable for a long time and only DMA part should go in shared branch. Everything else will need to go through relevant subsystems anyway. Thanks > > Jason