From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6CA6EC54E4A for ; Wed, 6 Mar 2024 15:05:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CDACB6B0074; Wed, 6 Mar 2024 10:05:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C624E6B0075; Wed, 6 Mar 2024 10:05:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB58A6B0078; Wed, 6 Mar 2024 10:05:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 95AE76B0074 for ; Wed, 6 Mar 2024 10:05:29 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 6F914120FBC for ; Wed, 6 Mar 2024 15:05:29 +0000 (UTC) X-FDA: 81866937978.24.369C7BB Received: from mail-qt1-f175.google.com (mail-qt1-f175.google.com [209.85.160.175]) by imf18.hostedemail.com (Postfix) with ESMTP id CC4731C0028 for ; Wed, 6 Mar 2024 15:05:22 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=fP4yl2Rs; spf=pass (imf18.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.160.175 as permitted sender) smtp.mailfrom=jgg@ziepe.ca; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709737523; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3q6XaWzdNlm5rZVTCa+MalaN5VSyURFchooXo3LCyRw=; b=o/aRCTu0UnQmTDC08GaunAauONb6ANO+MvbYG0Z9+oKJitqUTsbUf1e22lc6ziWLA19389 Ilj2TlqNTUA7npxUtHQ/lIawh7AcqVelbo/LC/IHVFQFb00PP4bx+bfNoggVr32biLIz4l rpOhgclHM4IqHy5pAP95zbk1R9Hp1po= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709737523; a=rsa-sha256; cv=none; b=rZ3FvkvMntEk2+o2AH2XRxy+Dw5m7xFXAphQ8qi31qWfWvCQXfNX2KAlbgqgzNM2LVVwqe mRX61XuXqjlfLR/RaIWvJTcRtr3OSmQ2Hqu+OZNFSAD5IgQEAcytaBl63a+WPKyQb9BOpP l75Tto0ZdVz816C/HNT4smsiZ2nekEU= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=fP4yl2Rs; spf=pass (imf18.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.160.175 as permitted sender) smtp.mailfrom=jgg@ziepe.ca; dmarc=none Received: by mail-qt1-f175.google.com with SMTP id d75a77b69052e-42e8e85a969so37096651cf.2 for ; Wed, 06 Mar 2024 07:05:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1709737521; x=1710342321; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=3q6XaWzdNlm5rZVTCa+MalaN5VSyURFchooXo3LCyRw=; b=fP4yl2Rs0PxEQg3DytduSYV13DWJ8BetUMkuG4gT7FRYpoNrn12nJMFApeK6bbbvYw IHW8B8RoWw0nSwA12dDl8BXmXRF1/ERgyCVUQeUNaW72J/zmywYaJd1BbdSRH5GeLyRj 14TwreaiblxRPlVI40tiFE4Wf54bV9JUmWGdUQhar6GFpvitBbP6gK9VH9Dpzy1LOKXy bz9mUIoDn5cfehtQvwl14hPEyJmDEXyu+N4OWsLmRndl9fpyQMo4BSNvHEbiIJ+yNYYO JdIV9hug2xmAwY9SUz0LJ+wFlsKEZPfQneNF6FZr7/5JjdHNGTATD++E8Z9euqHb/mCg nn7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709737521; x=1710342321; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=3q6XaWzdNlm5rZVTCa+MalaN5VSyURFchooXo3LCyRw=; b=HDwgC50MIIQpgqGRWNAUsrp77JX2lCIRKnM1eHMh+jPhgtQe6wwQe8w+kjsZq3MuRa AFvfNI7RZAa6IhET8nbs8HfV4ZdzYNdi3WKYFkK5lzX17F/L4ZDLkjTWd8BtC5DcXghv Q8X/RO2l8x1Nc/kz+9ZNVoysI1jEqrgn0OljMVh5H/Kmzr5NC6sgqBvyo5C8WqUhP5q9 VYUKvQQbhsu+wwRPEtZSa4plAfiYa9Hgfve3J/eo9RIz8kR5/Zi3/v+1O0XesncHEs5c QvP1RnAgD8XBfmpqs0I06oDCoCuYhZFmr8eGBHs8rBrMfr4zLt7x7DtEDq4hQUVYpmB9 ZRsg== X-Forwarded-Encrypted: i=1; AJvYcCWWssJE+jfTGBfzB2rdDRAt+frJKBCORx/ug9l4mx1ukG5E2qURrdKVfyeANQZy1FqUOJ3in/RlfAJeRl0tdJsUOkE= X-Gm-Message-State: AOJu0Yxp0BKivMIjCtgoGJh7GP2PtE5PnJ7aCGdczC+ziqiLkhVrOUBE kOF+nEyc/Gv84BJseXt63KbHqmtvyWY0odwIHjLka4Kh1o/QTu4c3uruyc4W4Fg= X-Google-Smtp-Source: AGHT+IF3DD5YVqRayIYcxbWGOreOFw/mqikGG0c1CCT98Poz3twNOMiPHe1g+1Vg/YON1gYeWp1F9Q== X-Received: by 2002:a05:622a:1743:b0:42e:bcc4:2959 with SMTP id l3-20020a05622a174300b0042ebcc42959mr5571780qtk.4.1709737520530; Wed, 06 Mar 2024 07:05:20 -0800 (PST) Received: from ziepe.ca (hlfxns017vw-142-68-80-239.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.80.239]) by smtp.gmail.com with ESMTPSA id f5-20020a05622a1a0500b0042f01390d5csm1686793qtb.30.2024.03.06.07.05.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 06 Mar 2024 07:05:19 -0800 (PST) Received: from jgg by wakko with local (Exim 4.95) (envelope-from ) id 1rhspa-001WNB-Tx; Wed, 06 Mar 2024 11:05:18 -0400 Date: Wed, 6 Mar 2024 11:05:18 -0400 From: Jason Gunthorpe To: Christoph Hellwig Cc: Keith Busch , Leon Romanovsky , Robin Murphy , Marek Szyprowski , Joerg Roedel , Will Deacon , Chaitanya Kulkarni , Chaitanya Kulkarni , Jonathan Corbet , Jens Axboe , Sagi Grimberg , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?utf-8?B?SsOpcsO0bWU=?= Glisse , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, kvm@vger.kernel.org, linux-mm@kvack.org, Bart Van Assche , Damien Le Moal , Amir Goldstein , "josef@toxicpanda.com" , "Martin K. Petersen" , "daniel@iogearbox.net" , Dan Williams , "jack@suse.com" , Leon Romanovsky , Zhu Yanjun Subject: Re: [RFC RESEND 16/16] nvme-pci: use blk_rq_dma_map() for NVMe SGL Message-ID: <20240306150518.GL9225@ziepe.ca> References: <016fc02cbfa9be3c156a6f74df38def1e09c08f1.1709635535.git.leon@kernel.org> <20240306143321.GA19711@lst.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240306143321.GA19711@lst.de> X-Stat-Signature: 5pikpp4i5qmdcx4jriu6zj1p4nk7h4bb X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: CC4731C0028 X-Rspam-User: X-HE-Tag: 1709737522-57559 X-HE-Meta: U2FsdGVkX18l9iXmkq3YwXsWzLXYLcnWIGZqBnm341U9Ek29OgtHsSzniMTbWWH0XZ0zQmF92MFen+yVXXZzs2GBAYMc575rWv33YD/b4CxA+ZDlAOEnc/b1PHjjaK+nTAbjnNiTi13TWQ8NwYy7TQSQCJBb/vIgSd0v7UY12oGIZzt0HJKJmh/Ha0jbFqlItaWdMx6AE+/4DjFCseu3+cX2Rv35Dbt7RzfZ3IJFinJeXBy/2z1uWxgTOB+7eKQok+3z2Y2eBKHwLvmz4cMBA0XIteAsJghKzax6tqWvuHyVHgjjlkVvFDSI5imDTNfIHsm2o1YBzr5PwWeUkEWLQkbOgV4lT1cpcqRgBGzdIyPepyX6ViAT+rFY/Qr/knWdKmFB2k7XqGx4WFZ4YXAEBNUAO/2d8vNiM6tTcpWlc1geRVfZdqMTjiRZIKlGPTZD2YOFTHJdcgKfo/9MTev6gM4xMkShfDTRnb/6x1rHAorS5ZQZWGSxoKCaqQuTUBMgip0vEoDLbb6NeoNn9Fi86fl55Ohf9AjwHYtvl/M+VI/DsDUsq13xp8hArucIRceyCV+N3RGHLkvF3wnwTwG+TggJdBncp157lT7BFPUggHwK4BbrC7sewCpgJnBNm+XAIfD5LH5MfkjNyqQuAk4VpJMZqm7eWPQmjLBppXp1Kqu0PyLcXJzNjmBfna6E2aLTyB0oPM//jjiqrd6wE2lGfabGrVMYb+toKgYPYNqVcGaMWkC/4DbNrMGwclMQt+wN0DzvD/7XIswmAnbNuyVs8zsoYrJDKX7g8BPa0PjmQ6f90Iq2Y1zPHQM54MtLQv0a729vQJYyzZqQtTt5IkRQi18dqHmVQzrUstrsLNBHyl+4v1/2g7CNgXLoLwZgeqIzLVIf7e65Z2ZEBYKqafH8zKzRhwePRE+4BpWz4TZy4ExlmUQgJ9ZkVLWe6F6jfcTa2v3EYPyIdGehEjmQ15B GXYG7kFi hnX4k5ySfY0FsSwm7FCIHvSuJq+KJj43HlNecab7nbaUlqLveY/Eov62ULPnaJXMVxjj8wvSFkTnfmNky9gbvmy+D1MsrdMIq6y7+fuJNmMtHEirf9BDEkUh1jxYK0+dKEQw+UIZW/W7ox7fPuvSqvdg6sYnRmJmB6lEwqlOPzXI7UtxwDAty9KexDa755dyjr+S738J06ECjhxeta47nD08eTHCNZs1PtrX1nKGvQoFjet1400C2l2P1DubHRwKEQ2f+oyceVDRf2UBanCjQngFOS6X0RMntG9W3tFSZEstRiz7ldmEniwK+qwrK6OwIGAkMAm0JKmLqsWjkz/lgjpUV+e+VpK75wygCRl+YiVV8TWST4eocJKNTiWMgwFX4jNmlPbJTVGqLpbRtHYrmDT8SBYlw+zws3rCA X-Bogosity: Ham, tests=bogofilter, spamicity=0.000042, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Mar 06, 2024 at 03:33:21PM +0100, Christoph Hellwig wrote: > On Tue, Mar 05, 2024 at 08:51:56AM -0700, Keith Busch wrote: > > On Tue, Mar 05, 2024 at 01:18:47PM +0200, Leon Romanovsky wrote: > > > @@ -236,7 +236,9 @@ struct nvme_iod { > > > unsigned int dma_len; /* length of single DMA segment mapping */ > > > dma_addr_t first_dma; > > > dma_addr_t meta_dma; > > > - struct sg_table sgt; > > > + struct dma_iova_attrs iova; > > > + dma_addr_t dma_link_address[128]; > > > + u16 nr_dma_link_address; > > > union nvme_descriptor list[NVME_MAX_NR_ALLOCATIONS]; > > > }; > > > > That's quite a lot of space to add to the iod. We preallocate one for > > every request, and there could be millions of them. > > Yes. And this whole proposal also seems clearly confused (not just > because of the gazillion reposts) but because it mixes up the case > where we can coalesce CPU regions into a single dma_addr_t range > (iommu and maybe in the future swiotlb) and one where we need a I had the broad expectation that the DMA API user would already be providing a place to store the dma_addr_t as it has to feed that into the HW. That memory should simply last up until we do dma unmap and the cases that need dma_addr_t during unmap can go get it from there. If that is how things are organized, is there another reason to lean further into single-range case optimization? We can't do much on the map side as single range doesn't imply contiguous range, P2P and alignment create discontinuities in the dma_addr_t that still have to be delt with. Jason