From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DD3BC369C2 for ; Fri, 25 Apr 2025 18:03:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A099A6B000E; Fri, 25 Apr 2025 14:03:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 990466B0010; Fri, 25 Apr 2025 14:03:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 830866B0011; Fri, 25 Apr 2025 14:03:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 602BB6B000E for ; Fri, 25 Apr 2025 14:03:00 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 8AC00160477 for ; Fri, 25 Apr 2025 18:03:01 +0000 (UTC) X-FDA: 83373337362.03.2918375 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf21.hostedemail.com (Postfix) with ESMTP id 08DAE1C0019 for ; Fri, 25 Apr 2025 18:02:59 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Rgkfw+BQ; spf=pass (imf21.hostedemail.com: domain of kbusch@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=kbusch@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745604180; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1CfGXA88R8hMMyIBt8aOhC7AURfnCYMI4bHM+e4cOiU=; b=ube/DgAeWbViZMOuaZZ6xojIYrRbEi7cpsJBK9F9yhv/ErbM7oxaLStEo7lzoMHpWK00nl a/IPtIyjm7wnQ4KPz7Zq8wEh2p3iIU3ovpkZHPnFzQwRklNEAdmGjn0U5XW/UH1q8xab95 r00ZxlyXSZdvEbzLVTA751MgIxbsRR8= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Rgkfw+BQ; spf=pass (imf21.hostedemail.com: domain of kbusch@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=kbusch@kernel.org; dmarc=pass (policy=quarantine) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745604180; a=rsa-sha256; cv=none; b=S86oriIPIVLBJv5heQTNwH45Ve+DHl8c0iS5UN4z1Jm7S+9fYGxYaPLnBEA/g7vQlVbMVs lnp4TKUG/EY5MfVIPHbQ4+k1HSR+wlRFJzL79lgCqXDkwFt4fD+3xj9/VaK7KVrs6XFzZe 1ebTVs8vBLk0k1Ir4+CxwoWxkcH/QwU= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id D80745C6A2E; Fri, 25 Apr 2025 18:00:41 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 0EFD1C4CEE4; Fri, 25 Apr 2025 18:02:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1745604178; bh=GlHZ3nQIl/lKAjbzyAfvCDkSdKESLA3OEmyBcQjujck=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=Rgkfw+BQTICZthivqCvV/L24rzpWQHbDcyxggr1y+m/v43rfeZ8mVc0gW9um7fv0b 8UafxsdBy2OSnbMDS5NFMohqw/JUMNlm1lLeRM9bksCbnWhkpvG1sMaT+pD4u+ffRH 8pIUlmyLfoSlvzGXqDwnfiebUuADE431RCYOd6tbrbLbvBjoynuBlQ5zFLUwzx+Bi4 rGux6Foo2lHXO20hT1/pHPb+gHQz9MKCOsluoedgGVWhhJf8tGKuOY5k2+/cT7G2O3 Lha8hXq2f86SbjO/kQE7iwnWqluHmT2utJrQpEDxaiqrLaVadzaMFYa480cXg0cBjW /ZIf/+bOW5TJw== Date: Fri, 25 Apr 2025 12:02:55 -0600 From: Keith Busch To: Christoph Hellwig Cc: Caleb Sander Mateos , Jens Axboe , Sagi Grimberg , Andrew Morton , Kanchan Joshi , linux-nvme@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v5 3/3] nvme/pci: make PRP list DMA pools per-NUMA-node Message-ID: References: <20250422220952.2111584-1-csander@purestorage.com> <20250422220952.2111584-4-csander@purestorage.com> <20250424141249.GA18970@lst.de> <20250425132111.GA5797@lst.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20250425132111.GA5797@lst.de> X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 08DAE1C0019 X-Stat-Signature: 6y5o35pjw1wiz8wbqxzajoq13uerhoyp X-HE-Tag: 1745604179-637727 X-HE-Meta: U2FsdGVkX1/GeaRUYPsjt50SZSrqPCk35+HKM/arqRI2DJXaNvNNcajHwiL/m//AYRvzLIWPnN6jUzYUHQJKTe1mYNwb8RGwB5ExcmPTXboIrlMauuvlBh2G9Z3Bwv8EDOvNtFV0PXW7Sd/6N8oriF4JijB1pa7HEAALOTqF+UcsFettdjBltFFHRWUQ6XHABft0NjE9cha2nYKXCukaKA0J4zeenZOYbAAf1xc2aNH2+0Ar2uWfpro12aq7sQfgefyTnitYhX+1pMpDXz5l4HC1ohyqPRp5NtwRO8QOBBNpb3RdV4RUV3Q+gpBrQACKfhvydYZvfqBs4bS0Ejny23hAYocGViniDjo3XkOufZseTRpigwOi94PD2H8h7P3Q5mUiFuS9IP52nGU6KeF2oNPVLQ1JZb3LXRG2YZpIfUxHlVu8rW7A3auFlATQPDgzXgRvbNU3B+nd9IVcJeCqDTbvOwSRr8E70cxBUCWD2JMbx2usaPi9n5GRD0H0m0p4s9PWtiuN+lz/53E2N5DOXwnOfKCQbU6yO6X7qWk5mEyKbRvqyv4WRJaTEnzXl3uElOpqdoB2lU9waJEKZi5nizD+Ri78PROn9a9LMOQSa8KUpwOwSoepjN6ihBcEg//7RlxvbNpkYZMJCgpwc/LNWvxeOxRN1xs6UleHwuh+CHC49wE20jKVAU8zgNTh+2qID7vlkzUwsz3Lo+usYC9Ba4Ram9gRQ9HBdn/KIyk7VznUD3/LPmU5Nbg9FYbyrJ4JvyIIyt1zrdQc1peiZzIVDma9Bm4ytgb77YNzylYvWqT3J9MBXQc9AtP7b/I4+mHLkw1Xm9tB7CYXOMSoCeJeIb3JScvbh1MGdWIMGhlWOCqxhlY7YlMJ386I8R2uTDq//g6z5CQWMAk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Apr 25, 2025 at 03:21:11PM +0200, Christoph Hellwig wrote: > On Thu, Apr 24, 2025 at 09:40:18AM -0600, Keith Busch wrote: > > The dmapool allocates dma coherent memory, and it's mapped for the > > remainder of lifetime of the pool. Allocating slab memory and dma > > mapping per-io would be pretty costly in comparison, I think. > > True. Although we don't even need dma coherent memory, a single > cache writeback after writing the PRPs/SGLs would probably be more > efficient on not cache coherent platforms. But no one really cares > about performance on those anyway.. Sure, but it's not just about non-coherent platform performance concerns. Allocations out of the dma pool are iommu mapped if necessary too. We frequently allocate and free these lists, and the dmapool makes it quick and easy to reuse previously mapped memory.