From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F25CC369C2 for ; Fri, 25 Apr 2025 13:21:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E2C7A6B0005; Fri, 25 Apr 2025 09:21:19 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DDCE66B0006; Fri, 25 Apr 2025 09:21:19 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CCCD06B0007; Fri, 25 Apr 2025 09:21:19 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id AD8E16B0005 for ; Fri, 25 Apr 2025 09:21:19 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C70901CFBB1 for ; Fri, 25 Apr 2025 13:21:19 +0000 (UTC) X-FDA: 83372627478.23.7163FC2 Received: from verein.lst.de (verein.lst.de [213.95.11.211]) by imf02.hostedemail.com (Postfix) with ESMTP id BBEF480017 for ; Fri, 25 Apr 2025 13:21:17 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=lst.de; spf=pass (imf02.hostedemail.com: domain of hch@lst.de designates 213.95.11.211 as permitted sender) smtp.mailfrom=hch@lst.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1745587278; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=eRgyWaTCnXTJn4ybjkzyDK/I4iuHXsx7IJBf2P1l8tk=; b=FqorILSIeuLdt7IWEwB9RmijfWUqEAXXQJq1e3oW4yY3Z8tvM/HwZFWoMgQpmhzbQH24MQ iEFzCEv+IQdqGMsOYbRgvGZ0duexFFn/ZkzILSZGG0dkaxnSkXtDCq0BjXspsQuemYynmr 3Jk1FX44YZqd97+EuOF/tc+KFCCf+q4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1745587278; a=rsa-sha256; cv=none; b=JHQd6gxVCWE5Z37PgfpQhEb1R2MVNMeV39zCcpWHcIKNdix8PaOToHd+TdmE3yD8RLUrg5 1SsZQ2qhFYx3HKyiZI01R0CnXH/jG0IsHWaFFd9oHj8WIJAtGf+bNLPwMsr745KGH0tEyX Tt9c2VXI4AgxcJBVCU+udn+mQ/GgMis= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=lst.de; spf=pass (imf02.hostedemail.com: domain of hch@lst.de designates 213.95.11.211 as permitted sender) smtp.mailfrom=hch@lst.de Received: by verein.lst.de (Postfix, from userid 2407) id 596F968B05; Fri, 25 Apr 2025 15:21:12 +0200 (CEST) Date: Fri, 25 Apr 2025 15:21:11 +0200 From: Christoph Hellwig To: Keith Busch Cc: Christoph Hellwig , Caleb Sander Mateos , Jens Axboe , Sagi Grimberg , Andrew Morton , Kanchan Joshi , linux-nvme@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: Re: [PATCH v5 3/3] nvme/pci: make PRP list DMA pools per-NUMA-node Message-ID: <20250425132111.GA5797@lst.de> References: <20250422220952.2111584-1-csander@purestorage.com> <20250422220952.2111584-4-csander@purestorage.com> <20250424141249.GA18970@lst.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.17 (2007-11-01) X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: BBEF480017 X-Rspam-User: X-Stat-Signature: tu4yauy54godwmzd8se8ug9sxjzfoasi X-HE-Tag: 1745587277-893565 X-HE-Meta: U2FsdGVkX1/HRGGJ0jQKuTHDL58hK3O/GwXGvpMoWX0bTSTdZPNRcrIFaNtPWGt28t4ncwnY/Ygeu0BehLR22YguXC9FvFYT9XxfjZwF5kVm2J+ZhRVYHhDUNvVSlJ4gIyBTwNOlqA8kEzN+JrJng+OFynNktENw5plnehsSFCwMY/Qt2qD434SszBl/V59/D8MDsHk0ihJUqZBHGTTanGF5GBeijht9LYS3sbM3AT3EbsQwQd7dnjsvGnDPEJDgAu/yUQREvUmEkGns71T8OZhpcacwRumKhN21llmGIFspLBP2ju+GTpfl1fSQypREbt0bscqvLtIO//QEzYBmOkcsuy53X/zVkLLw6rzWDaEY1f4010eJJUxbSTq9nFHH2xq6tf9tT4g8MckhzILDsW70S0WsN0NE+iycyzPoy2UCDe1Y0wczag5UjK+C0aXbu7bq70GglGBN4nybX4EcmOa6vm/beNan0j3jdRJ8n/XahPxIQJVwT7M6XGVwZwHRpSWY1aeSgN27Fu3bcoOb41euwPH2qxL8rRBS76KS25tLRA9BH7RgRMTOahgL9mIuE0RlrwA/5P6cpDw9/QGXN0p7tc9L8c/DVaRv4wbVQyCU2Y+kYCSwh+OzHLVuhcSt6bUMeVRnDrSVOVC+KSR9MTZzYKOz0Vj8lW6CIx2Mfr+MRXzGjRexg/vdj5n49V+oCuCt4WvPzHIvYKpLgBXxILoZZd1IsDT8rYGz4dm9i8rsxWy/JNiGP7k7XM1Z+4vyluZhyMLOX+rgC9Pd5kCCjyvLq2Rw/O1wOZPgKmBKbcgfZH+//a67qLZIe25nei0pqK+Q0ge223ZNh4/3QzkoAfdVv/c599uOZ0Qs5ZPQrJ7xYvJvoIlCboYwh/Bm6a6040OHrVKfH4df9LhHXTCAG7VSygORQjQuC+9W6GL6NhkccTStUyxbp7X4GUdrPTQGSzXZoMcTawjyACdvV/M 8p8PtVm8 mShIts8qw8CU6AceRkJj5x/JzmlQylRa57UR03Ot5vtR+Wfsgj1awMklQQfT0aG/AOa3EqeTRpSRUoXAH1myjJvj/0gzvmdYPcVgt X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Apr 24, 2025 at 09:40:18AM -0600, Keith Busch wrote: > On Thu, Apr 24, 2025 at 04:12:49PM +0200, Christoph Hellwig wrote: > > On Tue, Apr 22, 2025 at 04:09:52PM -0600, Caleb Sander Mateos wrote: > > > NVMe commands with more than 4 KB of data allocate PRP list pages from > > > the per-nvme_device dma_pool prp_page_pool or prp_small_pool. > > > > That's not actually true. We can transfer all of the MDTS without a > > single pool allocation when using SGLs. > > Let's just change it to say discontiguous data, then. > > Though even wtih PRP's, you could transfer up to 8k without allocating a > list, if its address is 4k aligned. Yeah. > > Should we try to simply do a slab allocation first and only allocate > > from the dmapool when that fails? That should give you all the > > scalability from the slab allocator without very little downsides. > > The dmapool allocates dma coherent memory, and it's mapped for the > remainder of lifetime of the pool. Allocating slab memory and dma > mapping per-io would be pretty costly in comparison, I think. True. Although we don't even need dma coherent memory, a single cache writeback after writing the PRPs/SGLs would probably be more efficient on not cache coherent platforms. But no one really cares about performance on those anyway..