From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8CDCC54E41 for ; Tue, 5 Mar 2024 16:08:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3700D6B0081; Tue, 5 Mar 2024 11:08:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 31FA76B0082; Tue, 5 Mar 2024 11:08:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1BFFB6B0085; Tue, 5 Mar 2024 11:08:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 0A9306B0081 for ; Tue, 5 Mar 2024 11:08:51 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id B4E6C1A0F1A for ; Tue, 5 Mar 2024 16:08:50 +0000 (UTC) X-FDA: 81863468820.14.26F0021 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf20.hostedemail.com (Postfix) with ESMTP id A901A1C0003 for ; Tue, 5 Mar 2024 16:08:48 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=ysa0yBs1; spf=pass (imf20.hostedemail.com: domain of axboe@kernel.dk designates 209.85.214.172 as permitted sender) smtp.mailfrom=axboe@kernel.dk; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709654928; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ySXwHaWnwWfuqJedGiV+eti2ui1J2NF0RPAqqwKPf6w=; b=KuAueB/G0yCztMvsLyNqbS+rSCf816i025WxFKzscHcZt0J4QPnfLfn+jFJb8Vrk2LeZoW RYzVxLN03CnGBIM8QthngzRLJz/bHCXhKn3KCqa/Zzo2te0RYqauL4SXuqZYGm/KXrohCj N/kRhiwZ9JLaAd8voDiXN1IuSdZ+2Co= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709654928; a=rsa-sha256; cv=none; b=3peyRNK58+bXGMZOEDzQy7G80WmIsGa+WeSO1tjo99V9lgnJix9YuKB/hH1q2wdaApYTL0 WGmtwzdCeFkccqYIhM9+m0MbWmj3XwGrUUGcfyoRXcejUfOLmsssxniqRKw4rj4kw/SXnY ule3hRHI/FnKDoqM2WAADf+GAFKBheU= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=ysa0yBs1; spf=pass (imf20.hostedemail.com: domain of axboe@kernel.dk designates 209.85.214.172 as permitted sender) smtp.mailfrom=axboe@kernel.dk; dmarc=none Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-1dcc0d163a1so17925245ad.0 for ; Tue, 05 Mar 2024 08:08:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1709654927; x=1710259727; darn=kvack.org; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=ySXwHaWnwWfuqJedGiV+eti2ui1J2NF0RPAqqwKPf6w=; b=ysa0yBs1PNRuNdq1Bs09lTPl4e3fwF6MPG+xOb0gClbP4BLYySmKHPaJJ9r7jIsiBS F8WbW//kVJ8rdMxFvwdI+FMJVEEEIhUdXRxBkrwQcbKM3ek3pa7TNQZhu3CqDNRnHalE CzoLgT5Xi9I0kCfDBz9sHs5AsBpGUCovK1XCHnQuOJovkGhvR2bIRivD2ttL55DQp1bw 0kcmsUgJ5fMwpPL2dOjHAqDNaYEt3FeUNfiP3VOUIppHeWaTf0HnXZ2pk/jlBvkdMBjG vnnZxIkAL3c2Uk6Qtl0NhDji6uAl0mvdL2SL7yMBW/ntHxT0HDfS47Cuc5l7QJB0qtPx YwfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709654927; x=1710259727; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ySXwHaWnwWfuqJedGiV+eti2ui1J2NF0RPAqqwKPf6w=; b=muVpRN12sj7CWo4JKI4h8fP3Su/lWnuOfJJ1EhJrKgn+0XDCrs1lZTMSa/bdFPrapT ex8okOMr6QjbQqtuvgrzaolHipsMKP6NQBk36TwFSjwDPyacx10kxWfryeMHZIqGa83u qbzRoBmH4edkeTYGkRmt2q9GG9ipyvvuABGQOLznx+y/d+YKdlANIw1NSiK6wPtmYdoc UTJeqi9WCJNWCQThWtLMe6C/gF6fIuC9k5bFxUojre/eidyFuc3QsbFzkgLYzRTQFSmn ZcjBhqB1Nq9j0SGC4w1czM0Xv9elixypI39LyV1z6aHg//n2osuzZihumSzaOuU1z/ZG Xmtg== X-Forwarded-Encrypted: i=1; AJvYcCUAg/LsDUHtIJCzJzcs6kgquJNWki1rhdkNt7o9QvbabuypfdZwLbGwF5rogGFKtYLmwLdKdg4YZKxOZgqKj02EsVc= X-Gm-Message-State: AOJu0YwQw6AMKWARF2uCqPbnlylUye16L37lwXk2Vy+Wzff3kouRndkR 75nfFiNwr+qTlZbJ7JrFDKef6pjOrJ/cX2sMO5aev2JH7BkJL73jC415sGIvJKA= X-Google-Smtp-Source: AGHT+IEQyMNsi91T5s32CUzK+ybKvXY5CY8IcxZgQ/czWV6ISp8aawkZe242UlnjfOr6MXQ5ygWb4g== X-Received: by 2002:a17:902:b08b:b0:1dc:82bc:c072 with SMTP id p11-20020a170902b08b00b001dc82bcc072mr977677plr.1.1709654927270; Tue, 05 Mar 2024 08:08:47 -0800 (PST) Received: from [192.168.1.150] ([198.8.77.194]) by smtp.gmail.com with ESMTPSA id b4-20020a170902d50400b001dcf93e90a0sm6901815plg.20.2024.03.05.08.08.44 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 05 Mar 2024 08:08:46 -0800 (PST) Message-ID: <06787e6a-4e78-4524-960d-ec24b9f38191@kernel.dk> Date: Tue, 5 Mar 2024 09:08:43 -0700 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [RFC RESEND 16/16] nvme-pci: use blk_rq_dma_map() for NVMe SGL Content-Language: en-US To: Keith Busch , Leon Romanovsky Cc: Christoph Hellwig , Robin Murphy , Marek Szyprowski , Joerg Roedel , Will Deacon , Jason Gunthorpe , Chaitanya Kulkarni , Chaitanya Kulkarni , Jonathan Corbet , Sagi Grimberg , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?UTF-8?B?SsOpcsO0bWUgR2xpc3Nl?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, kvm@vger.kernel.org, linux-mm@kvack.org, Bart Van Assche , Damien Le Moal , Amir Goldstein , "josef@toxicpanda.com" , "Martin K. Petersen" , "daniel@iogearbox.net" , Dan Williams , "jack@suse.com" , Leon Romanovsky , Zhu Yanjun References: <016fc02cbfa9be3c156a6f74df38def1e09c08f1.1709635535.git.leon@kernel.org> From: Jens Axboe In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: A901A1C0003 X-Rspam-User: X-Stat-Signature: ebz8d7cfqux38mn1s5h8eoghk7xnb7qu X-Rspamd-Server: rspam03 X-HE-Tag: 1709654928-285621 X-HE-Meta: U2FsdGVkX1886zlBOwE1c5uUgFC4cptwxVERbPaCK0ux9kjbrAIVxpIDiB+Yg4hezjVPG6QapX3Lgu0QMM55Be3bP/j9r3Oni8LuDlEUf3EGXnZOZMmGM4jThatcEwIVldLb0YV98Va6ujJUhQ2aY3N453S7mJegoCOT1GowojICmOOoPOqBohcV87J7iRTXBI5nuzbGiuVyzyvzvyzqNXiDLZoKQPPLZmuC2FjhxpAQkEGUFeEcjLyOXPS2JyKNTjiMr7DoT1zBNKtrYc/jC/SsOuMwd4+kAL4KO2mTSp6pSlytKRN91fXWTJ1GbQlAOvr5Y4POD78yj/wsF/ZGlJM/bEPy5bfPabE39XmgUyhpUwrwcjhulTB96idKgClYUsZQyWIGwfOVMpxOVUJU09Rw87Vl9RYVP6YF4DnX5n+Yh9qgzv3EhRLFQUpnyhQYTISsc+2HjlZjP4DLO5C8+Qq8EdhyRxVO8McrJvapMIXo8wMNxXDNvJyXbkgjTbcn9dgI8wB95XEbvjZEL8v3U/rrB358gtPuEZlGskKCySP567lxuACd5S24oOoTOIX+vprV2DkdyO27AOY6Ga9WWArtAJ95fWYgpOMR4zRBbPoFJx17PH/fzuaTra0sRVN9FV5bZLWBIC40YW7bW+CBIAMjMiRvzkEx1aW36CuOYBr/3cFyvD1EMEIHeC7CUCSFNezdbVPk2JJelhgWqLAHg8PMdNjcebOj85+bDdI7nJcDQN5cOeJ8liV80U113KmKIrxlBTYunqwyDzwiu9U8WGJcD+r8NtndqvnPGdMoi054ZLvyBs7X7/BMwAnoGhjk8EzAv2GSM8KTTPptTSy0w4A/LDW5nF9YaJv2fu+9hL6Ykf3x+EBpyVIcLoZXzThX4YlJPsfdJwnb6a5nNlqbDDb3HdjEHu9CRcmmVAr7klh/JxyiJLSFJgHULeYftdKfElD3GhP9U5i2veKWmtO q0ixBE9r fkDFZzYW9oVPQDcWTxRAseuD5lAY9qQ6VWPkzIYscVK2aGvF5JieUCIZCdPumwbnrQ3HN6VhblwHAiGJQz2JLDDUYX+D/OoXe9AnOBdqa+CEm035IkZ3ZWgT41LWs52nkbPxuu3haLdge5WVs/xRaGo1IQE8yZsAU4aG+j9xN9YEoQFIVJtuQKuq2MrdCpSgOM2/7f88cQDElk8CjneOCX/xPVBDJdm8KnU+pnpJYUe4e9qj6s5ugBVxLJQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 3/5/24 8:51 AM, Keith Busch wrote: > On Tue, Mar 05, 2024 at 01:18:47PM +0200, Leon Romanovsky wrote: >> @@ -236,7 +236,9 @@ struct nvme_iod { >> unsigned int dma_len; /* length of single DMA segment mapping */ >> dma_addr_t first_dma; >> dma_addr_t meta_dma; >> - struct sg_table sgt; >> + struct dma_iova_attrs iova; >> + dma_addr_t dma_link_address[128]; >> + u16 nr_dma_link_address; >> union nvme_descriptor list[NVME_MAX_NR_ALLOCATIONS]; >> }; > > That's quite a lot of space to add to the iod. We preallocate one for > every request, and there could be millions of them. Yeah, that's just a complete non-starter. As far as I can tell, this ends up adding 1052 bytes per request. Doing the quick math on my test box (24 drives), that's just a smidge over 3GB of extra memory. That's not going to work, not even close. -- Jens Axboe