From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 859BDEB64D8 for ; Tue, 20 Jun 2023 13:01:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D21778D0002; Tue, 20 Jun 2023 09:01:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CA99D8D0001; Tue, 20 Jun 2023 09:01:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B231D8D0002; Tue, 20 Jun 2023 09:01:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 9E1A78D0001 for ; Tue, 20 Jun 2023 09:01:10 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E2A5E120EAC for ; Tue, 20 Jun 2023 13:00:57 +0000 (UTC) X-FDA: 80923136154.24.521D5AF Received: from mail-wm1-f50.google.com (mail-wm1-f50.google.com [209.85.128.50]) by imf02.hostedemail.com (Postfix) with ESMTP id BADA58003E for ; Tue, 20 Jun 2023 13:00:36 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf02.hostedemail.com: domain of sagigrim@gmail.com designates 209.85.128.50 as permitted sender) smtp.mailfrom=sagigrim@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687266036; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=X93F7b0wOWdtuJBRHnqORIf5mXHGdp4ICFY23Klr0r0=; b=U0rjppHQGtHynnwYDWm9YmV2unNuUDtxWYq2GCyXiF80XS+/pcLSArSP2eCKYDQoItQIwb cWhNezU+QA+zlBZ3vL/ha68uTt/qHJ3QrZBMcx8+2lHQ0vwY0cQNamFd/cCG/DJYOvwPp7 ggyiVhRRusFBITv9tRRCAFWwJvE9Db8= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf02.hostedemail.com: domain of sagigrim@gmail.com designates 209.85.128.50 as permitted sender) smtp.mailfrom=sagigrim@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687266036; a=rsa-sha256; cv=none; b=hgE2Bx5v+MZ+Y/wF0ICfcmx7ZyE5FanZHfRB+BvF+xCJ8XkViLz2AnhcCPc9qEXdSZg3jk E/zZh0C/d9Bm6Z2AQNMaqmVCkht+BalOUG63Q0hNmK0ppA1RownBIl3Mpbt4GSD94TWoBB k4+38pmeaCxq19E2qgoroxu0bHmjvFI= Received: by mail-wm1-f50.google.com with SMTP id 5b1f17b1804b1-3f9b38e4e29so4357095e9.0 for ; Tue, 20 Jun 2023 06:00:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687266035; x=1689858035; h=content-transfer-encoding:in-reply-to:references:cc:to:from :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=X93F7b0wOWdtuJBRHnqORIf5mXHGdp4ICFY23Klr0r0=; b=JAQzMQC3Lk6j7NUhIG6QldYrTLWkCxsl2Y6gdsv8qhbdPjOnv98Tdl9+Pu4F/wQGgI PX0jJeFFWUVU5dJt6tCGHmYLPMIC+zmcnDknHQK3ckQHSv2EC3Ec+ySHKEEfQySJ3UBy duPiXNK6t4xyIHXlN9mFDIlWr9u40IiESJE91nB0hosL4Hj67o757PdK68UMnE7hXoXZ Cqvdnh4PXnW1oItaOico7gNVJAaEQLHu0XcicIU4U8mln72JP9FVwP42nNvTz8wkwg+k dwwFFayrNa5THSisaCZtC5wCEs0Db3YwqtnRpTYtFO58oVY7IXhrKarlwQ/cEY8ARXyS xDAA== X-Gm-Message-State: AC+VfDxb258yBwkKz+ADgnDc7iK7T6vBzKTcUSz9L/pwnqz2L/jGzv38 Kv2p/Vrq2Pck8DIbKq8Uejw= X-Google-Smtp-Source: ACHHUZ4fMadoVOeylQzjAgw+8ckeVgibkoIHqT/D3sxLYyMEY5mGEhxW4ZiDHDRruH9UX+wLysw2Mg== X-Received: by 2002:a1c:ed17:0:b0:3f5:f543:d81f with SMTP id l23-20020a1ced17000000b003f5f543d81fmr13305521wmh.3.1687266034561; Tue, 20 Jun 2023 06:00:34 -0700 (PDT) Received: from [192.168.64.192] (bzq-219-42-90.isdn.bezeqint.net. [62.219.42.90]) by smtp.gmail.com with ESMTPSA id z8-20020a05600c220800b003f9b12b1598sm5339723wml.22.2023.06.20.06.00.31 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 20 Jun 2023 06:00:34 -0700 (PDT) Message-ID: <028fd6f1-6fc2-6e8d-3de0-71c2b8f6b754@grimberg.me> Date: Tue, 20 Jun 2023 16:00:31 +0300 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: [PATCH net-next v2 10/17] nvme: Use sendmsg(MSG_SPLICE_PAGES) rather then sendpage Content-Language: en-US From: Sagi Grimberg To: David Howells , Willem de Bruijn Cc: netdev@vger.kernel.org, Alexander Duyck , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Keith Busch , Jens Axboe , Christoph Hellwig , Chaitanya Kulkarni , linux-nvme@lists.infradead.org References: <648f353c55ce8_33cfbc29413@willemb.c.googlers.com.notmuch> <20230617121146.716077-1-dhowells@redhat.com> <20230617121146.716077-11-dhowells@redhat.com> <755077.1687109321@warthog.procyon.org.uk> <55e7058b-07d0-3619-3481-2d70e95875ea@grimberg.me> In-Reply-To: <55e7058b-07d0-3619-3481-2d70e95875ea@grimberg.me> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: BADA58003E X-Stat-Signature: 36x5u1tzxdt5jsjf6mm54i3toboocc33 X-Rspam-User: X-HE-Tag: 1687266036-634701 X-HE-Meta: U2FsdGVkX18H1Xa77dJU/MT10GwQAe0zbukNkDNbrU8ACTFpkCgsNesbBqJyPzp2NcaybOPi1Gdn1ACN3LsGHrbYseWj6Mus5DMYrtVabwu4mqUaxgX3hkUKcbHtbfFn0dCVXOlKaFbYpbOeCj6xrsIapANTk/nh4uZc1rVjZtOAJmZqqtPGfDJyTdDjLvJ52VChP3Q9axO7AfUAKRUzMeXykd/KZupwqGq1bDXAhg+IRTDYDqZ4wRGBO68LCV3EYrSVZoaDvOipP2Pi9fyWOIcUuL/OeYHuncj4N4g2yZ8j3Dv7oWtg8aq7hFhR7TwEj/dO93jKkOnHykcDLCrLnnFiMM8x+d5MB144Tgz1w+UFBe+aZ2CFZRHji0MO1FSWP/w0TyU+5m8r89uTBn2oKD0n33sZpCD0NMQchu3RduB4/MGVMAZlUDHjBx4s17XFpmz3KAgodwEgkl9wxeZAzDW4SauTixlfuwiVffLwgr637XbRFvw2BU+Q3qg3nOUg5PbYNUrz2U36HdL3L3UfinJtAa3pGHOoBoQGm2O+8bDCuoMnjJWsuHyrGOLt+Y3L55+sgbaraD+4hnAojz7FMxLdxelE5DHFU7IkTFmHQK/G6fE9kw9NwZhiCX84KhoBKVHjtiXbO5ZXvJpmEYGPpFMAVHRaMY9dsDZ++ojSptLkBfE4MQpSWWyfhGlqyeY1GQzIeKAIMj0OZY+RXXzww7bfy3hhn+fWQqdu715HxXx1pgNT8H07a4PZbYzNcE7EpbIYL25Dx5oXEtY+NLOziTLl0N9nBEjPUhlFUJYkRJMVDXTBsc2Me973uwSyWdc/8nOV7rmzCRGz9KsdM0USzd8BnFbgm533F7bSjGd5TBbq2JaZTQg+W0rv6JjkUrIfeDWSfKFjNbZjDwP1DHCs42LGqcUMD8gCsChUVENC1pp3uUp9Gr91sPAOkbXbVUQ0zQCdOj1sSKE4LxzCeNV Mp88pKpq dc5IFasTJHE0zZbXJQZNgBw9CMKITrF9QZkxFir3zjsr1G9D0lFL/ryLvkSXQbalaIIJVe8GGAED9cSXUiIFnHnx6h88bXrJ30knx5WDpzE7bQOCd4EFuvceHGH3R0ONSFKyF6GsZozXWCCwWr9l/1QTtwHZvNLcyTBWQKUV7YGYQmWBfFGT7bIpQvXUhSHuSz3KeMcSdve8hqSEl6Ur5mkqi4wywHtPeZzEbMz4TWhABQ8kEMWHJqEX5SzpCYiSG7We1IO/LR8uGamxtT8nK0NzIzVCmmCc1pcb+GsKLCKMw5kPD3ccODKD65oT6fs0CD5jAeShDfnej3PmvMOZjgh/wDBhlezNnnaOuv183baJ5qokdYt9ETqieW8Ijvgd3fjyYXmD8W7T/4yPSPOpUXfUMkM/uXTWxb8Uf61RkvidJa8LnvuIWYJqSjkAOQ7Jq6Fnj+0wRqMo18IWzb7/J+yezHX1n9f94BddIMenvBMbx2e8S3tpszp1zvw9la0TZnJmYXCnj+FCTpbEymJo16saiaJLpkixGLfPhQ6OWeReqGOlOIPUb9GjNuJUam8c21wBG/X/Fz3sebyB5QsTztaNw9iWK/CUDOvYEVFI1K+oFKEarE0j7nYRmWGikWY6227y5 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: >>>      struct bio_vec bvec; >>>      struct msghdr msg = { .msg_flags = MSG_SPLICE_PAGES | ... }; >>> >>>      .. >>> >>>      bvec_set_virt >>>      iov_iter_bvec >>>      sock_sendmsg >>> >>> is a frequent pattern. Does it make sense to define a wrapper? Same >>> for bvec_set_page. >> >> I dunno.  I'm trying to move towards aggregating multiple pages in a bvec >> before calling sendmsg if possible rather than doing it one page at a >> time, >> but it's easier and more obvious in some places than others. > > That would be great to do, but nvme needs to calculate a data digest > and doing that in a separate scan of the payload is not very cache > friendly... > > There is also the fact that the payload may be sent in portions > asynchronously driven by how the controller wants to accept them, > so there is some complexity there. > > But worth looking at for sure. > > The patch looks good to me, taking it to run some tests > (from sendpage-3-frag branch in your kernel.org tree correct?) > > For now, you can add: > Reviewed-by: Sagi Grimberg Patches seem to hold up. Tested-by: Sagi Grimberg However if possible, can you please split nvme/host and nvme/target changes? We try to separate host side and target side changes in the same patch.