From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 41A38EB64D7 for ; Wed, 21 Jun 2023 10:15:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B1BCB8D0003; Wed, 21 Jun 2023 06:15:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ACBD08D0001; Wed, 21 Jun 2023 06:15:24 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9BA7A8D0003; Wed, 21 Jun 2023 06:15:24 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 8B5668D0001 for ; Wed, 21 Jun 2023 06:15:24 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 59E471208D4 for ; Wed, 21 Jun 2023 10:15:24 +0000 (UTC) X-FDA: 80926347768.23.40F782F Received: from mail-wr1-f49.google.com (mail-wr1-f49.google.com [209.85.221.49]) by imf09.hostedemail.com (Postfix) with ESMTP id 40043140016 for ; Wed, 21 Jun 2023 10:15:20 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf09.hostedemail.com: domain of sagigrim@gmail.com designates 209.85.221.49 as permitted sender) smtp.mailfrom=sagigrim@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687342521; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ufw3XdlG8f50ui3w7eQOIJRzxVEkWkebHT8qYe+C8qE=; b=tiWKoYuoFGkLagFlKp6LI+acvVDfRBAe0gAp+o8tvfqH35SAzqJ7YGv5ctYuIdAcsU2Nx7 E7K/QHGJ7G7LUfo8IzMxc/V/9M/cbcJAUVOPg3qeTtjNW9OeKg7k2aDxLu0chBHrXamfcF qT66qJ6BMKXvcwB8l3XppwOVEF5ukWk= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf09.hostedemail.com: domain of sagigrim@gmail.com designates 209.85.221.49 as permitted sender) smtp.mailfrom=sagigrim@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687342521; a=rsa-sha256; cv=none; b=CmNZeKdTBATNuMq6DProsmVm16tYEPtCltFdFpV2ng6FYV1LUTS1emnxFbH2ypjI+f+f7L S64UL/Vlh8aVFC1pDv0KIK0umBw+wdHKCONr8JoDk2p0vyYUhLfPm67P630H83V9aihiGS 93GrZ7xADJhTt0hnQHqnSSaBnkX2VPU= Received: by mail-wr1-f49.google.com with SMTP id ffacd0b85a97d-31114d47414so1276132f8f.1 for ; Wed, 21 Jun 2023 03:15:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687342520; x=1689934520; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ufw3XdlG8f50ui3w7eQOIJRzxVEkWkebHT8qYe+C8qE=; b=jrE1NB816RHY+g7N3IEZHH0L+1W/aYb/AOvHWRsAeK/Rwc5OxRThAkz2fsZscbQZ5y dTyeTrviTZWc80jbH7JO/RnMQ8vOja5olRrQUrIJJAyyTm6cgNNLshfmrDS0+jKDhwJR ttGxRmlE45mI/rl9aqDFrNtechv4TtniVy2sj1KH5m2+cKkVeVRv9N63eMgRagx9VAyH Qc1I43UdCSDLDpZibnx5K5Pph322ucG1KxWX75vXM8bToKYbKZBnqNzdhiMBu2/UnYaH XwKX1G5VzqL1X7PKfhnKKv8CgSop9WSa+Pj8wvxo10ozj3RT7zp+ZVnPY+zTe0VbxS1N T3wA== X-Gm-Message-State: AC+VfDzuhT4u9D3ZMpuelh+gCrTGTj8jSh+L0eAM/qGRMQQ2Pbv3lCL8 G3tE3COGaHsczliMl1LBX48= X-Google-Smtp-Source: ACHHUZ4tsTabGuEP0pusOoHjithDqeGTXPlhvE5IPuJ0eWoLH8lfxNjVcT1Icfws8enUiCQS9OC42A== X-Received: by 2002:a5d:468d:0:b0:30a:d0a0:266e with SMTP id u13-20020a5d468d000000b0030ad0a0266emr14265990wrq.2.1687342519535; Wed, 21 Jun 2023 03:15:19 -0700 (PDT) Received: from [192.168.64.192] (bzq-219-42-90.isdn.bezeqint.net. [62.219.42.90]) by smtp.gmail.com with ESMTPSA id p9-20020adfcc89000000b003113f0ba414sm4082968wrj.65.2023.06.21.03.15.16 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 21 Jun 2023 03:15:18 -0700 (PDT) Message-ID: <87f547b0-7826-b232-cd01-c879b6829951@grimberg.me> Date: Wed, 21 Jun 2023 13:15:15 +0300 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: [PATCH net-next v3 10/18] nvme/host: Use sendmsg(MSG_SPLICE_PAGES) rather then sendpage Content-Language: en-US To: David Howells , netdev@vger.kernel.org Cc: Alexander Duyck , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Willem de Bruijn , Keith Busch , Jens Axboe , Christoph Hellwig , Chaitanya Kulkarni , linux-nvme@lists.infradead.org References: <20230620145338.1300897-1-dhowells@redhat.com> <20230620145338.1300897-11-dhowells@redhat.com> From: Sagi Grimberg In-Reply-To: <20230620145338.1300897-11-dhowells@redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 40043140016 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: wd7wpdkf45anynof9c8b49oypcjjw9uu X-HE-Tag: 1687342520-289887 X-HE-Meta: U2FsdGVkX1+WqzkkM+d8PIoUp67KFIMWSEw0D8+j2rxudChLC4VQbqMBHe0d8qTkAa0M+grWYw1g0hCly8mjICRLDYNg78Urt39/07HxmHFcwXfbu+sbXqQw5fM0aV8HAml+cdsc/78MeaDXZ6zNgXLmOyUT4gtAXm6rvf01AM/wxZGzhxkQV0Ym+fcKpb2WossM+gDkh8FOSehSf4wYFdIgRwB5kRHs2T8FAxcap+7VvLpWAcggn3InhJbmplzR0D2txdghWC2zRBHjosxvUQPCdhSYf7yRdXpSJ9VJDhvjhgbFHrv45Apb7lFFfKMv8To1n9XfFLZAHhO6fS6mOXBjg0kfdV1DVPLf1P8i3DKy3uxfwIwslRKB4IW+gkdrkLBh71MarJM6vBGdsEVQPuWMgu2aUiwKDvGhKrnY81RVcAy3ihC6ZzeXdhBvlzK+in8tPZehyH2mChN8FsvgqoM5LblIzTvVLaPaeM0Nk8HjKT22ke7ZuvveEeNN0ozeoIpMestLlABnHg2XTBWQfkbGDUawoPs1QtotTQS2B5pn38Vtb3uC8ulL9sWbrzhJlhG/cEpX4rjc5SdKq9qBKOEVdQZsosHE0SUL4ieEkStE67PvsNGeXYkGALaACBTx1rTx219VwBiqcYp53n1CPz78jqR9qtOoglUXLfLmMSeU23pigaoGZCpsPXMTijUbydtXDgfc5uMqRkV6GK/h+NyOXrHAU2P2anMzm1XjJFKQQoMk0MZ9nvRnetBeb6qa2+cRCEVhs86GgDZcU/Yno6mTQbas70mvudaVD+xIatR4n3Rrd7CmMABG+SChcqB84K/wtZ5yZ6hQmIj9gGjgxxMzFyMKvBG9xImH3PvxCLYvvheJBMuq6KuKylTKHgNQ5B4p6vIP2bnLRRbJl0qxXmItvzNSLJJc6/YDyG72w81nThQ6P2jiOITpu4wGPn9Y7MGmZ7NL/qFgw0X1k3s V630kC0C /tkb3YV/maebiCWHyHggaUnYotHLfMsZGo3CSyQqlQuCuAUQJY1N8vW8y5U0hwsMCLlayVWMmw0ekVV1WQLP9vlJlrpq+4sdB7qFAjwPycka4hOTnEc0mEaoIi2hF/Ph9XPR6aFpi7AjCEUo81UQLYQwJpfVBGBFWTf5SDRIWwiQM0CiDB1l8Bwh6oBv9SMTaL0Wf/o+fFt1fN7JhjPcoaLhAYNbSDI+KRUS9Z2MrM8uCsfyQPiJ4heVqh+vIU1ZnGothv9i4Dkn6BIcsnMJZ3/it+3JWYJOTeprHOr10qU7bJMuPfLZJX9C1RAEaeF1koh/f5XphWS6IO0nv6AN4hSLMltdUfIOts0qT2Lxv6xp4wh/L47FmUxKnotUE6SO8zWK0ZuQwYhetHPVY2/1mW0qXRdycebp2E3+vXAeT8ctlODaeaTVHypAiF8uRg650JuvnP48Lf/ZcsYWuy2WkKsYwJdmGJWzh4MBPtUmyqasHaVJIXibS1+3R9qzIatXxJvoGkMxJ0C1JMYpwxeD1jzI1Nwo3Kh9snbbha0TIr1hb3wvyXyRqwfaPSln8+RWrTh+FRo0DFZd+s2bNwHmPJ6M7Xf4whIVVqr1PjyVJam21aXhL68FJosflEV+1Wa3r7r0AbnxWthTrf4INxGCt2f+6U0I3ebz6oOQtCVLpKIb2hFK0Ck+Gu9of6L1Xn/YZpeZ6KmecG/7iKAqdS1RflIjlYwZp+TGGjNrmGjaRqS/w6YXn8K6ymfDJKZxGUN3S5OMc7x4Iq1wGc9iQeRM9Vc1VrdEDEBkcN2LQTnFQr/6CYUIaYOECm6VIiKdY7vEa1vWYEjT8PZkxPVKO+9f9vXYP2ND7IqV+gd44+Ms2t8onq0BIMWJhjQPsc18ijDD36HZ1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: One comment: format for title in nvme-tcp is: "nvme-tcp: ..." for host patches, and "nvmet-tcp: ..." for target patches. But this can be fixed up when applying the patch set. Other than that, for both nvme patches: Reviewed-by: Sagi Grimberg What tree will this be going from btw? On 6/20/23 17:53, David Howells wrote: > When transmitting data, call down into TCP using a single sendmsg with > MSG_SPLICE_PAGES to indicate that content should be spliced rather than > performing several sendmsg and sendpage calls to transmit header, data > pages and trailer. > > Signed-off-by: David Howells > Tested-by: Sagi Grimberg > Acked-by: Willem de Bruijn > cc: Keith Busch > cc: Jens Axboe > cc: Christoph Hellwig > cc: Chaitanya Kulkarni > cc: "David S. Miller" > cc: Eric Dumazet > cc: Jakub Kicinski > cc: Paolo Abeni > cc: Jens Axboe > cc: Matthew Wilcox > cc: linux-nvme@lists.infradead.org > cc: netdev@vger.kernel.org > --- > > Notes: > ver #2) > - Wrap lines at 80. > > ver #3) > - Split nvme/host from nvme/target changes. > > drivers/nvme/host/tcp.c | 46 +++++++++++++++++++++-------------------- > 1 file changed, 24 insertions(+), 22 deletions(-) > > diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c > index bf0230442d57..6f31cdbb696a 100644 > --- a/drivers/nvme/host/tcp.c > +++ b/drivers/nvme/host/tcp.c > @@ -997,25 +997,25 @@ static int nvme_tcp_try_send_data(struct nvme_tcp_request *req) > u32 h2cdata_left = req->h2cdata_left; > > while (true) { > + struct bio_vec bvec; > + struct msghdr msg = { > + .msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES, > + }; > struct page *page = nvme_tcp_req_cur_page(req); > size_t offset = nvme_tcp_req_cur_offset(req); > size_t len = nvme_tcp_req_cur_length(req); > bool last = nvme_tcp_pdu_last_send(req, len); > int req_data_sent = req->data_sent; > - int ret, flags = MSG_DONTWAIT; > + int ret; > > if (last && !queue->data_digest && !nvme_tcp_queue_more(queue)) > - flags |= MSG_EOR; > + msg.msg_flags |= MSG_EOR; > else > - flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; > + msg.msg_flags |= MSG_MORE; > > - if (sendpage_ok(page)) { > - ret = kernel_sendpage(queue->sock, page, offset, len, > - flags); > - } else { > - ret = sock_no_sendpage(queue->sock, page, offset, len, > - flags); > - } > + bvec_set_page(&bvec, page, len, offset); > + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, len); > + ret = sock_sendmsg(queue->sock, &msg); > if (ret <= 0) > return ret; > > @@ -1054,22 +1054,24 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req) > { > struct nvme_tcp_queue *queue = req->queue; > struct nvme_tcp_cmd_pdu *pdu = nvme_tcp_req_cmd_pdu(req); > + struct bio_vec bvec; > + struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES, }; > bool inline_data = nvme_tcp_has_inline_data(req); > u8 hdgst = nvme_tcp_hdgst_len(queue); > int len = sizeof(*pdu) + hdgst - req->offset; > - int flags = MSG_DONTWAIT; > int ret; > > if (inline_data || nvme_tcp_queue_more(queue)) > - flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; > + msg.msg_flags |= MSG_MORE; > else > - flags |= MSG_EOR; > + msg.msg_flags |= MSG_EOR; > > if (queue->hdr_digest && !req->offset) > nvme_tcp_hdgst(queue->snd_hash, pdu, sizeof(*pdu)); > > - ret = kernel_sendpage(queue->sock, virt_to_page(pdu), > - offset_in_page(pdu) + req->offset, len, flags); > + bvec_set_virt(&bvec, (void *)pdu + req->offset, len); > + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, len); > + ret = sock_sendmsg(queue->sock, &msg); > if (unlikely(ret <= 0)) > return ret; > > @@ -1093,6 +1095,8 @@ static int nvme_tcp_try_send_data_pdu(struct nvme_tcp_request *req) > { > struct nvme_tcp_queue *queue = req->queue; > struct nvme_tcp_data_pdu *pdu = nvme_tcp_req_data_pdu(req); > + struct bio_vec bvec; > + struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_MORE, }; > u8 hdgst = nvme_tcp_hdgst_len(queue); > int len = sizeof(*pdu) - req->offset + hdgst; > int ret; > @@ -1101,13 +1105,11 @@ static int nvme_tcp_try_send_data_pdu(struct nvme_tcp_request *req) > nvme_tcp_hdgst(queue->snd_hash, pdu, sizeof(*pdu)); > > if (!req->h2cdata_left) > - ret = kernel_sendpage(queue->sock, virt_to_page(pdu), > - offset_in_page(pdu) + req->offset, len, > - MSG_DONTWAIT | MSG_MORE | MSG_SENDPAGE_NOTLAST); > - else > - ret = sock_no_sendpage(queue->sock, virt_to_page(pdu), > - offset_in_page(pdu) + req->offset, len, > - MSG_DONTWAIT | MSG_MORE); > + msg.msg_flags |= MSG_SPLICE_PAGES; > + > + bvec_set_virt(&bvec, (void *)pdu + req->offset, len); > + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, len); > + ret = sock_sendmsg(queue->sock, &msg); > if (unlikely(ret <= 0)) > return ret; > >