From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A7D5FEB64D7 for ; Sun, 18 Jun 2023 16:48:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 347C76B0071; Sun, 18 Jun 2023 12:48:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F7F76B0072; Sun, 18 Jun 2023 12:48:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1985B6B0075; Sun, 18 Jun 2023 12:48:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 0A5B26B0071 for ; Sun, 18 Jun 2023 12:48:00 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id C44821602EF for ; Sun, 18 Jun 2023 16:47:59 +0000 (UTC) X-FDA: 80916450678.24.865306B Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) by imf07.hostedemail.com (Postfix) with ESMTP id EA34A40015 for ; Sun, 18 Jun 2023 16:47:57 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=Cfj0sc49; spf=pass (imf07.hostedemail.com: domain of willemdebruijn.kernel@gmail.com designates 209.85.160.177 as permitted sender) smtp.mailfrom=willemdebruijn.kernel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687106878; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VPsd3RxvSK9akU38J5qW4D3wDnVV1+TTrRbY+MUugj8=; b=oNKPrau3H0eVE35+m4qjsHyrF/L2vlkIYpKSLKFzQxozObOl/Qe63u74NayGuYDjwUnjYq 2ittP55svl1DsQWULgi73mz7J9KzISpxiCiSYBdnk2cOIWRGqJbFv+heTKpyjptyiHHxzY eW9+Uu+97tIWDQGM815I8UHKFuvAqNo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687106878; a=rsa-sha256; cv=none; b=0MMC1doRDHsiL2111HzPHyHnuuUL1LfhxuGap5r6dAZmyeksGv9B1YuFHJY8YM8gx7i2mG DwHff3Kzlr3ExKFvlEKlfl33hF14p4d5QVBWjT13VKwrIBSzrV0ye+Bx/xMfp9m+Qmi50W H34LZxhyZ8VeposjmdV2nBL2RcjlE7Y= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=Cfj0sc49; spf=pass (imf07.hostedemail.com: domain of willemdebruijn.kernel@gmail.com designates 209.85.160.177 as permitted sender) smtp.mailfrom=willemdebruijn.kernel@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-qt1-f177.google.com with SMTP id d75a77b69052e-3f9e207f5f0so16799031cf.1 for ; Sun, 18 Jun 2023 09:47:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1687106877; x=1689698877; h=content-transfer-encoding:mime-version:subject:references :in-reply-to:message-id:cc:to:from:date:from:to:cc:subject:date :message-id:reply-to; bh=VPsd3RxvSK9akU38J5qW4D3wDnVV1+TTrRbY+MUugj8=; b=Cfj0sc49fEerL20hzENwz8PtcItwwSbnaClH5NS/pVHzmM/dvwLvhEaT6FYCNl0dmR OKA9zNJyQSpEsphH2tw/CmB52MYtdWASeDtaWkyUm3LzO0GqY0RFWBEMNwQKgRY1gte/ hFMC/AS9uLs82RUEb9AvmdAHt4dHGLIlvQtI1Qr2qE9QAJ1KEPOoriAEOnrDHQkQ+d6+ gYyzxf6y0xuGX7Gw8XirNiB3R4dTioKlXECEhHMm60fL4/KIEzIlvrrC3HawxafPHOPi CkcKx2T+h62Q42tysZQtLYJz8kH1cDimH+ATSgv62GnlbyeLcPaoPKDyV/U0EqAyz9T6 O39w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687106877; x=1689698877; h=content-transfer-encoding:mime-version:subject:references :in-reply-to:message-id:cc:to:from:date:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=VPsd3RxvSK9akU38J5qW4D3wDnVV1+TTrRbY+MUugj8=; b=AgRizsFb2sDQmQ83scxiSNckX5LxllDQXHP8UpPSvutyR1JUXf4hlXzPFCyEQyDiTB PXPFvog1LmpqeHv47ysTjdgZ5n8epF/12TOCPcKTb/RyaxN0tUY2ebj2sQ2QGsE/xVo3 UxhZTtsw80ps/ML3h8lhaR+gpg69rgyxhZtxXq7vcaGtUiQVuB7jUjejVVvflZqfrY2t N+xL3I8dOOsngqMOO6CURZ09h4CscyKlvUuLEB4gQ67fH+/eMR5O2aLZwjyZUQJ7IZN4 qUOVip78YEDXYcCde4mWX4ySo24FEHHpiNFcb4FBTuJRG4yeyqYTyfH1mJ1LuqqwjgRu N3xw== X-Gm-Message-State: AC+VfDyzbg17jt2/11Nr1hIOw8SejDNqW43lFjY0lskMDNNwo+rsVBxL /YI9UnjfR/QJbMzRHui/aYg= X-Google-Smtp-Source: ACHHUZ6JDA42tEVXmLhshxtfb/9NVBPcW7rIpixWMOPq33WSJX3rYKh2qwYXdHkZVaM5I6lap7+ZYQ== X-Received: by 2002:a05:622a:195:b0:3f8:2a37:20f with SMTP id s21-20020a05622a019500b003f82a37020fmr15741005qtw.34.1687106876943; Sun, 18 Jun 2023 09:47:56 -0700 (PDT) Received: from localhost (172.174.245.35.bc.googleusercontent.com. [35.245.174.172]) by smtp.gmail.com with ESMTPSA id bc14-20020a05622a1cce00b003fe8a96fb3bsm571571qtb.54.2023.06.18.09.47.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 18 Jun 2023 09:47:56 -0700 (PDT) Date: Sun, 18 Jun 2023 12:47:56 -0400 From: Willem de Bruijn To: David Howells , netdev@vger.kernel.org Cc: David Howells , Alexander Duyck , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Keith Busch , Jens Axboe , Christoph Hellwig , Sagi Grimberg , Chaitanya Kulkarni , linux-nvme@lists.infradead.org Message-ID: <648f353c55ce8_33cfbc29413@willemb.c.googlers.com.notmuch> In-Reply-To: <20230617121146.716077-11-dhowells@redhat.com> References: <20230617121146.716077-1-dhowells@redhat.com> <20230617121146.716077-11-dhowells@redhat.com> Subject: RE: [PATCH net-next v2 10/17] nvme: Use sendmsg(MSG_SPLICE_PAGES) rather then sendpage Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: EA34A40015 X-Rspam-User: X-Stat-Signature: c8p1byqii5zcdr1yq9dgd1ngsz4qjmyb X-Rspamd-Server: rspam03 X-HE-Tag: 1687106877-554939 X-HE-Meta: U2FsdGVkX19d8pTxaIZn0aXm81ghQapVEpfLlfRJVNrCL55G1MRO/noJp77/cW1twgyyvGThDAeqVp0vm0MXXBSIc130w8Qd+Ja8cJgyJwVMJj4mB28YEzHRdVv7vuOaExLZ7M9N6Fpc6kcGhNunJySQpg9jWlBU5kJGVfgkiTbPey1uKk1fSNOrHXmLHr+Om94L7WLGwJUY4U6EZKTOXHQPuUCRps0Jt+cz+6upZERkuCpx+jLtWU3JcYSMpY/i2naeKgqoKLA3Uj4gSXnk7Ef6OWKHQZ0cyBw0A1yMWWhPmWuAkYmD+2iGyUoYCWyraqQ5zKFrx6YC6xZHfOMnCRKvSvlO7n6vlj0ZA1nOnYR2kMJWHG/CigNO/8Sij9GaaDIy9Q/5Civ89wo42KPsxxnZcoY6jjDrNXPCwKh1QYwiVTeZXFMe5hC4s1VOqay23biQJieTqnNedZq49InACVb0Ld9ivok9EsU0ZGXocrVQyZ439mpVOr7DFh/dRSsGW2eR8+Hn3PthWQ3gJ1tlNATnUOU6cpS64DL1J42RJOVCT2LjYSSnpKSaBo9ha+j4ryPQXSAeXlWqjr6rK0LKhalKWODY+o1aSq85bYnpQXbvqEbvvBXMk3Gn0EkWEBOdswDpkNQfDogj5Z0GKR85vLAu7OthZzbrowrx39hESK8nc9goaVWLgo932XMlpb/OMHGMaAaiaX4T+VVAijZGo+AD5I65p59K1Rr80EQlKlG1xdDC14ikf4UIEe26WZRA+b+Dc0DvRjGjhu5BgOKrejDUtXB9B6afyBayz7dlklfom1lbnCnRqWB6m3t82j0GrKloR2Dm8MXn0hHNTyBaiN1zzu1Qmi+VOWVQmLzyYZS11oxK8ZJzqbYywbZDd4sgQU5UNZ1ZfDhvwXbRALXwj+Jy9IqyEFVaFr+ehz+CXml95V99VD27OZ2bK4fQiC/l3ZtzJRy2KcK4LzLBNpc m8Ou8zOe nGnDDFZPAdvRTqqKjtkRjqnpzscYeTpkZRQKrD8qqqjXXUNGjtOLZIWnZYMjM33Uwf63SMtze26pD/031fUl5gXQdkbnufV3zc4yIc5a/zl21kxgqFCsf65s+QCkchyTietlTX4+ciNp9M9gPDsSGQCrl6WdWzn7KiJhWZCdNIS/AjkHagNaAQyKRI3R//8AaXFnvO/Eqs4FJSHRAQj3kkq5cPQbcr8XOnrk6oN0/hHlq4+o9ko0ldsLJT1aPNSc3qNW53YBSTtA9M384+YkDJx/u/JRznw7nF2YdXLeGoXiQhLTbKceBdHfelrrYmOQaRZYInmfYINB8Q7Z8APirbQIBHIh18HseHfju8u+Tr0XnKDn3Otiy7/ZWN8t1ZL8OMTMZpl8tCoFQ9SWJ6f0s5QQW+xOJJxu4lBqmFacqWNQWTQzf/K9nzXtiJhUdsTc2T/IhtVAAM49gj2PspWil+OmXbrzQJhy4ctI2cOuIy2zFr4UfgEZJzIu+Sqb/evvpnF+HzDprWrbrbx+KKp9Wdd0jUS7W/Ao5rw4Tb+0Ihx1oxmaMjt7/QwwS4mHDcMQ2GoHaDpRes4ttvg6veZPYbsRkbBT3qMQhXdYK714YzxT6T12aHDUtl4Q5NZ9YJOd4Lz9BcOrrDXWKJWWrlaVWs+1GFf2Z6ehQQML74+qdyOoZ7FbpAxgcb053IkTdsdu+w24T1a1oMHkjM2b5irIRvWAOS0uiLLZGifa/odfEdqWxmhNCA6fkSEOMDZSBHrJyCXfICdWZFTwIOjP03vzZydoZ6xg+vuTXm90Ag4DFzqhzvWh4BRHSL692qcnTn6j6PFFBDsuoJdBUCra03xFORrp1Ri3QvRZXqlXKw7DXB1bEQEOLb9VAdxJyZQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: David Howells wrote: > When transmitting data, call down into TCP using a single sendmsg with > MSG_SPLICE_PAGES to indicate that content should be spliced rather than > performing several sendmsg and sendpage calls to transmit header, data > pages and trailer. > > Signed-off-by: David Howells > cc: Keith Busch > cc: Jens Axboe > cc: Christoph Hellwig > cc: Sagi Grimberg > cc: Chaitanya Kulkarni > cc: "David S. Miller" > cc: Eric Dumazet > cc: Jakub Kicinski > cc: Paolo Abeni > cc: Jens Axboe > cc: Matthew Wilcox > cc: linux-nvme@lists.infradead.org > cc: netdev@vger.kernel.org > --- > > Notes: > ver #2) > - Wrap lines at 80. > > drivers/nvme/host/tcp.c | 46 ++++++++++++++++++++------------------- > drivers/nvme/target/tcp.c | 46 ++++++++++++++++++++++++--------------- > 2 files changed, 53 insertions(+), 39 deletions(-) > > diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c > index bf0230442d57..6f31cdbb696a 100644 > --- a/drivers/nvme/host/tcp.c > +++ b/drivers/nvme/host/tcp.c > @@ -997,25 +997,25 @@ static int nvme_tcp_try_send_data(struct nvme_tcp_request *req) > u32 h2cdata_left = req->h2cdata_left; > > while (true) { > + struct bio_vec bvec; > + struct msghdr msg = { > + .msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES, > + }; > struct page *page = nvme_tcp_req_cur_page(req); > size_t offset = nvme_tcp_req_cur_offset(req); > size_t len = nvme_tcp_req_cur_length(req); > bool last = nvme_tcp_pdu_last_send(req, len); > int req_data_sent = req->data_sent; > - int ret, flags = MSG_DONTWAIT; > + int ret; > > if (last && !queue->data_digest && !nvme_tcp_queue_more(queue)) > - flags |= MSG_EOR; > + msg.msg_flags |= MSG_EOR; > else > - flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; > + msg.msg_flags |= MSG_MORE; > > - if (sendpage_ok(page)) { > - ret = kernel_sendpage(queue->sock, page, offset, len, > - flags); > - } else { > - ret = sock_no_sendpage(queue->sock, page, offset, len, > - flags); > - } > + bvec_set_page(&bvec, page, len, offset); > + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, len); > + ret = sock_sendmsg(queue->sock, &msg); > if (ret <= 0) > return ret; > > @@ -1054,22 +1054,24 @@ static int nvme_tcp_try_send_cmd_pdu(struct nvme_tcp_request *req) > { > struct nvme_tcp_queue *queue = req->queue; > struct nvme_tcp_cmd_pdu *pdu = nvme_tcp_req_cmd_pdu(req); > + struct bio_vec bvec; > + struct msghdr msg = { .msg_flags = MSG_DONTWAIT | MSG_SPLICE_PAGES, }; > bool inline_data = nvme_tcp_has_inline_data(req); > u8 hdgst = nvme_tcp_hdgst_len(queue); > int len = sizeof(*pdu) + hdgst - req->offset; > - int flags = MSG_DONTWAIT; > int ret; > > if (inline_data || nvme_tcp_queue_more(queue)) > - flags |= MSG_MORE | MSG_SENDPAGE_NOTLAST; > + msg.msg_flags |= MSG_MORE; > else > - flags |= MSG_EOR; > + msg.msg_flags |= MSG_EOR; > > if (queue->hdr_digest && !req->offset) > nvme_tcp_hdgst(queue->snd_hash, pdu, sizeof(*pdu)); > > - ret = kernel_sendpage(queue->sock, virt_to_page(pdu), > - offset_in_page(pdu) + req->offset, len, flags); > + bvec_set_virt(&bvec, (void *)pdu + req->offset, len); > + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, len); > + ret = sock_sendmsg(queue->sock, &msg); > if (unlikely(ret <= 0)) > return ret; > struct bio_vec bvec; struct msghdr msg = { .msg_flags = MSG_SPLICE_PAGES | ... }; .. bvec_set_virt iov_iter_bvec sock_sendmsg is a frequent pattern. Does it make sense to define a wrapper? Same for bvec_set_page.