From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E66DC0015E for ; Fri, 11 Aug 2023 22:50:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D51146B0074; Fri, 11 Aug 2023 18:50:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D01396B0078; Fri, 11 Aug 2023 18:50:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BC8526B007B; Fri, 11 Aug 2023 18:50:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id AE4F46B0074 for ; Fri, 11 Aug 2023 18:50:18 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 5355A1207F8 for ; Fri, 11 Aug 2023 22:50:18 +0000 (UTC) X-FDA: 81113318916.17.C735A26 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf14.hostedemail.com (Postfix) with ESMTP id F1F65100003 for ; Fri, 11 Aug 2023 22:50:14 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=EkXomq4E; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf14.hostedemail.com: domain of jlayton@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=jlayton@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691794215; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VyHdSV0R704eg+oymjb62w9SaruwTmlTZbEUdHMGCbQ=; b=l+X4n+cDD/0obfFaHktBzi3/CibNzOIv6c3rerqF6b4zH3E70M287goc2AafmdbJ75s6Ry 6jC18W1cL8cBn5gUF8lM91cqqhImooOitEVH2zXGQ0Ws9IdK6973xtYQk0kVR2BMsY/bVa bbk+13GY3me6KJO5xTdf2WhWfojh/8Q= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=EkXomq4E; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf14.hostedemail.com: domain of jlayton@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=jlayton@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691794215; a=rsa-sha256; cv=none; b=qev9fFRLXWyhFTtEC2dS3P9ZW4Tz70klsx3Bhu1q8DrOVNE03Ri1b08Zo9QDKHYrqKYfrd F6fYHUoqTdn3BFtaqQOz/VpqJQGmjK06z5uXSltBLdH+GECjg3ZCXubUgR5HgfA0138Lgn ivbQcqfXtj+DnUHudpXS09FVOSNSG74= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id C567666EB3; Fri, 11 Aug 2023 22:50:13 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BB58AC433C8; Fri, 11 Aug 2023 22:50:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1691794213; bh=k02u5dlw3yPXK+0T6pf00+UIRqTxEUsHTDDQW8+4BpM=; h=Subject:From:To:Cc:Date:In-Reply-To:References:From; b=EkXomq4EYP3brX2c1RiuwvfQyeNHIM6fA2Ddf6dQZ0zg4uvqyzm03zrFEA3bQrJU8 XwJ5qMahpl8ei8/YMDrSGRgWu7Rjoe43IvHGb75QG8Oa78TPGTLDqYqoqpoepd5TfB 0fIhUchFKQPw4fbbbC6l0jOgGdIOSuejq7HklleN7CsszJCyNx+hVSnqMeJ5RK601j IZ3EWhWiyUbB19KDld1JfOQ+wWbfQwidzuNQnszJGIRW2mKLQS3DRUXuTqnS8G8KR5 iNjzl9c0zTf1oW9HK/T3sIGVrbJnEpVOezkrEdLKUsLY7RkVZYDIB7/iKyHrVIC6g4 Jm/8Ow02NWtEw== Message-ID: <104f68073d00911668ed6ea38239ef5f1d15567d.camel@kernel.org> Subject: Re: [PATCH net-next 3/6] sunrpc: Use sendmsg(MSG_SPLICE_PAGES) rather then sendpage From: Jeff Layton To: David Howells , netdev@vger.kernel.org Cc: Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Chuck Lever , Trond Myklebust , Anna Schumaker , linux-nfs@vger.kernel.org Date: Fri, 11 Aug 2023 18:50:10 -0400 In-Reply-To: <20230609100221.2620633-4-dhowells@redhat.com> References: <20230609100221.2620633-1-dhowells@redhat.com> <20230609100221.2620633-4-dhowells@redhat.com> Content-Type: text/plain; charset="ISO-8859-15" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.48.4 (3.48.4-1.fc38) MIME-Version: 1.0 X-Rspamd-Queue-Id: F1F65100003 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: z43b67qqdyxwta37qhc7cfsfdkrcrjhr X-HE-Tag: 1691794214-601888 X-HE-Meta: U2FsdGVkX19lKaVrjOTZV8s2BOqqNdU6CrSecTU74//ZlowOE9BUvtFKxFGDyEH3OnJ8ekmDL0wAZ5KX2K5ixl+YITK2hjBsv9qNGlmXOPf/eXRUpKoHJfSigeACnGBCr6kU9yoN4LV90WD3z8Jgsc99kNiJi58nwYGR0oCU6TI6C3E+BOqXFpjzDTY2B8MzIo2YaYCHSFqxw0LRNQEplr004DAaT9mv8XMwA1niBysLczF61ZHHWRv121XbAhpnTkiRCYoZHSjgVIjd7ifUUGrB8s15fsgxlGwjsgs6zSaaRa9A4RerEFAbJFujW+C+IKswZd2rDFx4Sh0vrDfH1xFZGMk9k9WCSuGp11m935DHbw8j1T0GpXd64xCt/d7NupfZtlU7vab8xzb/tlvTlxGMg6TJoI4b3WYCLlVf1/jXfKCMFSoP76LTA1/9fTDN4hta5ZoyUWkndpVN9Wbtu2fYLSs4yGwWWfkfuV1TQZxrEatnsOSJUj/IG4j07rAYlnc9T3DcLmJUvzt8rdVHRUp7DXbY8bfDWvJgH8FQTuwGRTKzYwCTID3nnVXAYOXTTCY1+B9H3iMkxexMR5u1uY6mHzAb+xK4kPastF/gkIJWGQc2DOdTSg73PrAWlkOvmCq/Lcjkf+/yVPPoMa+9Fby9HZ6sqmFB7XpHypPCcHKQF9iDSKQ4hgYL3v2w6Hw6f5dU5czsYwUI0hkb8TakYhBvFhKaT7IJ/2Zqu5lI0VHF94oBrr1xF2gOK9MQ22g0+2uGDgTV6r0eq08slUlWhFXyXTK+7nTaHXaQpJCojol4ECEMkJlvS/Fro4RPvQjhMSISo7AcUFOOzFwOXb/2CXoVY1/r7aM4o3sFRPbn/u0KzD1PkQ7fIDCRgXJfvrYZAjCXDJO5LnPoWgWn0EnrG3OBiQGPWqxFTtVKRULyu4IOiwV5pKW9SrpgVyqf1TosPtFJ95RFNauQEuXnjfa eFk5MGtf Wt3yP1Qrs+LYFbcv/Tnv6v+okUEIaN4u3QaRHe5G/O3Uc0dgoFcQqceCLWAPx2JM9JykYowGynS8fhsXqyESeS4KVXNa9W8CUEW6aiYkbOz+O/1SQtxpafxG3ZK72elr2aRZh0kNj+hjwPeMusMsFXNebjkVqBkmfepc6gNZMB5diUtdNtwp3f38aAFkjk1KUU6KcxdNUHkymylBd+6pWHDbWjrYIwtQ+KwuhENhumMaKoeLb19khLL2Wxdw8mDbw+d8Mjc6hhz0OGbjd7aEvGNYejh5zDmm48E3+HHg5YpU2+buMgcngxmOpoZzVCnQasu1Usx4u+BMd5iFvzpzzkmqeAf9l/4hOi38jNII1VGiqyEmYciaJ9/CrRxCMp5dIl0A8jpXx34zYdDbNnMH4VKcnlLV5t6mGNd/e8Usg+3Y2jIcTBKhYJb5v6dZQOa1XcNSFEQBF1e+DHWM3bfUf1FWtlXsEtEzputjLwIBpSpVpGPa8U6X/rqPcXPfoRPp9RLGYR9NfxFpBLX+IOWa1Uo/S1L7lp6KVRV5smrrbGLFfLsJP/RhCkL4MnYRxg8X5HdWgquCj78xY5Ctq8Fn0ptizD2IyvQMtCbTe8+Vpz2Ad+0CEtptOppGzBc0z/prXyXlp4lBtoBA2ieQSTzL4P9SWuckqYKrrsujlokY6MbxBK0SEKzPMJWhfHZpirC02dNOepJQj9GG9ZSj0Rm+r5VVUzA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, 2023-06-09 at 11:02 +0100, David Howells wrote: > When transmitting data, call down into TCP using sendmsg with > MSG_SPLICE_PAGES to indicate that content should be spliced rather than > performing sendpage calls to transmit header, data pages and trailer. >=20 > Signed-off-by: David Howells > Acked-by: Chuck Lever > cc: Trond Myklebust > cc: Anna Schumaker > cc: Jeff Layton > cc: "David S. Miller" > cc: Eric Dumazet > cc: Jakub Kicinski > cc: Paolo Abeni > cc: Jens Axboe > cc: Matthew Wilcox > cc: linux-nfs@vger.kernel.org > cc: netdev@vger.kernel.org > --- > include/linux/sunrpc/svc.h | 11 +++++------ > net/sunrpc/svcsock.c | 38 ++++++++++++-------------------------- > 2 files changed, 17 insertions(+), 32 deletions(-) >=20 I'm seeing a regression in pynfs runs with v6.5-rc5. 3 tests are failing in a similar fashion. WRT1b is one of them [vagrant@jlayton-kdo-nfsd nfs4.0]$ ./testserver.py --rundeps --maketree --= uid=3D0 --gid=3D0 localhost:/export/pynfs/4.0/ WRT1b = =20 ************************************************** = = =20 WRT1b st_write.testSimpleWrite2 : FAILURE= = =20 READ returned = = =20 b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00',= = =20 expected b'\x00\x00\x00\x00\x00write data' = = =20 INIT st_setclientid.testValid : PASS = = =20 MKFILE st_open.testOpen : PASS = = =20 ************************************************** = = =20 Command line asked for 3 of 679 tests = = =20 Of those: 0 Skipped, 1 Failed, 0 Warned, 2 Passed = =20 This test just writes "write data" starting at offset 30 and then reads the data back. It looks like we're seeing zeroes in the read reply where the data should be. A bisect landed on this patch, which I'm assuming is the same as this commit in mainline: 5df5dd03a8f7 sunrpc: Use sendmsg(MSG_SPLICE_PAGES) rather then sendpage ...any thoughts as to what might be wrong? > diff --git a/include/linux/sunrpc/svc.h b/include/linux/sunrpc/svc.h > index 762d7231e574..f66ec8fdb331 100644 > --- a/include/linux/sunrpc/svc.h > +++ b/include/linux/sunrpc/svc.h > @@ -161,16 +161,15 @@ static inline bool svc_put_not_last(struct svc_serv= *serv) > extern u32 svc_max_payload(const struct svc_rqst *rqstp); > =20 > /* > - * RPC Requsts and replies are stored in one or more pages. > + * RPC Requests and replies are stored in one or more pages. > * We maintain an array of pages for each server thread. > * Requests are copied into these pages as they arrive. Remaining > * pages are available to write the reply into. > * > - * Pages are sent using ->sendpage so each server thread needs to > - * allocate more to replace those used in sending. To help keep track > - * of these pages we have a receive list where all pages initialy live, > - * and a send list where pages are moved to when there are to be part > - * of a reply. > + * Pages are sent using ->sendmsg with MSG_SPLICE_PAGES so each server t= hread > + * needs to allocate more to replace those used in sending. To help kee= p track > + * of these pages we have a receive list where all pages initialy live, = and a > + * send list where pages are moved to when there are to be part of a rep= ly. > * > * We use xdr_buf for holding responses as it fits well with NFS > * read responses (that have a header, and some data pages, and possibly > diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c > index f77cebe2c071..9d9f522e3ae1 100644 > --- a/net/sunrpc/svcsock.c > +++ b/net/sunrpc/svcsock.c > @@ -1203,13 +1203,14 @@ static int svc_tcp_recvfrom(struct svc_rqst *rqst= p) > static int svc_tcp_send_kvec(struct socket *sock, const struct kvec *vec= , > int flags) > { > - return kernel_sendpage(sock, virt_to_page(vec->iov_base), > - offset_in_page(vec->iov_base), > - vec->iov_len, flags); > + struct msghdr msg =3D { .msg_flags =3D MSG_SPLICE_PAGES | flags, }; > + > + iov_iter_kvec(&msg.msg_iter, ITER_SOURCE, vec, 1, vec->iov_len); > + return sock_sendmsg(sock, &msg); > } > =20 > /* > - * kernel_sendpage() is used exclusively to reduce the number of > + * MSG_SPLICE_PAGES is used exclusively to reduce the number of > * copy operations in this path. Therefore the caller must ensure > * that the pages backing @xdr are unchanging. > * > @@ -1249,28 +1250,13 @@ static int svc_tcp_sendmsg(struct socket *sock, s= truct xdr_buf *xdr, > if (ret !=3D head->iov_len) > goto out; > =20 > - if (xdr->page_len) { > - unsigned int offset, len, remaining; > - struct bio_vec *bvec; > - > - bvec =3D xdr->bvec + (xdr->page_base >> PAGE_SHIFT); > - offset =3D offset_in_page(xdr->page_base); > - remaining =3D xdr->page_len; > - while (remaining > 0) { > - len =3D min(remaining, bvec->bv_len - offset); > - ret =3D kernel_sendpage(sock, bvec->bv_page, > - bvec->bv_offset + offset, > - len, 0); > - if (ret < 0) > - return ret; > - *sentp +=3D ret; > - if (ret !=3D len) > - goto out; > - remaining -=3D len; > - offset =3D 0; > - bvec++; > - } > - } > + msg.msg_flags =3D MSG_SPLICE_PAGES; > + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, xdr->bvec, > + xdr_buf_pagecount(xdr), xdr->page_len); > + ret =3D sock_sendmsg(sock, &msg); > + if (ret < 0) > + return ret; > + *sentp +=3D ret; > =20 > if (tail->iov_len) { > ret =3D svc_tcp_send_kvec(sock, tail, 0); >=20 --=20 Jeff Layton