From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BB9CC6FD18 for ; Thu, 30 Mar 2023 01:45:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EAA576B0074; Wed, 29 Mar 2023 21:45:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E5A3B6B0075; Wed, 29 Mar 2023 21:45:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CFB426B0078; Wed, 29 Mar 2023 21:45:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id BEC796B0074 for ; Wed, 29 Mar 2023 21:45:49 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 7F841AC687 for ; Thu, 30 Mar 2023 01:45:49 +0000 (UTC) X-FDA: 80623873218.02.CFFB338 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf15.hostedemail.com (Postfix) with ESMTP id 2E1A9A0003 for ; Thu, 30 Mar 2023 01:45:46 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="h/Uqi4yG"; spf=pass (imf15.hostedemail.com: domain of xiubli@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=xiubli@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1680140747; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kv0JdKs3b/J2KtWtqtGbcl417IkaIZ3v3L6FuRPDYCY=; b=zvwJ31kD9/sLYvl9SMZrFKF4NfkTowUSk5bcAm8apL97feNeindNDEAnSAk7tjIgKLltbD ZbVDKVDDnV3cK/WAOLWgO7+JYNt/jbiyLpAAPjy6sO0aitteeS76MNnsGLye3H2C0tayPA JwtOKFBJxOUiePt7IvGDmo4lnpT4yPM= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="h/Uqi4yG"; spf=pass (imf15.hostedemail.com: domain of xiubli@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=xiubli@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1680140747; a=rsa-sha256; cv=none; b=Zlr5evUDJD0siiXt4aAIOcQtrB42fdupj9CPCnrDNX84RZr+nBwVzjI0brmuF2wnBBD8xV R3hyPFzDn0IU24LYtn0i1zSxbgrBTdhaMzcoXgE5P/mtekuUjYrkNpxwLo5irFHb+5ZYIv KtzPiDULedOMMlp3JmJGwiLgODMBF+Y= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1680140745; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kv0JdKs3b/J2KtWtqtGbcl417IkaIZ3v3L6FuRPDYCY=; b=h/Uqi4yGR08up02Zp61RPKEXUyJbWh7djpHoEcS1vpEhk5cFUREN3od4Xp+SOdecmDEp++ uRgLPTR3Yt7sFw0/BDaNmO/dn5DO0A+6D1F3gB53VMxX1BYQ7LoJ6B7eRRcmnLeH68Ig62 AKn6DIFn+JwzjwWji6sVZEp/xxaVCdA= Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-133-7AXthu0ENr-dTWyb0pOcyA-1; Wed, 29 Mar 2023 21:45:44 -0400 X-MC-Unique: 7AXthu0ENr-dTWyb0pOcyA-1 Received: by mail-pf1-f198.google.com with SMTP id i7-20020a626d07000000b005d29737db06so8134547pfc.15 for ; Wed, 29 Mar 2023 18:45:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1680140743; h=content-transfer-encoding:in-reply-to:from:references:cc:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=kv0JdKs3b/J2KtWtqtGbcl417IkaIZ3v3L6FuRPDYCY=; b=oJt2gSbYs51gRaYnU77xdOVY8s4QcVEnPOhdt/obNLu5FAsP9AEy1K6DRyuZml2vc8 TSPwGIW3bE/ODyqcZntxReMNxcBYbAIiZykiNSz3F70C2vEG9v6/bHbGeSHsnbF3AUHd 7amgpk1QGCnM+kkEJpc6LnovXFGd+SOKvcb8t12mhJognrcBXLo6Z6sf70Bkt0F+oQXv n+XuSqgcORCmSoJ5fhGVaxq5p7/FUtu8Ac2bU8p0eLU51LdTu8GiXRANsdzMbExArLKl Jt83Gc2AC8W0sn5/3bDD3TMO0xLajiUGtrKn7d4VP5+Vt0cOARM7Axy42af2SaM1P5rE fSWQ== X-Gm-Message-State: AAQBX9d0HXk4YAab8PjAdUBOVBRqw8/ywXBAaMciLUvB7ehxZfQGyBWw b+Au2S8bmKi5+ocZC954w8slu4jQxvNV7VSv7XX+UCGZ6R/x59cP0LqjqJYO6kUmzsqZy5Kg+21 HBTS7z6T1IpQ= X-Received: by 2002:a17:903:22d2:b0:19a:7217:32a9 with SMTP id y18-20020a17090322d200b0019a721732a9mr4636127plg.26.1680140743046; Wed, 29 Mar 2023 18:45:43 -0700 (PDT) X-Google-Smtp-Source: AKy350ZC7R/sqsXLmXMgRh5yYmDYt07re8Mz0pqFOB7hKr/eVqrBonJ1fWeZySGpK6QRntg/4jjpKg== X-Received: by 2002:a17:903:22d2:b0:19a:7217:32a9 with SMTP id y18-20020a17090322d200b0019a721732a9mr4636092plg.26.1680140742679; Wed, 29 Mar 2023 18:45:42 -0700 (PDT) Received: from [10.72.12.51] ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id jj2-20020a170903048200b0019fea4bb887sm23709042plb.157.2023.03.29.18.45.34 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 29 Mar 2023 18:45:42 -0700 (PDT) Message-ID: <7f7947d6-2a03-688b-dc5e-3887553f0106@redhat.com> Date: Thu, 30 Mar 2023 09:45:32 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.9.0 Subject: Re: [RFC PATCH v2 37/48] ceph: Use sendmsg(MSG_SPLICE_PAGES) rather than sendpage() To: David Howells , Matthew Wilcox , "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni Cc: Al Viro , Christoph Hellwig , Jens Axboe , Jeff Layton , Christian Brauner , Chuck Lever III , Linus Torvalds , netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Ilya Dryomov , ceph-devel@vger.kernel.org References: <20230329141354.516864-1-dhowells@redhat.com> <20230329141354.516864-38-dhowells@redhat.com> From: Xiubo Li In-Reply-To: <20230329141354.516864-38-dhowells@redhat.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: f1cwdx3pf416awk37xku5g4iqb7ow8rw X-Rspamd-Queue-Id: 2E1A9A0003 X-HE-Tag: 1680140746-243026 X-HE-Meta: U2FsdGVkX19fQCph6pKbJBxBD8L/uoP0bDdrOtuGCAKqEP8RTbGjV0Lo2RoVhw+5mON//rHK63eQ3zO7bIk/fLQnM+Ec+Idlo23m2K0iajs7KpgTP35qiaj4HzCyZ4y8VEl0iP3p1iAlx1GVlvCQ6dHnGZCw1KHZPFF5j1L+aP6ZODFyIUDRI7I0f5KMsbHEmWY74DgK7oOGtJRAQZBwIwrq+Xf4Tn4dCbFy/OlM9XZ6jgawBQL41xwAoxS1WxcP7XPgwyVNjvRFpkyAOxYKnJHtVx044H6aByTuC0BXEN1unrrvca/Zhi1zhZMAjbFzKvd955MW3ivFkEGeq/12e4oTtRKduvCQ9gbbnzoBQlXoXyzdLUAeSVgU6ASugqu9O0zY308eLgOPxMWvc/wMoo/NnUOzB84Z/Eo+DRd0bDZNqJ3x+FEbmpflK0yaAVj07CEsLWWpQgbohaz3ywae7pXXL9LwJC6P6UwEql7/TeNy0GtiDzWGbZeSXWW8eaNmiymANXjY2r4eYGuV4tF/70UGuUFBU/mV3P2zKkoSi9vZ/2FrqvSb+399HHCHJ2v6WtGxGeNFIE5sZUd1yn3BqhbcTTXHKoHp3tUlNclNm5kS6as3lrMTN+lhylA7sdNsDA7o0fkRKbYnXVzOv1VaWJtFQKolwnbPcY77JbTTv4Wa/OaF8wREpnN3wr2fk0GZcNN4FbmZSyqMHz+aWMEFvgAXeGjhqnQD7SnKv+nkw23JaRn7OzGwafqReD2z+EBUBvjMQFP+bT5T2WKE/2KjnoN3zxl/2FIkwq9hOBe3gk7NVxdUIgC2jffI9ny1ahH6TKpiw/s9fvZqOlFVhNCVnWKt9d843eNqsJCJoyVv60+JEu47ypPuuYeigfJ1ukrfg8mnHcJx8PQGWUbfovDXuspDILoy6HyNnaOO1A/R7WPlUhrvR4A2JXoBTfYdLzWAJ9B3sodTmr2BWjH0nOM 5kGFOJl3 hmrLCL9E2K3dYgiJ7rrHp+OnZJIDgzGDvk+sxQJHMYRJMGdDU3zUxmUTN+hufmBBemORN0pbVZXUEWIdXNW9BfoCCSxjKJ31FwPyBYrm5TTl8ergVJb+GCGcJECG0v9CpzpTT8mLTg4bnA15IlWHqVyVWD8by4mS+DRJ6cCN5VbyI54Las0Iiatcvzmi2eFetpkEsPcRLPUJ3oaXxdkZv5hfQMhfCZVGI07JLa9dmnC5AiXVaoAcMLd39yYVD/JH6tuWyPlP4/uIdALHilgAxQE7xO8cw8XojgsDxDjb7nNpGeDBTrP8QH1jYkFO8DiNrt32pCpFCyq2XXidnjTrTQHqgcVFQWkqZKU2LjXU3wa2lj1y033qpL+XOy3MQedor0I2Un3ZGMlegEyK3Xclz5VtzPR0kpzJesOg5rxLvPY00rAuLk09DNQCvoDPuWgsfWhFPmrJ6Aq5YuV1o9iGpBLiGERcyGQPtyy9/a8J6RRno4Z6GnfSZbqMXh/v6YmJ0NVYbS8B+16zLYGQH0mBHigD6257ZmCC5LJs96I2sqnccyae/XNR4wOZWZ0zCIBpl8A5tmJC2J2t5/78LqzlmWg0lbuaf/hfHAerGy/dp2RuQkUZ5jnq6I+Y406KKpNNyG1lECwQYqNCMHnkVBc0qH3ywNHx1kEHpnMFoZBqOIbtYRsKE1bTnQt2d4Wfmtp6AUKtFVg8Qz1YhY3n9DAU1l9Wm1BqdOT/R36KV X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: David, BTW, will this two patch depend on the others in this patch series ? I am planing to run a test with these two later. Thanks - Xiubo On 29/03/2023 22:13, David Howells wrote: > Use sendmsg() and MSG_SPLICE_PAGES rather than sendpage in ceph when > transmitting data. For the moment, this can only transmit one page at a > time because of the architecture of net/ceph/, but if > write_partial_message_data() can be given a bvec[] at a time by the > iteration code, this would allow pages to be sent in a batch. > > Signed-off-by: David Howells > cc: Ilya Dryomov > cc: Xiubo Li > cc: Jeff Layton > cc: "David S. Miller" > cc: Eric Dumazet > cc: Jakub Kicinski > cc: Paolo Abeni > cc: Jens Axboe > cc: Matthew Wilcox > cc: ceph-devel@vger.kernel.org > cc: netdev@vger.kernel.org > --- > net/ceph/messenger_v2.c | 89 +++++++++-------------------------------- > 1 file changed, 18 insertions(+), 71 deletions(-) > > diff --git a/net/ceph/messenger_v2.c b/net/ceph/messenger_v2.c > index 301a991dc6a6..1637a0c21126 100644 > --- a/net/ceph/messenger_v2.c > +++ b/net/ceph/messenger_v2.c > @@ -117,91 +117,38 @@ static int ceph_tcp_recv(struct ceph_connection *con) > return ret; > } > > -static int do_sendmsg(struct socket *sock, struct iov_iter *it) > -{ > - struct msghdr msg = { .msg_flags = CEPH_MSG_FLAGS }; > - int ret; > - > - msg.msg_iter = *it; > - while (iov_iter_count(it)) { > - ret = sock_sendmsg(sock, &msg); > - if (ret <= 0) { > - if (ret == -EAGAIN) > - ret = 0; > - return ret; > - } > - > - iov_iter_advance(it, ret); > - } > - > - WARN_ON(msg_data_left(&msg)); > - return 1; > -} > - > -static int do_try_sendpage(struct socket *sock, struct iov_iter *it) > -{ > - struct msghdr msg = { .msg_flags = CEPH_MSG_FLAGS }; > - struct bio_vec bv; > - int ret; > - > - if (WARN_ON(!iov_iter_is_bvec(it))) > - return -EINVAL; > - > - while (iov_iter_count(it)) { > - /* iov_iter_iovec() for ITER_BVEC */ > - bvec_set_page(&bv, it->bvec->bv_page, > - min(iov_iter_count(it), > - it->bvec->bv_len - it->iov_offset), > - it->bvec->bv_offset + it->iov_offset); > - > - /* > - * sendpage cannot properly handle pages with > - * page_count == 0, we need to fall back to sendmsg if > - * that's the case. > - * > - * Same goes for slab pages: skb_can_coalesce() allows > - * coalescing neighboring slab objects into a single frag > - * which triggers one of hardened usercopy checks. > - */ > - if (sendpage_ok(bv.bv_page)) { > - ret = sock->ops->sendpage(sock, bv.bv_page, > - bv.bv_offset, bv.bv_len, > - CEPH_MSG_FLAGS); > - } else { > - iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bv, 1, bv.bv_len); > - ret = sock_sendmsg(sock, &msg); > - } > - if (ret <= 0) { > - if (ret == -EAGAIN) > - ret = 0; > - return ret; > - } > - > - iov_iter_advance(it, ret); > - } > - > - return 1; > -} > - > /* > * Write as much as possible. The socket is expected to be corked, > * so we don't bother with MSG_MORE/MSG_SENDPAGE_NOTLAST here. > * > * Return: > - * 1 - done, nothing (else) to write > + * >0 - done, nothing (else) to write > * 0 - socket is full, need to wait > * <0 - error > */ > static int ceph_tcp_send(struct ceph_connection *con) > { > + struct msghdr msg = { > + .msg_iter = con->v2.out_iter, > + .msg_flags = CEPH_MSG_FLAGS, > + }; > int ret; > > + if (WARN_ON(!iov_iter_is_bvec(&con->v2.out_iter))) > + return -EINVAL; > + > + if (con->v2.out_iter_sendpage) > + msg.msg_flags |= MSG_SPLICE_PAGES; > + > dout("%s con %p have %zu try_sendpage %d\n", __func__, con, > iov_iter_count(&con->v2.out_iter), con->v2.out_iter_sendpage); > - if (con->v2.out_iter_sendpage) > - ret = do_try_sendpage(con->sock, &con->v2.out_iter); > - else > - ret = do_sendmsg(con->sock, &con->v2.out_iter); > + > + ret = sock_sendmsg(con->sock, &msg); > + if (ret > 0) > + iov_iter_advance(&con->v2.out_iter, ret); > + else if (ret == -EAGAIN) > + ret = 0; > + > dout("%s con %p ret %d left %zu\n", __func__, con, ret, > iov_iter_count(&con->v2.out_iter)); > return ret; > -- Best Regards, Xiubo Li (李秀波) Email: xiubli@redhat.com/xiubli@ibm.com Slack: @Xiubo Li