From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A67A8C77B73 for ; Wed, 31 May 2023 12:45:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 439918E000A; Wed, 31 May 2023 08:45:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3E93F8E0002; Wed, 31 May 2023 08:45:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 23D538E000A; Wed, 31 May 2023 08:45:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 149C18E0002 for ; Wed, 31 May 2023 08:45:56 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 9BDE01A018D for ; Wed, 31 May 2023 12:45:55 +0000 (UTC) X-FDA: 80850522270.11.D5895C4 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf10.hostedemail.com (Postfix) with ESMTP id A336CC0023 for ; Wed, 31 May 2023 12:45:53 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=eaTXNOEV; spf=pass (imf10.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1685537153; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=W8aEdMrYnuprZg7eZCUzLy+kKHthZDT31VQbMyJsAn8=; b=I1w6O3TyaMFRM0AvXTdPH87WGDOSn2mNkHAjm3WO1PXiiS6FkAhad4KteKOkBsxXy5RGXE OqtWXFQRnS0NeQO6NjNpjI4g2ZRf1tEUtLLMnIzpYEZL5VAF5CD5j3+r4Py9b+yJslplt4 qiGYkAv0alWZbXsNaD9AtFMcKcQDfCk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1685537153; a=rsa-sha256; cv=none; b=oK3vxWULLs7/pVDBe/6Wcivd9Hie7vwt1PE2xcsF/nzhY5Pw025vzlcCvkeEem59lJE73O 0VTR9wjzEpdmJ05H0ed8Dt9zDGJuJExwApg2azU2okEpm5xr1rtM7p6fNNUWPE9hJQ9M3Y 7rqNar6r84u0anKkFhzQqAuc7Z9fJXI= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=eaTXNOEV; spf=pass (imf10.hostedemail.com: domain of dhowells@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=dhowells@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1685537152; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=W8aEdMrYnuprZg7eZCUzLy+kKHthZDT31VQbMyJsAn8=; b=eaTXNOEVZzqJSbHKasDeuNYAsd6mxcp/ot3sEY/pE/7KSM6zXRUhOm/Woz3c0sVZPu1Zi6 jXdyZ6zW0ROaqKK27vZAr+7MLw1SIrTc91q/YtLU6MpwFjG1IDAH4RqHA3xZ/0tvwMZfyv PHXiTIzzMcwkJqRrqOl/HkOF+TV1bi8= Received: from mimecast-mx02.redhat.com (mx3-rdu2.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-370-LbK9yftePiayyMeYHhVc5g-1; Wed, 31 May 2023 08:45:48 -0400 X-MC-Unique: LbK9yftePiayyMeYHhVc5g-1 Received: from smtp.corp.redhat.com (int-mx07.intmail.prod.int.rdu2.redhat.com [10.11.54.7]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id D0DF53825BA9; Wed, 31 May 2023 12:45:47 +0000 (UTC) Received: from warthog.procyon.org.uk (unknown [10.42.28.182]) by smtp.corp.redhat.com (Postfix) with ESMTP id F3555140E963; Wed, 31 May 2023 12:45:44 +0000 (UTC) From: David Howells To: netdev@vger.kernel.org Cc: David Howells , Chuck Lever , Boris Pismenny , John Fastabend , Jakub Kicinski , "David S. Miller" , Eric Dumazet , Paolo Abeni , Willem de Bruijn , David Ahern , Matthew Wilcox , Jens Axboe , linux-mm@kvack.org, linux-kernel@vger.kernel.org, bpf@vger.kernel.org Subject: [PATCH net-next v2 4/6] tls/sw: Convert tls_sw_sendpage() to use MSG_SPLICE_PAGES Date: Wed, 31 May 2023 13:45:26 +0100 Message-ID: <20230531124528.699123-5-dhowells@redhat.com> In-Reply-To: <20230531124528.699123-1-dhowells@redhat.com> References: <20230531124528.699123-1-dhowells@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 3.1 on 10.11.54.7 X-Rspamd-Queue-Id: A336CC0023 X-Rspam-User: X-Stat-Signature: qqz3ydugir8q4fj4pbksomxj3gayubxf X-Rspamd-Server: rspam03 X-HE-Tag: 1685537153-879555 X-HE-Meta: U2FsdGVkX19ssiptl7BaqJARUokmyB5VFHuoyUu26KhbG2/UWNyvl4VPg7YIZWHPlRuH9VPMqICPxtLOM2SrCJkensIOyfTPUwe2afa+s7p5m8K2sYYlBMzREB+O3AVHIAMK21AC5ZDl00a4pgT3kYHLER98SeHcDcif7VV/NGAFyxA39xWw0/WVG0fmProdt9UfCJvzk8ee5mej+JJ2Yis8PBTDK9ikthrlVddUsswrl2xzfLZnl5562DVmACu9TXv+IJgs/uP+g063c58OZjVYZUJXuDcvHvX9u8toQ1hgJuE0FlJDb6tGAJ4fvCCNLPkih9U0b6/h0EOna4LjCIAlFLJVjSRnxcyw/q5BMVdf1J7Ubnbqb6a4VAdeURKdUEkBHtu+cIkahVV/nHP5Vfkr/UVus2sPZxDcSktF5kawT69Z4oD4C83pqcw6ohDL0xwD44CN9TZOYNTiuZtF0A3G8xVsxgQtV8EScRJRgM5E2mMBmmbxWioQ9/P2tfaet9SaLT3MYTUjUk5xxXOG7YdJYfqRDDb8FFHIAyKTchOcWRwIexZcFskzrBqyUYrVIWnmdYwWOIuSmgLwDauceEUDw6GL+T3M1IGKXeL1FJJrNEb8nok6ebfPRUTzWSX46eGoCMHeFwoxIUhfzDgQhPpuTo9c2jASvSfSpHdoiQFSmU/FiuuyFu4LJF7H11GllFqoCsowpcLjjSQAcUF82xV1AgifVF83o6Duk3RsKc8zhPF9G8fiiCV+dsupdbHDvvpmrvM1xXn2Wbfpyn5713DjRgYyv13yU/qWzdq3sA5ua4JnoLnn6UcL8osT5x43ADVz8uEPHWeGGoqpnM1NLddRo+o+ZXvLALMU9XFj7qkLOR3VM1j9uwxjy1Y+Y/uLEsRGTT5bOq0VqMIijXlhEujGm5037xPYmmDGhTXy1VNAFGgDacPmQXCrRRq4by/Dieh4QI3Umg+o7QAICcz 91+3LrZy +ByPeOixWmRwx4yuD+TZLj+ElTfbVxUL98zyqtLSpNbJuV6HA50aljJgLh4Mrps2RHndv3rVbKJC2pcChEtdSeot8qv0PA5PT5vtCr+lbJ8V2j9IcfxQboG+RuR9AGif0ClD3NW5l37kwYdECGJ8Lk1t+4FuVto6fprPYD23IAHM81E4xY1ijmOoJ9EAbZqwkH2V94k2Lx6uCjgMqqixJnrPLR/r4ya/2+p0Uvu/a7VBk5962cQwPaYRkvF2BVgwXqMkDwC1yZ4mQFJnoIpSVnx+coC4gEPbk3lNJ86yQSu59h9cNiO6G0Qu4HHSalJsNuOJKlFICos9HCVOyVrD7MMhQAOhg/NM85yBbZF031NpUMKG4Uuw0wwH2LIA/04d3x82+lUUhwgUS/F6HdE3L49bDHt7SuMI0puy82/dPUCo+PQ43GKcLK8zl9AeoSB3hgafQGp2W6KMVxawLupZUiVARW91z+h4LcweiiWDqvO3mJu0v2QefvJOQMNn1s+i28zAcCcfjfFFSvXLu5CcP9RYEKPF+/3qwXj0jSVWFgs8OUbAii6BFanwPwA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Convert tls_sw_sendpage() and tls_sw_sendpage_locked() to use sendmsg() with MSG_SPLICE_PAGES rather than directly splicing in the pages itself. [!] Note that tls_sw_sendpage_locked() appears to have the wrong locking upstream. I think the caller will only hold the socket lock, but it should hold tls_ctx->tx_lock too. This allows ->sendpage() to be replaced by something that can handle multiple multipage folios in a single transaction. Signed-off-by: David Howells cc: Chuck Lever cc: Boris Pismenny cc: John Fastabend cc: Jakub Kicinski cc: Eric Dumazet cc: "David S. Miller" cc: Paolo Abeni cc: Jens Axboe cc: Matthew Wilcox cc: netdev@vger.kernel.org cc: bpf@vger.kernel.org --- net/tls/tls_sw.c | 165 +++++++++-------------------------------------- 1 file changed, 31 insertions(+), 134 deletions(-) diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c index 60d807b63f75..f63e4405cf34 100644 --- a/net/tls/tls_sw.c +++ b/net/tls/tls_sw.c @@ -961,7 +961,8 @@ static int tls_sw_sendmsg_splice(struct sock *sk, struct msghdr *msg, return 0; } -int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) +static int tls_sw_sendmsg_locked(struct sock *sk, struct msghdr *msg, + size_t size) { long timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT); struct tls_context *tls_ctx = tls_get_ctx(sk); @@ -984,15 +985,6 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) int ret = 0; int pending; - if (msg->msg_flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | - MSG_CMSG_COMPAT)) - return -EOPNOTSUPP; - - ret = mutex_lock_interruptible(&tls_ctx->tx_lock); - if (ret) - return ret; - lock_sock(sk); - if (unlikely(msg->msg_controllen)) { ret = tls_process_cmsg(sk, msg, &record_type); if (ret) { @@ -1193,157 +1185,62 @@ int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) send_end: ret = sk_stream_error(sk, msg->msg_flags, ret); - - release_sock(sk); - mutex_unlock(&tls_ctx->tx_lock); return copied > 0 ? copied : ret; } -static int tls_sw_do_sendpage(struct sock *sk, struct page *page, - int offset, size_t size, int flags) +int tls_sw_sendmsg(struct sock *sk, struct msghdr *msg, size_t size) { - long timeo = sock_sndtimeo(sk, flags & MSG_DONTWAIT); struct tls_context *tls_ctx = tls_get_ctx(sk); - struct tls_sw_context_tx *ctx = tls_sw_ctx_tx(tls_ctx); - struct tls_prot_info *prot = &tls_ctx->prot_info; - unsigned char record_type = TLS_RECORD_TYPE_DATA; - struct sk_msg *msg_pl; - struct tls_rec *rec; - int num_async = 0; - ssize_t copied = 0; - bool full_record; - int record_room; - int ret = 0; - bool eor; - - eor = !(flags & MSG_SENDPAGE_NOTLAST); - sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); - - /* Call the sk_stream functions to manage the sndbuf mem. */ - while (size > 0) { - size_t copy, required_size; - - if (sk->sk_err) { - ret = -sk->sk_err; - goto sendpage_end; - } - - if (ctx->open_rec) - rec = ctx->open_rec; - else - rec = ctx->open_rec = tls_get_rec(sk); - if (!rec) { - ret = -ENOMEM; - goto sendpage_end; - } - - msg_pl = &rec->msg_plaintext; - - full_record = false; - record_room = TLS_MAX_PAYLOAD_SIZE - msg_pl->sg.size; - copy = size; - if (copy >= record_room) { - copy = record_room; - full_record = true; - } - - required_size = msg_pl->sg.size + copy + prot->overhead_size; - - if (!sk_stream_memory_free(sk)) - goto wait_for_sndbuf; -alloc_payload: - ret = tls_alloc_encrypted_msg(sk, required_size); - if (ret) { - if (ret != -ENOSPC) - goto wait_for_memory; - - /* Adjust copy according to the amount that was - * actually allocated. The difference is due - * to max sg elements limit - */ - copy -= required_size - msg_pl->sg.size; - full_record = true; - } - - sk_msg_page_add(msg_pl, page, copy, offset); - sk_mem_charge(sk, copy); - - offset += copy; - size -= copy; - copied += copy; - - tls_ctx->pending_open_record_frags = true; - if (full_record || eor || sk_msg_full(msg_pl)) { - ret = bpf_exec_tx_verdict(msg_pl, sk, full_record, - record_type, &copied, flags); - if (ret) { - if (ret == -EINPROGRESS) - num_async++; - else if (ret == -ENOMEM) - goto wait_for_memory; - else if (ret != -EAGAIN) { - if (ret == -ENOSPC) - ret = 0; - goto sendpage_end; - } - } - } - continue; -wait_for_sndbuf: - set_bit(SOCK_NOSPACE, &sk->sk_socket->flags); -wait_for_memory: - ret = sk_stream_wait_memory(sk, &timeo); - if (ret) { - if (ctx->open_rec) - tls_trim_both_msgs(sk, msg_pl->sg.size); - goto sendpage_end; - } + int ret; - if (ctx->open_rec) - goto alloc_payload; - } + if (msg->msg_flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | + MSG_CMSG_COMPAT | MSG_SPLICE_PAGES | + MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY)) + return -EOPNOTSUPP; - if (num_async) { - /* Transmit if any encryptions have completed */ - if (test_and_clear_bit(BIT_TX_SCHEDULED, &ctx->tx_bitmask)) { - cancel_delayed_work(&ctx->tx_work.work); - tls_tx_records(sk, flags); - } - } -sendpage_end: - ret = sk_stream_error(sk, flags, ret); - return copied > 0 ? copied : ret; + ret = mutex_lock_interruptible(&tls_ctx->tx_lock); + if (ret) + return ret; + lock_sock(sk); + ret = tls_sw_sendmsg_locked(sk, msg, size); + release_sock(sk); + mutex_unlock(&tls_ctx->tx_lock); + return ret; } int tls_sw_sendpage_locked(struct sock *sk, struct page *page, int offset, size_t size, int flags) { + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; + if (flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY | MSG_NO_SHARED_FRAGS)) return -EOPNOTSUPP; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - return tls_sw_do_sendpage(sk, page, offset, size, flags); + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return tls_sw_sendmsg_locked(sk, &msg, size); } int tls_sw_sendpage(struct sock *sk, struct page *page, int offset, size_t size, int flags) { - struct tls_context *tls_ctx = tls_get_ctx(sk); - int ret; + struct bio_vec bvec; + struct msghdr msg = { .msg_flags = flags | MSG_SPLICE_PAGES, }; if (flags & ~(MSG_MORE | MSG_DONTWAIT | MSG_NOSIGNAL | MSG_SENDPAGE_NOTLAST | MSG_SENDPAGE_NOPOLICY)) return -EOPNOTSUPP; + if (flags & MSG_SENDPAGE_NOTLAST) + msg.msg_flags |= MSG_MORE; - ret = mutex_lock_interruptible(&tls_ctx->tx_lock); - if (ret) - return ret; - lock_sock(sk); - ret = tls_sw_do_sendpage(sk, page, offset, size, flags); - release_sock(sk); - mutex_unlock(&tls_ctx->tx_lock); - return ret; + bvec_set_page(&bvec, page, size, offset); + iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, size); + return tls_sw_sendmsg(sk, &msg, size); } static int