From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C86EC61D99 for ; Thu, 23 Nov 2023 02:26:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9533B6B0624; Wed, 22 Nov 2023 21:26:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 902EE6B0627; Wed, 22 Nov 2023 21:26:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7A33F6B062C; Wed, 22 Nov 2023 21:26:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 652B36B0624 for ; Wed, 22 Nov 2023 21:26:01 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 3C7D1B6C07 for ; Thu, 23 Nov 2023 02:26:01 +0000 (UTC) X-FDA: 81487628922.10.35BCAA2 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf20.hostedemail.com (Postfix) with ESMTP id 72FEC1C000B for ; Thu, 23 Nov 2023 02:25:59 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="jr/593Dd"; spf=pass (imf20.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1700706359; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=FcUm7sKMXvG8tSKnNWjRclN75lSemXwn0ahaPo3EnXA=; b=Uq7mxpnc4PqUSYN2FkUCTabvR6//bdayzv2Ga65SlNONAsgC82o5KnJBnNxHM6aIL00qj1 Pv2aybHhy1zMUM3VRX8kWOQ3i5rHjZbciZhk4/g3Ex7imfKZ3k9+R1kWSI3T7+cD3chCKR 8+E/3dKaKZW6GL5hgLHNvgpTSUASVh8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1700706359; a=rsa-sha256; cv=none; b=xMQqsH5VDckjKaKQbnwr1GMQpQZ6JTartULiGN/sYPhd28AILITeJP/ocB4xuKUXYZybCX tDcHBNw6evlRz44y8h2j/lMNsDu7HVV2pAZR/KNMZf/1KXkLI2swONZdlCaS0mZxmnKpqf EabUDh3yGOh3M7rZU8TYYhgoGE2a9R8= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="jr/593Dd"; spf=pass (imf20.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.214.180 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-1ce627400f6so3104885ad.2 for ; Wed, 22 Nov 2023 18:25:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1700706358; x=1701311158; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=FcUm7sKMXvG8tSKnNWjRclN75lSemXwn0ahaPo3EnXA=; b=jr/593DdfXAavmfW5io/fbTNiS0hmCfmpN4ka5aOYlMtDfDhG1WTIA3FZYfznC6GmF 6vHETkUNbEuWyvNxgGeXiI9MsUS9EBT4+LKOt6iDeVfIbP98exT4xIxRBOrDCML8wUFb mymLDmkFM7qE4N1aKg2xcBGEi6HJT8CL8+ZKmfPdc9xlSMzPb62u9E2bJD1QExe7781e NUXQO63QpIeriRow2Sn68ehj+gwIS71v9RC3uzB6rQn+MvvU3h4iwmEo8r00uSxdGLuB v7z3mOsiZVe340t701V7RrZcsKkgHoMZ/snnSXIVvLkxfnq6DbZqxXxjyWkpxxFM0B6X zj9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1700706358; x=1701311158; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FcUm7sKMXvG8tSKnNWjRclN75lSemXwn0ahaPo3EnXA=; b=WO62gyuWodS6EDf6gBEEs5VNLwt8J9h/TdLtyxo1yea0P49YYxu/Z566Mq/Nxs95fZ KOZhjW7pW7bPBf71UL8YaT5Go3d8zlX4XPaLtxSEybSxMX/c3PdmiezrZKs37wN2hbKp KDGctyNtzrwqsz+fwVySmAorwHUTGwEUvJ1fxpb5XeNfqjmrkJW+McO5p1Z14eYfnCWA N3mwtPv1dE2ZDE0wKUp6XqmukfJpdoTgs78uSw1YrFqjMi2XoNZzzUlx4BJLu4ENM9Rr X77ehNS8s4y4EJgJJtZDbEd5n2KX6I4FfQ/PuV4GN3uxAMt/rEDHpTDM4DPu+1+UQgGz eRLQ== X-Gm-Message-State: AOJu0YzPiBIYZrR0mbHqj5l/jriuIBV78s1xNO7clNDgIIseISgrNaYN GOB2IJsl5fgGk/64fvnDe6k= X-Google-Smtp-Source: AGHT+IFiZNDN/L8IVBHtC08TXY6dWV+uBrdTTvHXRrC02gIXt8JsDtvM1M7tudA0hEvkiURg9laLqA== X-Received: by 2002:a17:902:d2cd:b0:1cf:6d8c:c8f1 with SMTP id n13-20020a170902d2cd00b001cf6d8cc8f1mr4677075plc.46.1700706358201; Wed, 22 Nov 2023 18:25:58 -0800 (PST) Received: from localhost.localdomain ([204.44.110.111]) by smtp.gmail.com with ESMTPSA id v11-20020a170902d68b00b001ce6452a741sm32880ply.25.2023.11.22.18.25.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 22 Nov 2023 18:25:57 -0800 (PST) From: Liang Chen To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, hawk@kernel.org, ilias.apalodimas@linaro.org, linyunsheng@huawei.com Cc: netdev@vger.kernel.org, linux-mm@kvack.org, liangchen.linux@gmail.com Subject: [PATCH net-next v2 3/3] skbuff: Optimization of SKB coalescing for page pool Date: Thu, 23 Nov 2023 10:25:16 +0800 Message-Id: <20231123022516.6757-3-liangchen.linux@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231123022516.6757-1-liangchen.linux@gmail.com> References: <20231123022516.6757-1-liangchen.linux@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 72FEC1C000B X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: xsjuyrkbeuy7zb5shhpd5o479qmdjj8k X-HE-Tag: 1700706359-700692 X-HE-Meta: U2FsdGVkX1/db03azMu2klXZv7uuWZ4KsQHMG0wFGczU/taljHn4YBdkOVemd8EThzS4JbUxGaBBJ8WlT3Ys3AoB2VxExTThhfEN8MXa8NL3RuhEgTpd23eWiY6qhwg7SqEh3a0GtJFqLaKXPf4Y4s7sWYn1i/agzCL/dDBboH3Z1yO/mYe79Y4pkIlaPUe2xZIGSpQ2CraGpy5aVTiM/Bhsz79tgwnAxPDPC5zcbN/zcN02gV6FK+eoTZLpbYjQa1Y3Dwr0E381B7mveuV16KpvnI3DwcWqXd09z3FrrdewH+BWXsMdYrC2Zz8B1h9M4PnB9RH0K4g97HEVN0O70QckcYVXJV1Lrp51zMdy8pP5HouGdiq1D8OqXyzQtHkSWZ1s72M9UX7gPW2PBgRmaY0+nfyKr7ItyJ25KpRyh9vJzNO5UXcr7ZgPmL2JbWvMvM/Pd7Sp2psoaD4GATldlrLE3RgfdqnkJoKW6dO6pHQeTpBllohCZl6/BJV4tNev+6NrVaSfxRWHbJ0OuQVHnCPzb2GjccuCCfVZlnv5FMlTwkqEQK+gXHREADL4BeVvvz1sIAJXSGSdejduZN3h6ZCp75eSV6QjDmxTJ12eImS0qLb9/7TcRZ6kUOyA/YuUl0Reg0IraHPP8jbiVUNCtQPwg/COFGKXmYUNyqaFYzOWRqbNs8IiRZKVCSOVHR59A3h3f3r7Wg08xeli8LVhKJJNpQ5iDDtwh5MfPFdWx3K6APXjNpiHlqy4dWWnIdYGQfAXw+EF3TrGw8WJlMWE0uzZzGi6kJouvSy+0In2N0vsRF3urd1u10VSGtXpbbRznyXrV5zidn0shf7CbsxgkQHu+lypbdAt6VZ9GqrGhlB4nEcTu9gI2LlifFE7UG0bbK2hJAxQTYpslMT8f4bMJ+a4VJVFfAPbwF2LPUdUzuCXGrHEWMXoTGbtOe/hlV4X67S8yxm3lqdt6bag8iZ TqTvdr0j 3GgmaF6doZPCpYsvo768QLmlIWHpFpud/tq6ZcbVIPep9yhNY0GJ+NMSgthyprbZ/PEqOJfSq08Q2NfbRbpdQcYAs8cActNRAjtNADq3a9VG0kcwBOFFO/usYOnKRh/+eyYbQhqSTqHvpRmSTjQD/721njwTrzMcyFpPamwKLI650hyDR8yKt92nzDdRdTBHYGPJbEdyajxIPnJVY4MjuCd0gwddD+m89ryARXbxFofz51cB/odonqWDXoctTfYgEUMahfYVFL+19zlVeytgBodfKha901kvzoRlWDK9/sHMEouakmlT9bB8KVTLhSfNDfqah50FnVN0qUV6uzBPOmrCrL7tKOqFui7JcuQSrCuRqcKoCXdE+RvfPb3QYFvqyHLO/6Iahzm6Ah3pSHAK73IF2afT95F8am9AXQODwE2h3RNV9CPEng4sG+Pbw9k2tLidhyXWt0D0G10OtY6NXAXmZ/k0XQ8P5KRphqnZXhbGFxL7e5Iu469+1aZYvuCXpDVZWTHx4f7nYqOtbVt4+hWKKqMDz6INR65ij44n3J9x35S8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In order to address the issues encountered with commit 1effe8ca4e34 ("skbuff: fix coalescing for page_pool fragment recycling"), the combination of the following condition was excluded from skb coalescing: from->pp_recycle = 1 from->cloned = 1 to->pp_recycle = 1 However, with page pool environments, the aforementioned combination can be quite common(ex. NetworkMananger may lead to the additional packet_type being registered, thus the cloning). In scenarios with a higher number of small packets, it can significantly affect the success rate of coalescing. For example, considering packets of 256 bytes size, our comparison of coalescing success rate is as follows: Without page pool: 70% With page pool: 13% Consequently, this has an impact on performance: Without page pool: 2.57 Gbits/sec With page pool: 2.26 Gbits/sec Therefore, it seems worthwhile to optimize this scenario and enable coalescing of this particular combination. To achieve this, we need to ensure the correct increment of the "from" SKB page's page pool reference count (pp_ref_count). Following this optimization, the success rate of coalescing measured in our environment has improved as follows: With page pool: 60% This success rate is approaching the rate achieved without using page pool, and the performance has also been improved: With page pool: 2.52 Gbits/sec Below is the performance comparison for small packets before and after this optimization. We observe no impact to packets larger than 4K. packet size before after improved (bytes) (Gbits/sec) (Gbits/sec) 128 1.19 1.27 7.13% 256 2.26 2.52 11.75% 512 4.13 4.81 16.50% 1024 6.17 6.73 9.05% 2048 14.54 15.47 6.45% 4096 25.44 27.87 9.52% Signed-off-by: Liang Chen --- Changes from v1: - make use of the unified pp fragment and non-fragment api --- include/net/page_pool/helpers.h | 22 ++++++++++++++++++++++ net/core/skbuff.c | 23 +++++++++++------------ 2 files changed, 33 insertions(+), 12 deletions(-) diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helpers.h index a6dc9412c9ae..8d2e3495f17d 100644 --- a/include/net/page_pool/helpers.h +++ b/include/net/page_pool/helpers.h @@ -402,4 +402,26 @@ static inline void page_pool_nid_changed(struct page_pool *pool, int new_nid) page_pool_update_nid(pool, new_nid); } +static inline bool page_pool_is_pp_page(struct page *page) +{ + return (page->pp_magic & ~0x3UL) == PP_SIGNATURE; +} + +/** + * page_pool_get_frag_ref() - Increase fragment reference count of a page + * @page: page of the fragment on which to increase a reference + * + * Increase fragment reference count (pp_ref_count) on a page, but if it is + * not a page pool page, fallback to increase a reference(_refcount) on a + * normal page. + */ +static inline void page_pool_get_frag_ref(struct page *page) +{ + struct page *head_page = compound_head(page); + + if (likely(page_pool_is_pp_page(head_page))) + atomic_long_inc(&head_page->pp_ref_count); + else + get_page(head_page); +} #endif /* _NET_PAGE_POOL_HELPERS_H */ diff --git a/net/core/skbuff.c b/net/core/skbuff.c index b157efea5dea..54e6945ead56 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -5764,17 +5764,12 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, return false; /* In general, avoid mixing page_pool and non-page_pool allocated - * pages within the same SKB. Additionally avoid dealing with clones - * with page_pool pages, in case the SKB is using page_pool fragment - * references (page_pool_alloc_frag()). Since we only take full page - * references for cloned SKBs at the moment that would result in - * inconsistent reference counts. - * In theory we could take full references if @from is cloned and - * !@to->pp_recycle but its tricky (due to potential race with - * the clone disappearing) and rare, so not worth dealing with. + * pages within the same SKB. In theory we could take full + * references if @from is cloned and !@to->pp_recycle but its + * tricky (due to potential race with the clone disappearing) and + * rare, so not worth dealing with. */ - if (to->pp_recycle != from->pp_recycle || - (from->pp_recycle && skb_cloned(from))) + if (to->pp_recycle != from->pp_recycle) return false; if (len <= skb_tailroom(to)) { @@ -5831,8 +5826,12 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, /* if the skb is not cloned this does nothing * since we set nr_frags to 0. */ - for (i = 0; i < from_shinfo->nr_frags; i++) - __skb_frag_ref(&from_shinfo->frags[i]); + if (from->pp_recycle) + for (i = 0; i < from_shinfo->nr_frags; i++) + page_pool_get_frag_ref(skb_frag_page(&from_shinfo->frags[i])); + else + for (i = 0; i < from_shinfo->nr_frags; i++) + __skb_frag_ref(&from_shinfo->frags[i]); to->truesize += delta; to->len += len; -- 2.31.1