From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A17EC4167B for ; Wed, 29 Nov 2023 03:12:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0BFA96B0392; Tue, 28 Nov 2023 22:12:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 06F786B0393; Tue, 28 Nov 2023 22:12:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E2AA96B0394; Tue, 28 Nov 2023 22:12:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D02A96B0392 for ; Tue, 28 Nov 2023 22:12:52 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id AD65AA04E0 for ; Wed, 29 Nov 2023 03:12:52 +0000 (UTC) X-FDA: 81509519784.20.5C4A62C Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) by imf01.hostedemail.com (Postfix) with ESMTP id DA9A440005 for ; Wed, 29 Nov 2023 03:12:50 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=G9r3zbaJ; spf=pass (imf01.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.210.180 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1701227570; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8GuKszrQDXABm8eNJf/PjFxMICrN4TD1NgLfUfAFkdU=; b=Nf93rf1Tl5rLWAmQ8qAJNPIAY566Sa4TvL9pC4BgNBAuQm38iIGG04jFcSiU7dJWaRh6pQ ylBhd13cGn9ni3haN8i9pVeivFJZOF1w+FhShHH7HtVwsfwbWWOhx0pyavMUIg1R8ARsSs ncJAqC/KhZDpvel4peUDaMefJIjWjBk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1701227570; a=rsa-sha256; cv=none; b=ijlF4krdgpYCPO+wvmVW9xPi4NJDJ35FH38gp/KpfyPeHVr0xxZuc4loBLLhkhSAlwKgDC a77TY7Ybmsm7tQhKSHPWO1h8wuqwVWt7bqE7eBzNDcLlKXMgIPCOFOLavvbW2iIKvx1aPQ He3701ZTUT52MPP05bC5bzN1nVFVjvw= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=G9r3zbaJ; spf=pass (imf01.hostedemail.com: domain of liangchen.linux@gmail.com designates 209.85.210.180 as permitted sender) smtp.mailfrom=liangchen.linux@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pf1-f180.google.com with SMTP id d2e1a72fcca58-6b20577ef7bso5460722b3a.3 for ; Tue, 28 Nov 2023 19:12:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1701227570; x=1701832370; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8GuKszrQDXABm8eNJf/PjFxMICrN4TD1NgLfUfAFkdU=; b=G9r3zbaJEOXsT8Q+ldHBELm+0jH8PvSLj6BjED8gRLWb5nNn6avuuO2MKLzwBt0VhK /TSSiPFj6wWNyQXAXqcoEy3EcR8Iu8s+vjBIRkTzNKobMU9B+yycfD7VUUAh0eKD7Bi/ 7e2SQsPhueyT25LQD5YEAechNPEfjiFm+BXyry+klM+g/yDUliR43fCy9nDgLuP+ZHQy g/dkvHOq57/lkcdmpsRWWk7qjvBsZSS+7Ig+W6/d4YOyARvW/MHoTmJZeWgapt2SPIlp rskc4xzDfGFT+oyCM2aCYzZjuOE90ES/gKzYjV2ducbGEMCi2Vnhrj0yRCgPtRSBKQ1R 2oFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1701227570; x=1701832370; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8GuKszrQDXABm8eNJf/PjFxMICrN4TD1NgLfUfAFkdU=; b=DmMa6VHjcuvRVIr9lOYqqIMTIXIDFT1xxR9fnGC7K3ruDul7C9UeCILIAm88X/Fdt/ 2+Vejfh++5hZlwgFm9OfDiYOoVJufTzKfSTC74kpkfYqVGqLt0P8D0y+OtdxPMYYD1OC urZZX3fFIc16z/bASkOYMvmXlSjPXhbYEEbiXmt8wILzeLt6yf/vaDB29kOMeSUYTOsj QZKaWLlz07lBEzIkoN+SdLF8lSgqAkC2UZMrjPif6hYiOzc8eEKcbgwBQ927c79+u8AR sj4cZfjpJ4+6EnUD9ZxwcziWkVhpW3C82ASNVor5rGOxaK05iJZ/1gFR9K+5BwpAW9kN Wz1A== X-Gm-Message-State: AOJu0Ywv+SCslQz3flC13nGyfjLQAcTPY7K3iJCpgWo7uDzIHZa3ovsJ PlPRgAPTvXGqxXF211FBmBo= X-Google-Smtp-Source: AGHT+IGsEEJdquJBPiG/GBkkobsxtcbFdcHzqMKOZN1SiLNC912zbQYtY1Un2ROe0Cj6zPrY6mX63g== X-Received: by 2002:a05:6a20:3d07:b0:18c:4105:9aa8 with SMTP id y7-20020a056a203d0700b0018c41059aa8mr15402788pzi.51.1701227569700; Tue, 28 Nov 2023 19:12:49 -0800 (PST) Received: from localhost.localdomain ([89.187.161.180]) by smtp.gmail.com with ESMTPSA id q10-20020a170902daca00b001cfc46baa40sm5669287plx.158.2023.11.28.19.12.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 28 Nov 2023 19:12:48 -0800 (PST) From: Liang Chen To: davem@davemloft.net, edumazet@google.com, kuba@kernel.org, pabeni@redhat.com, hawk@kernel.org, ilias.apalodimas@linaro.org, linyunsheng@huawei.com Cc: netdev@vger.kernel.org, linux-mm@kvack.org, liangchen.linux@gmail.com Subject: [PATCH net-next v4 4/4] skbuff: Optimization of SKB coalescing for page pool Date: Wed, 29 Nov 2023 11:12:01 +0800 Message-Id: <20231129031201.32014-5-liangchen.linux@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20231129031201.32014-1-liangchen.linux@gmail.com> References: <20231129031201.32014-1-liangchen.linux@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Stat-Signature: pzkxed8zou7j55fyewy9pysmxn7rizhd X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: DA9A440005 X-Rspam-User: X-HE-Tag: 1701227570-731237 X-HE-Meta: U2FsdGVkX19slvMPlA57a0mt4pWMgLSz22nCFaordybMnEULwUjLErgmamFFuNCvjrQ8oolWoP4GmpBJf264bs5Rk7kUsbE1Anb+gsxsKfBehcn10i+caRIzpvQ44PhKpzJ1w+P4mxx7TbfrVmI26UlV2UrZUU4Xfw6Nu8AVwFKIHshkgnMYRNM36ix/7VJ2sG6BtrZMDmjTD/srOzh4Nzs3VwYI3z3PTXCRVIRO5aU+m7vyzlfjt95ZEYQF0B/ot29WvHB/ABHHcG+02y4wrDU8u3GUdlT6vDbi0NZsFA+YEi2opZnJQVZeOco+tTXbaXfNLx40VwQ3gnu9lzdLX4JmViIffL8+9EdiLz77vBOn2gXvhKQCXUp+as37Vp3upuzwmssp/cNB9WOw/dAGjD462aHb96qj7VOHmfLmPv6AAK1rO/2QveAIuv7lJ0Jz0QEJk8geuLR0M9TnVDwKijFC6rNFWQapfDMfoPkY9Qf19926EY6SAvyWb1kDPSYqPj+t9gGIV9TKp3vHbTzbExCI+Ewiu9A9CFymIqPIzyYu550r9i7tH7YOZCYUV9Y6I7RQxB49wEK5TKSRFuBdvBUDPgBIHde8MiGfedsy/URfmrhArM3Z8vRldtJahs7OUcJBOrd/bUFVZILUA7ar8FZufTAWhjn1FMhFWooRxm5oINEnpDnUltmwB1o3dGOh/2wmf5a7fzKqvitVO2a+aUGYsL5fxaz+84/lYzdY179roh6SLdCk1dVi6LRyYVVmNFtQposJUrorPQiauwHsE5CAdeCkdD9i3v/Wj/SeUXHMjv1aeL27latik0OL96Ua4Yaf5daUKp+gZMc5CHsBXOw5NZjEY1fU772vbi38j+43FqdoryVbVjpbd2Yx6Ga41yFZ0HQkXoI6FjSguLnMfyRREIXyPppPjMIywEesaDA0ZhUWF5od9C2tb6o++Q5p5HNEkHA+RVxrmWpZdHb TIa80gnh qGFDhIbfFhgDxPv1DrUOMpoVVGWG0kSEJ3oa1ECDmcYpTv4ojN8R+J/J2BdksB+V5pX1tOvjczt2dMOpDy/rrhLwzgvr0SNmfmgN5Wu5/YjfbQYR7+YuHQGSxlyH0imZS0gSH5YWm4yomnJw9TvrUFyW2XM7+ndoZCq1ChFyrcJZqraZ1f5+KyBpHFCGZAijy51uvSPZsri5zZ0XQahhqdnOLZZ16pPA7PzA4EbutGqFAbzCrPHh4wftn++FfIKo/e0sD6Qpy5k7zwNUP5vG4Fbvqn1i8sEamFaK//UxX8NGdrF6xIPPllmnPa/kRo2nl+0sYMcxvttanW8Y40MBD4Z+aWI2Tqgtp8afsK9soLpUc+iGUvpiMkzQeV08Bmvm0z7ZYPtYCsgTpzB4GBj5PkkUayOmhh6877ASoGk17EBU1KDHGHRRBb05JWsgnS26AYx1PG5+okJpTo5t9bhu3eTl5laSf0oSqqPo7kzwTIyScBIhIxO6yWvxC0SdZrEKA5RLeHa0lMt/9viYlORgMS/eO9nuJroAwnIUTv2qmrsTrx90= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In order to address the issues encountered with commit 1effe8ca4e34 ("skbuff: fix coalescing for page_pool fragment recycling"), the combination of the following condition was excluded from skb coalescing: from->pp_recycle = 1 from->cloned = 1 to->pp_recycle = 1 However, with page pool environments, the aforementioned combination can be quite common(ex. NetworkMananger may lead to the additional packet_type being registered, thus the cloning). In scenarios with a higher number of small packets, it can significantly affect the success rate of coalescing. For example, considering packets of 256 bytes size, our comparison of coalescing success rate is as follows: Without page pool: 70% With page pool: 13% Consequently, this has an impact on performance: Without page pool: 2.57 Gbits/sec With page pool: 2.26 Gbits/sec Therefore, it seems worthwhile to optimize this scenario and enable coalescing of this particular combination. To achieve this, we need to ensure the correct increment of the "from" SKB page's page pool reference count (pp_ref_count). Following this optimization, the success rate of coalescing measured in our environment has improved as follows: With page pool: 60% This success rate is approaching the rate achieved without using page pool, and the performance has also been improved: With page pool: 2.52 Gbits/sec Below is the performance comparison for small packets before and after this optimization. We observe no impact to packets larger than 4K. packet size before after improved (bytes) (Gbits/sec) (Gbits/sec) 128 1.19 1.27 7.13% 256 2.26 2.52 11.75% 512 4.13 4.81 16.50% 1024 6.17 6.73 9.05% 2048 14.54 15.47 6.45% 4096 25.44 27.87 9.52% Signed-off-by: Liang Chen --- net/core/skbuff.c | 41 +++++++++++++++++++++++++++++------------ 1 file changed, 29 insertions(+), 12 deletions(-) diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 310207389f51..893d03264afc 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -947,6 +947,24 @@ static bool skb_pp_recycle(struct sk_buff *skb, void *data, bool napi_safe) return napi_pp_put_page(virt_to_page(data), napi_safe); } +/** + * skb_pp_get_frag_ref() - Increase fragment reference count of a page + * @page: page of the fragment on which to increase a reference + * + * Increase fragment reference count (pp_ref_count) on a page, but if it is + * not a page pool page, fallback to increase a reference(_refcount) on a + * normal page. + */ +static inline void skb_pp_get_frag_ref(struct page *page) +{ + struct page *head_page = compound_head(page); + + if (likely(skb_frag_is_pp_page(head_page))) + atomic_long_inc(&head_page->pp_ref_count); + else + get_page(head_page); +} + static void skb_kfree_head(void *head, unsigned int end_offset) { if (end_offset == SKB_SMALL_HEAD_HEADROOM) @@ -5769,17 +5787,12 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, return false; /* In general, avoid mixing page_pool and non-page_pool allocated - * pages within the same SKB. Additionally avoid dealing with clones - * with page_pool pages, in case the SKB is using page_pool fragment - * references (page_pool_alloc_frag()). Since we only take full page - * references for cloned SKBs at the moment that would result in - * inconsistent reference counts. - * In theory we could take full references if @from is cloned and - * !@to->pp_recycle but its tricky (due to potential race with - * the clone disappearing) and rare, so not worth dealing with. + * pages within the same SKB. In theory we could take full + * references if @from is cloned and !@to->pp_recycle but its + * tricky (due to potential race with the clone disappearing) and + * rare, so not worth dealing with. */ - if (to->pp_recycle != from->pp_recycle || - (from->pp_recycle && skb_cloned(from))) + if (to->pp_recycle != from->pp_recycle) return false; if (len <= skb_tailroom(to)) { @@ -5836,8 +5849,12 @@ bool skb_try_coalesce(struct sk_buff *to, struct sk_buff *from, /* if the skb is not cloned this does nothing * since we set nr_frags to 0. */ - for (i = 0; i < from_shinfo->nr_frags; i++) - __skb_frag_ref(&from_shinfo->frags[i]); + if (from->pp_recycle) + for (i = 0; i < from_shinfo->nr_frags; i++) + skb_pp_get_frag_ref(skb_frag_page(&from_shinfo->frags[i])); + else + for (i = 0; i < from_shinfo->nr_frags; i++) + __skb_frag_ref(&from_shinfo->frags[i]); to->truesize += delta; to->len += len; -- 2.31.1