From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F8BCC43334 for ; Thu, 23 Jun 2022 22:50:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A4CE08E019D; Thu, 23 Jun 2022 18:50:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9FD258E0192; Thu, 23 Jun 2022 18:50:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8C40C8E019D; Thu, 23 Jun 2022 18:50:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7BDB18E0192 for ; Thu, 23 Jun 2022 18:50:36 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 52EB6614FC for ; Thu, 23 Jun 2022 22:50:36 +0000 (UTC) X-FDA: 79610996472.16.089A2D5 Received: from mail-oi1-f177.google.com (mail-oi1-f177.google.com [209.85.167.177]) by imf31.hostedemail.com (Postfix) with ESMTP id 02CDA2002B for ; Thu, 23 Jun 2022 22:50:35 +0000 (UTC) Received: by mail-oi1-f177.google.com with SMTP id u9so1338241oiv.12 for ; Thu, 23 Jun 2022 15:50:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=Y+Dg0mM96GxdgWzW8Q0806WYRu+tu0Ie+VcO+cMcI4U=; b=FYk61bvDh5JwGf5WdZIwNZW8B6+57IJxN4Nk3SgOVQUCUsYiOm3ErEX+kq3vz0b1jl ehO4ycV50KjmXNOG82d4ajlgj9CqbebkKJ4SWJJqHREo4pk+ilQPgbCv6IB6zHid79iP rqmZHNnlfQ/rPeGGBvDbNZN9ZMa975cK8vv+RyGLosEe6hd4C5Y35Ob1uvZYTwiiw7Y6 b7DLV4jzyNghScUUIIDElRVSlbKCQALz3tZIOKZJAGj2eZTZ+H5JRAhWbVmRsDvMSzlp R9wiUFpG2WjWK042boAletT6C5cPLJc0Zo+RjXzFx7sR8PaYb7ukt2dzVz3wkwtF6YIr Md9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=Y+Dg0mM96GxdgWzW8Q0806WYRu+tu0Ie+VcO+cMcI4U=; b=A/Ala6DeaE4UmzHNlp8y8rncadL1sF04QVuAznL/E98XGmu/e4oGS5ZWa9dP4c0ck6 VtHrFuIPonFBQMqNa3K+AgCeX/sZPFpCrQC9HlSEon24XteOPLZ6mobin3ngcdOtO9aQ LhqF83dQxY1paLVZTjveC5RQcul7D4wE6DwWyWITl+hxotvytLyShg8pW1K79I8BprYq 7pOIuZup8wyqcYlCJ6hQu0vTKRBWf/reTlmUMJ2wEygEbS83rHgJCRh7SZa3FtJseyIh xVewXdZ5spQvvW9+qnzG+hztNhmhRcOWebbitMyTLUDPIDQfjIcdl1rKhHHfR5nh12Pb rfCA== X-Gm-Message-State: AJIora8z7UiQQaX1pRgobFDeF8QBpdC0h6TVhCwA8VrXZZe36Af+lW7o GDAlKYlvioEmJPZ6ZBAXKmShMDItfCU//OeOR94= X-Google-Smtp-Source: AGRyM1uj/Q9s5zKn4L4uThaMvEnMiPzOqEF1sLtJVqTO12h9dB2vaIo0yv0Uh2i5b+OALrND/sp8sTGWW12NFpQK0Us= X-Received: by 2002:a05:6808:179a:b0:32f:fd4:3ad6 with SMTP id bg26-20020a056808179a00b0032f0fd43ad6mr199172oib.190.1656024634956; Thu, 23 Jun 2022 15:50:34 -0700 (PDT) MIME-Version: 1.0 References: <20220619150456.GB34471@xsang-OptiPlex-9020> <20220622172857.37db0d29@kernel.org> In-Reply-To: From: Xin Long Date: Thu, 23 Jun 2022 18:50:07 -0400 Message-ID: Subject: Re: [net] 4890b686f4: netperf.Throughput_Mbps -69.4% regression To: Jakub Kicinski Cc: Marcelo Ricardo Leitner , kernel test robot , Eric Dumazet , Shakeel Butt , Soheil Hassas Yeganeh , LKML , Linux Memory Management List , network dev , linux-s390@vger.kernel.org, mptcp@lists.linux.dev, "linux-sctp @ vger . kernel . org" , lkp@lists.01.org, kbuild test robot , Huang Ying , feng.tang@intel.com, zhengjun.xing@linux.intel.com, fengwei.yin@intel.com, Ying Xu Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656024636; a=rsa-sha256; cv=none; b=WQjBD00QVLPqKs79bUdEUHMaMCFRyFSRaP/kUVrxJ7WiT/ksR+nn6fzT6UV9687JwU6qV4 VNZbpLsLDAVxRrJHoBIkzgQR9dfdt/1AfXxXWKtaqvP47h5EuYoHww+mwoGXJMarcuiz+r Q0gCSgq8LZ1xqWJLDyRLZ6nccuDuGqY= ARC-Authentication-Results: i=1; imf31.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=FYk61bvD; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf31.hostedemail.com: domain of lucien.xin@gmail.com designates 209.85.167.177 as permitted sender) smtp.mailfrom=lucien.xin@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656024636; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Y+Dg0mM96GxdgWzW8Q0806WYRu+tu0Ie+VcO+cMcI4U=; b=zsnql7omQwGsx7MvXnNkeX7RGvrNjLau4bosT25wsAY14xX3I5Wty8ToYNVgogHnoEU/q5 nMDTMwDxD3hzqtQHMXDXOC5qnKspHqU3eNEXglHPngSlquPcja4+xXUaWgfMGwIqrAlfsb 1HnCsLS56JLTJS2tjKVKKXivJujgDlg= X-Stat-Signature: r3jf9edi3m53xzsn5gafdeaytuzp5pst X-Rspam-User: X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 02CDA2002B Authentication-Results: imf31.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=FYk61bvD; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf31.hostedemail.com: domain of lucien.xin@gmail.com designates 209.85.167.177 as permitted sender) smtp.mailfrom=lucien.xin@gmail.com X-HE-Tag: 1656024635-276107 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Jun 22, 2022 at 11:08 PM Xin Long wrote: > > Yes, I'm working on it. I couldn't see the regression in my env with > the 'reproduce' script attached. > I will try with lkp tomorrow. > > Thanks. > > On Wed, Jun 22, 2022 at 8:29 PM Jakub Kicinski wrote: > > > > Could someone working on SCTP double check this is a real regression? > > Feels like the regression reports are flowing at such rate its hard > > to keep up. > > > > > > > > commit: > > > 7c80b038d2 ("net: fix sk_wmem_schedule() and sk_rmem_schedule() err= ors") > > > 4890b686f4 ("net: keep sk->sk_forward_alloc as small as possible") > > > > > > 7c80b038d23e1f4c 4890b686f4088c90432149bd6de > > > ---------------- --------------------------- > > > %stddev %change %stddev > > > \ | \ > > > 15855 -69.4% 4854 netperf.Throughput_Mbps > > > 570788 -69.4% 174773 netperf.Throughput_tota= l_Mbps ... > > > 0.00 +5.1 5.10 =C2=B1 5% perf-profile.callt= race.cycles-pp.__sk_mem_reduce_allocated.sctp_wfree.skb_release_head_state.= consume_skb.sctp_chunk_put > > > 0.17 =C2=B1141% +5.3 5.42 =C2=B1 6% perf-profile.= calltrace.cycles-pp.skb_release_head_state.consume_skb.sctp_chunk_put.sctp_= outq_sack.sctp_cmd_interpreter > > > 0.00 +5.3 5.35 =C2=B1 6% perf-profile.callt= race.cycles-pp.sctp_wfree.skb_release_head_state.consume_skb.sctp_chunk_put= .sctp_outq_sack > > > 0.00 +5.5 5.51 =C2=B1 6% perf-profile.callt= race.cycles-pp.__sk_mem_reduce_allocated.skb_release_head_state.kfree_skb_r= eason.sctp_recvmsg.inet_recvmsg > > > 0.00 +5.7 5.65 =C2=B1 6% perf-profile.callt= race.cycles-pp.skb_release_head_state.kfree_skb_reason.sctp_recvmsg.inet_re= cvmsg.____sys_recvmsg ... > > > 0.00 +4.0 4.04 =C2=B1 6% perf-profile.child= ren.cycles-pp.mem_cgroup_charge_skmem > > > 2.92 =C2=B1 6% +4.2 7.16 =C2=B1 6% perf-profile.= children.cycles-pp.sctp_outq_sack > > > 0.00 +4.3 4.29 =C2=B1 6% perf-profile.child= ren.cycles-pp.__sk_mem_raise_allocated > > > 0.00 +4.3 4.32 =C2=B1 6% perf-profile.child= ren.cycles-pp.__sk_mem_schedule > > > 1.99 =C2=B1 6% +4.4 6.40 =C2=B1 6% perf-profile.= children.cycles-pp.consume_skb > > > 1.78 =C2=B1 6% +4.6 6.42 =C2=B1 6% perf-profile.= children.cycles-pp.kfree_skb_reason > > > 0.37 =C2=B1 8% +5.0 5.40 =C2=B1 6% perf-profile.= children.cycles-pp.sctp_wfree > > > 0.87 =C2=B1 9% +10.3 11.20 =C2=B1 6% perf-profile.= children.cycles-pp.skb_release_head_state > > > 0.00 +10.7 10.66 =C2=B1 6% perf-profile.child= ren.cycles-pp.__sk_mem_reduce_allocated ... > > > 0.00 +1.2 1.19 =C2=B1 7% perf-profile.self.= cycles-pp.try_charge_memcg > > > 0.00 +2.0 1.96 =C2=B1 6% perf-profile.self.= cycles-pp.page_counter_uncharge > > > 0.00 +2.1 2.07 =C2=B1 5% perf-profile.self.= cycles-pp.page_counter_try_charge > > > 1.09 =C2=B1 8% +2.8 3.92 =C2=B1 6% perf-profile.= self.cycles-pp.native_queued_spin_lock_slowpath > > > 0.29 =C2=B1 6% +3.5 3.81 =C2=B1 6% perf-profile.= self.cycles-pp.sctp_eat_data > > > 0.00 +7.8 7.76 =C2=B1 6% perf-profile.self.= cycles-pp.__sk_mem_reduce_allocated >From the perf data, we can see __sk_mem_reduce_allocated() is the one using CPU the most more than before, and mem_cgroup APIs are also called in this function. It means the mem cgroup must be enabled in the test env, which may explain why I couldn't reproduce it. The Commit 4890b686f4 ("net: keep sk->sk_forward_alloc as small as possible") uses sk_mem_reclaim(checking reclaimable >=3D PAGE_SIZE) to reclaim the memory, which is *more frequent* to call __sk_mem_reduce_allocated() than before (checking reclaimable >=3D SK_RECLAIM_THRESHOLD). It might be cheap when mem_cgroup_sockets_enabled is false, but I'm not sure if it's still cheap when mem_cgroup_sockets_enabled is true. I think SCTP netperf could trigger this, as the CPU is the bottleneck for SCTP netperf testing, which is more sensitive to the extra function calls than TCP. Can we re-run this testing without mem cgroup enabled? Thanks.