From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32101C43334 for ; Mon, 27 Jun 2022 14:57:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A3CE16B0071; Mon, 27 Jun 2022 10:57:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9C5A66B0072; Mon, 27 Jun 2022 10:57:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 865228E0001; Mon, 27 Jun 2022 10:57:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 70BF86B0071 for ; Mon, 27 Jun 2022 10:57:00 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 448A3602D2 for ; Mon, 27 Jun 2022 14:57:00 +0000 (UTC) X-FDA: 79624318200.29.BF9F05F Received: from mail-yw1-f179.google.com (mail-yw1-f179.google.com [209.85.128.179]) by imf22.hostedemail.com (Postfix) with ESMTP id CA051C002B for ; Mon, 27 Jun 2022 14:56:59 +0000 (UTC) Received: by mail-yw1-f179.google.com with SMTP id 00721157ae682-3137316bb69so87870057b3.10 for ; Mon, 27 Jun 2022 07:56:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=V2nkfUAYBDu+tfL2X9orUiMGatP89Pe54EDyxN988kQ=; b=i+OGq2gIOrj1f7wJzQ/eJW7dkmfzigU161pJ9k36Z3eExzuYM+VOxhAoq8edk1Lj6g GI/bPHIeILex5TIvyQUh9j5heq/SmDq7hfMD81AoulOR5oXjLHEwbRahu/0IOZ6XBzcb kzJH4QV57nE3AOYV6nw52zlZNjxqVQ7rk8D+tA58OeeOHf+3PtKM96u1+os+o3UFPnqU qfVS2dWuH5VfyxtdEiJk6Rw0iPTSMhC9CdmRxAoqt3dPzOsnhCbsa288dFK5oepKzJzI IFAbYUxAe2mFHV+UT4i5fQJ53K6bAoD/8IXzOQMCLsQjhBZmYNgAleopWVgDXEb9DCuU PeGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=V2nkfUAYBDu+tfL2X9orUiMGatP89Pe54EDyxN988kQ=; b=EqQFYz6GzLKqyHz44iQc3L7dLTtYWn5+9FbB8I8cSFGC6WWKinlOjvhvVbLRcgkTl5 8FauZ6xPhq6bwx07jcjnuscSh1SLSLbz4FrQfFHlEEBHksWaS3R5nSRgNFdSDrs/7z5U 7qNizP8LitY1hJYdjpYow62NG4EdmyZ/VenUMx2DPeqBuZvZRNfcCerB4vvq6BzOjgwT WO/BaJB2ux8EkZgOACre3HvtDTkXE/FogxrP41ayBzJagBeF2cLoEmKMysM0B8IYUrTQ 0msZwZsxJj+Lm0ThZRxFDkU/wrNJ+CKhSTo1Buq2SDtC6efVfWB2GToEsZzdiB6M+VDH xZcQ== X-Gm-Message-State: AJIora/YKgFjUxg0mI9AxnmIeewxHN9bq4OUoxNWknTKlQgETu4F8Qll MO+vHZlJ/Jd4qKEaMHnHSA14rZr+YXOCobuFFk+F9KB09nfHxw== X-Google-Smtp-Source: AGRyM1u8B4kcLVWeZgzZxIUcRpqFs89OsvNHHhNb/IgGIfmRs5gfWD02FqhOdPn0RnPeAh7WgtQAQ18XliZui2xrm3U= X-Received: by 2002:a0d:df50:0:b0:317:9c40:3b8b with SMTP id i77-20020a0ddf50000000b003179c403b8bmr15687403ywe.332.1656341818790; Mon, 27 Jun 2022 07:56:58 -0700 (PDT) MIME-Version: 1.0 References: <20220623185730.25b88096@kernel.org> <20220624070656.GE79500@shbuild999.sh.intel.com> <20220624144358.lqt2ffjdry6p5u4d@google.com> <20220625023642.GA40868@shbuild999.sh.intel.com> <20220627023812.GA29314@shbuild999.sh.intel.com> <20220627123415.GA32052@shbuild999.sh.intel.com> In-Reply-To: From: Eric Dumazet Date: Mon, 27 Jun 2022 16:56:47 +0200 Message-ID: Subject: Re: [net] 4890b686f4: netperf.Throughput_Mbps -69.4% regression To: Shakeel Butt Cc: Feng Tang , Linux MM , Andrew Morton , Roman Gushchin , Michal Hocko , Johannes Weiner , Muchun Song , Jakub Kicinski , Xin Long , Marcelo Ricardo Leitner , kernel test robot , Soheil Hassas Yeganeh , LKML , network dev , linux-s390@vger.kernel.org, MPTCP Upstream , "linux-sctp @ vger . kernel . org" , lkp@lists.01.org, kbuild test robot , Huang Ying , Xing Zhengjun , Yin Fengwei , Ying Xu Content-Type: text/plain; charset="UTF-8" ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=i+OGq2gI; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of edumazet@google.com designates 209.85.128.179 as permitted sender) smtp.mailfrom=edumazet@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656341819; a=rsa-sha256; cv=none; b=lWS4kGxBIZ/ogv0sLRnNhtr/GD02asJJ9AevKL/h7eb2wCYfXIjQNoCbNMatQTDEJdtlxm Lz45hl393A0yWi0l0DvYXyVq7AxoM5HEP77LFvY6W7j8DpNbt6AVntAmBc7zmZtYBQSPFy HmUHo5ohQueJTyoh2XxDXokKlqiIGx8= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656341819; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=V2nkfUAYBDu+tfL2X9orUiMGatP89Pe54EDyxN988kQ=; b=2vzf3vtdSZNtFtMtIe11Z5nD8QzSKGlHXGOMkX/uqJPzapoA8nHbDcK3Ebw9j1STogbQ4a gCkJ1n9dcrOZq+jfcEIPjcXoRNT8NgjShgok3uPPPUYX+HhCbdwT0vRlxRmajY9A2kIGEL rIfoO4lSAXZlycj1XeJCfRMJXkKnmn4= X-Stat-Signature: jkt7xyi371oaxzys3r1tdy77t3i5u7xy X-Rspamd-Server: rspam08 X-Rspam-User: X-Rspamd-Queue-Id: CA051C002B Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=i+OGq2gI; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of edumazet@google.com designates 209.85.128.179 as permitted sender) smtp.mailfrom=edumazet@google.com X-HE-Tag: 1656341819-971615 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Jun 27, 2022 at 4:53 PM Shakeel Butt wrote: > Am I understanding correctly that this 69.4% (or 73.7%) regression is > with cgroup v2? > > Eric did the experiments on v2 but on real hardware where the > performance impact was negligible. > > BTW do you see similar regression for tcp as well or just sctp? TCP_RR with big packets can show a regression as well. I gave this perf profile: 28.69% [kernel] [k] copy_user_enhanced_fast_string 16.13% [kernel] [k] intel_idle_irq 6.46% [kernel] [k] page_counter_try_charge 6.20% [kernel] [k] __sk_mem_reduce_allocated 5.68% [kernel] [k] try_charge_memcg 5.16% [kernel] [k] page_counter_cancel And this points to false sharing on (struct page_counter *)->usage I guess memcg had free lunch, because of per-socket cache, that we need to remove.