From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17FB5C433EF for ; Fri, 24 Jun 2022 04:22:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6F6FC8E01C7; Fri, 24 Jun 2022 00:22:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 67F7D8E01BF; Fri, 24 Jun 2022 00:22:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 51FE78E01C7; Fri, 24 Jun 2022 00:22:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 3C0248E01BF for ; Fri, 24 Jun 2022 00:22:55 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 0871220C53 for ; Fri, 24 Jun 2022 04:22:55 +0000 (UTC) X-FDA: 79611833910.25.FBB097F Received: from mail-yw1-f177.google.com (mail-yw1-f177.google.com [209.85.128.177]) by imf01.hostedemail.com (Postfix) with ESMTP id A88BA40011 for ; Fri, 24 Jun 2022 04:22:54 +0000 (UTC) Received: by mail-yw1-f177.google.com with SMTP id 00721157ae682-318889e6a2cso13318867b3.1 for ; Thu, 23 Jun 2022 21:22:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=LrpVbauQhlY6sbHoa1+haYyof5/5LMZu+rrUVujXJmY=; b=DVqUHWwjqfSyUrKEEmRQ3laXS5aZCV+h8P/IkyQC8m5k2r+n3tUMsOoQKXSoh9WRlI m09tKWwK3GiflUWF0Dr20XIr6c7X7GVgN3HpaTX/FUe7X5ZdSfJ4ElUGmC7149/pIdQJ 9o0+iH8IHPVcRhmTCZQMzuuMazOytWvgsLgpPgHhvHqc3gB5h6ZXqeagU/N6bqv7xKHW U5kAm0QEVu6jjxLZp8nNu53bqGtb1ImVP9XTBXqxr0HqrLOQoUvfXuq/KF0SsTK5lMEB F8UaqHOrxK2yqpZS0IusbDmoUsv65akGOoFjouvrzFa14L3TLe/6MHJKyUHyZ6f5eNMe WH8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=LrpVbauQhlY6sbHoa1+haYyof5/5LMZu+rrUVujXJmY=; b=WqdTKS+EZ+vHVT+Uzluyce/NogMPhWv6Mi7J8Q1zfuTXuhsZDAT6QRKBC1y5VcBSGc ILklq1+XzPW5PZaYG/tns/XIIQZ9nSCDN5a8TM15R0M+6ZnsvIY2qE/IJdcWlFfaUNHp ZdCf1MBbvrlf6a/v0mjKTV2RbCTz96w5RAAOr0zIlNkZChDiITUCk9PrQnkghW7EU8ml kTOjLNlrwBRsJqe1mScrSmLaz5e9zKMmP50EXT1g7SZ6quat5f2ICJrUawfmoFdApiCM p7uinsrNmvsmKEUXL2JFWF0iYKluqt/J/bBGHfdUxm8q0gdugJcmyAClKA6a7xsJwGjE ynRg== X-Gm-Message-State: AJIora8ZLqfh5YPwWnTyejXVcGDdSrn6dSJPrANoQ8zsoG9DSRCnlvpW u1jcGyJbe4aW8599eY00nIPqOYEvfmq6RLj7g2YTsA== X-Google-Smtp-Source: AGRyM1sL30OODStpdjLZ8dh2J46BQTAQn1IcN8JlLsvqWmybh46yRWiK6BObk+Guv/QOX5qo3870LRterozi/iWkdf0= X-Received: by 2002:a81:9b93:0:b0:317:8c9d:4c22 with SMTP id s141-20020a819b93000000b003178c9d4c22mr14652446ywg.278.1656044573688; Thu, 23 Jun 2022 21:22:53 -0700 (PDT) MIME-Version: 1.0 References: <20220619150456.GB34471@xsang-OptiPlex-9020> <20220622172857.37db0d29@kernel.org> <20220623185730.25b88096@kernel.org> In-Reply-To: From: Eric Dumazet Date: Fri, 24 Jun 2022 06:22:42 +0200 Message-ID: Subject: Re: [net] 4890b686f4: netperf.Throughput_Mbps -69.4% regression To: Jakub Kicinski Cc: Xin Long , Marcelo Ricardo Leitner , kernel test robot , Shakeel Butt , Soheil Hassas Yeganeh , LKML , Linux Memory Management List , network dev , linux-s390@vger.kernel.org, MPTCP Upstream , "linux-sctp @ vger . kernel . org" , lkp@lists.01.org, kbuild test robot , Huang Ying , "Tang, Feng" , zhengjun.xing@linux.intel.com, fengwei.yin@intel.com, Ying Xu Content-Type: text/plain; charset="UTF-8" ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656044574; a=rsa-sha256; cv=none; b=ctidIA6rWyzZNUtfyHxmsKc7xDICsXbELiEFYQJAqY/w5EU0fyAO/tw3QImTrQTY7+S29Z 0K/sKF8nTW25KcDSOxprpqiGKTKHzc8mq5JlNMnHgbsk1ozc5W3YIrJFC/FV0fDZufOH9T Hjf/8TcmbnCBIvegcBi+6F603pINGms= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656044574; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LrpVbauQhlY6sbHoa1+haYyof5/5LMZu+rrUVujXJmY=; b=4B1SKL1NPj2EucS0NPJ1iybTxSMfUT1SOkAftIUvNF82VuQoNhTK1/meiT4diZ2y++wGLr Gs4oGCcBxiZ8/kakWS5Q82RNfdcsfWcOatekiiZ3Lr4fZPOm7vroY7ALVgI4NB0EOCzDJw 0FoTPXHom3qg7XQDUM+NJtUu3ssg414= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=DVqUHWwj; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of edumazet@google.com designates 209.85.128.177 as permitted sender) smtp.mailfrom=edumazet@google.com X-Stat-Signature: cyf4truyem3ucyteek8nrimd4ez35td3 X-Rspamd-Queue-Id: A88BA40011 Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=DVqUHWwj; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of edumazet@google.com designates 209.85.128.177 as permitted sender) smtp.mailfrom=edumazet@google.com X-Rspam-User: X-Rspamd-Server: rspam04 X-HE-Tag: 1656044574-460964 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Jun 24, 2022 at 6:13 AM Eric Dumazet wrote: > > On Fri, Jun 24, 2022 at 3:57 AM Jakub Kicinski wrote: > > > > On Thu, 23 Jun 2022 18:50:07 -0400 Xin Long wrote: > > > From the perf data, we can see __sk_mem_reduce_allocated() is the one > > > using CPU the most more than before, and mem_cgroup APIs are also > > > called in this function. It means the mem cgroup must be enabled in > > > the test env, which may explain why I couldn't reproduce it. > > > > > > The Commit 4890b686f4 ("net: keep sk->sk_forward_alloc as small as > > > possible") uses sk_mem_reclaim(checking reclaimable >= PAGE_SIZE) to > > > reclaim the memory, which is *more frequent* to call > > > __sk_mem_reduce_allocated() than before (checking reclaimable >= > > > SK_RECLAIM_THRESHOLD). It might be cheap when > > > mem_cgroup_sockets_enabled is false, but I'm not sure if it's still > > > cheap when mem_cgroup_sockets_enabled is true. > > > > > > I think SCTP netperf could trigger this, as the CPU is the bottleneck > > > for SCTP netperf testing, which is more sensitive to the extra > > > function calls than TCP. > > > > > > Can we re-run this testing without mem cgroup enabled? > > > > FWIW I defer to Eric, thanks a lot for double checking the report > > and digging in! > > I did tests with TCP + memcg and noticed a very small additional cost > in memcg functions, > because of suboptimal layout: > > Extract of an internal Google bug, update from June 9th: > > -------------------------------- > I have noticed a minor false sharing to fetch (struct > mem_cgroup)->css.parent, at offset 0xc0, > because it shares the cache line containing struct mem_cgroup.memory, > at offset 0xd0 > > Ideally, memcg->socket_pressure and memcg->parent should sit in a read > mostly cache line. > ----------------------- > > But nothing that could explain a "-69.4% regression" I guess the test now hits memcg limits more often, forcing expensive reclaim, and the memcg limits need some adjustments. Overall, tests enabling memcg should probably need fine tuning, I will defer to Intel folks. > > memcg has a very similar strategy of per-cpu reserves, with > MEMCG_CHARGE_BATCH being 32 pages per cpu. > > It is not clear why SCTP with 10K writes would overflow this reserve constantly. > > Presumably memcg experts will have to rework structure alignments to > make sure they can cope better > with more charge/uncharge operations, because we are not going back to > gigantic per-socket reserves, > this simply does not scale.