From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69313C433F5 for ; Mon, 13 Sep 2021 20:09:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E5069603E7 for ; Mon, 13 Sep 2021 20:09:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org E5069603E7 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 6CBF2900002; Mon, 13 Sep 2021 16:09:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 67B386B0071; Mon, 13 Sep 2021 16:09:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 51C03900002; Mon, 13 Sep 2021 16:09:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0143.hostedemail.com [216.40.44.143]) by kanga.kvack.org (Postfix) with ESMTP id 42F876B006C for ; Mon, 13 Sep 2021 16:09:25 -0400 (EDT) Received: from smtpin36.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id CD5A12C5A4 for ; Mon, 13 Sep 2021 20:09:24 +0000 (UTC) X-FDA: 78583639848.36.2B422B3 Received: from mail-lj1-f179.google.com (mail-lj1-f179.google.com [209.85.208.179]) by imf21.hostedemail.com (Postfix) with ESMTP id 871AAD02EE6C for ; Mon, 13 Sep 2021 20:09:24 +0000 (UTC) Received: by mail-lj1-f179.google.com with SMTP id l18so19419521lji.12 for ; Mon, 13 Sep 2021 13:09:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=SMsg97l00l07oCtcmcZSQu1PJBjVeWs3GkR0hZw6dIQ=; b=OEVqTP7bLY9vgVkI9Xz/mX9hc+2K43vUH/Ddy/z8+OQ8i3LKL8oMHKJNOMMEmT2FQA or6Ea/NN1JVc3aMwVFI57e3Y0X8YxzIE5TDaL6K3xoKXE2zuVKT71IF8nKc1zU0vjRgm uLu4v+NJ8b33vdbdCxLMVLh+H13FybIBCjiAfhFwt8+dB2mmH95uS2LKiGvqFN8yuBW5 R13Q4LkNAbWaXt1f7pKtAh6FRgKmfXRmg5f1uk9E5yOh/AhZkoabYxEDivGEOLF0nRyL DK7t6zyXUnjL2hxWx3+H8C0JWzWY10SaCafKxpTyhJ4woN/Cqi/DRWBoL5q4TWQmh8cE EMJw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=SMsg97l00l07oCtcmcZSQu1PJBjVeWs3GkR0hZw6dIQ=; b=Tw+xt8eWrG09ik+jxS39cEIBA+IqhO14QFxKPAwxhZuA4ZX22R42xP/PegEEI2k6KU cQfZdJjmLiKhadUA/GlIP3kCLSlHQTAsigxoS+x2gkVsAgHti4Xr7ENHxbme/zuqBkzW qt4Gh/rY8vKL7DFCYdcMJvu5SnVGHhyGtmj1suYC6SA+Bg77TF80+R7jo72FHp8BYsoj fPxG0yt0GHdk2wDjKz8RTWOndMwZh0qoaP/mfSqdQZUqqZ4G1izjp+Vaz26m1T139fJ7 AXxkzsD+V8FCr0hfkKPUbSlme01xRwARiSebnvXOl7/h4Zmp6JRUKBvJcnM50Gx4gZbx opZw== X-Gm-Message-State: AOAM532muNnaMYlgn3Ycl66lGlrdvFbULnayoCcyL2tgfExSf8eUL1hv cQGYdigmOHNZ7PsVdqULLZK89VD7hxLC8ZvP76+zvg== X-Google-Smtp-Source: ABdhPJyLyyBzBDlii/148UlbF+rM9IFcc8sAvJXelm26Rs/xwETntp1x/YIRfVq/35b9la5OaaKJh8KhhZJCRM0qVPc= X-Received: by 2002:a2e:9ed9:: with SMTP id h25mr11740309ljk.40.1631563762705; Mon, 13 Sep 2021 13:09:22 -0700 (PDT) MIME-Version: 1.0 References: <20210902215504.dSSfDKJZu%akpm@linux-foundation.org> <20210905124439.GA15026@xsang-OptiPlex-9020> <20210907033000.GA88160@shbuild999.sh.intel.com> <20210912111756.4158-1-hdanton@sina.com> <20210912132914.GA56674@shbuild999.sh.intel.com> In-Reply-To: From: Shakeel Butt Date: Mon, 13 Sep 2021 13:09:11 -0700 Message-ID: Subject: Re: [memcg] 45208c9105: aim7.jobs-per-min -14.0% regression To: Tejun Heo Cc: Feng Tang , Hillf Danton , LKML , Xing Zhengjun , Linux MM Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 871AAD02EE6C X-Stat-Signature: bzouq81a5muoa5q5kgpogury5rebsiq9 Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=OEVqTP7b; spf=pass (imf21.hostedemail.com: domain of shakeelb@google.com designates 209.85.208.179 as permitted sender) smtp.mailfrom=shakeelb@google.com; dmarc=pass (policy=reject) header.from=google.com X-HE-Tag: 1631563764-932967 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Sep 13, 2021 at 12:42 PM Tejun Heo wrote: > > Hello, > > On Mon, Sep 13, 2021 at 12:40:06PM -0700, Shakeel Butt wrote: > > I did one more experiment with same workload but with system_wq > > instead system_unbound_wq and there is clear difference in profile: > > > > With system_unbound_wq: > > - 4.63% 0.33% mmap [kernel.kallsyms] [k] queue_work_on > > 4.29% queue_work_on > > - __queue_work > > - 3.45% wake_up_process > > - try_to_wake_up > > - 2.46% ttwu_queue > > - 1.66% ttwu_do_activate > > - 1.14% activate_task > > - 0.97% enqueue_task_fair > > enqueue_entity > > > > With system_wq: > > - 1.36% 0.06% mmap [kernel.kallsyms] [k] queue_work_on > > 1.30% queue_work_on > > - __queue_work > > - 1.03% wake_up_process > > - try_to_wake_up > > - 0.97% ttwu_queue > > 0.66% ttwu_do_activate > > > > Tejun, is this expected? i.e. queuing work on system_wq has a > > different performance impact than on system_unbound_wq? > > Yes, system_unbound_wq is putting the work item on the global shared > workqueue while the system_wq is per-cpu, so on a loaded system, overhead > difference showing up isn't too surprising. > Thanks a lot for the explanation. Are there any concerns to call cgroup_rstat_flush_irqsafe(root_mem_cgroup->css.cgroup) in system_wq? This will be called every 2 seconds, so, we can assume the updated tree would be small most of the time. Thanks, Shakeel