From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DAB0CC433EF for ; Tue, 14 Sep 2021 04:20:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5AF98610A6 for ; Tue, 14 Sep 2021 04:20:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 5AF98610A6 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id A50546B006C; Tue, 14 Sep 2021 00:20:27 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A00C16B0071; Tue, 14 Sep 2021 00:20:27 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8EF0E6B0072; Tue, 14 Sep 2021 00:20:27 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7EAC66B006C for ; Tue, 14 Sep 2021 00:20:27 -0400 (EDT) Received: from smtpin33.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 1329B2C6AF for ; Tue, 14 Sep 2021 04:20:27 +0000 (UTC) X-FDA: 78584877294.33.53B8747 Received: from mail-lj1-f172.google.com (mail-lj1-f172.google.com [209.85.208.172]) by imf13.hostedemail.com (Postfix) with ESMTP id C2EF0102FC70 for ; Tue, 14 Sep 2021 04:20:26 +0000 (UTC) Received: by mail-lj1-f172.google.com with SMTP id q21so21273444ljj.6 for ; Mon, 13 Sep 2021 21:20:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=FFQKtCicsoRb4jBcWE+kkWFD8gMvEX1ULgtaJcmBLyw=; b=QD7ueQYq88HtqxaEsTj/L85GKbnbMp0ZWRHoBv0wdr80fYWYMJU+rD90zFAif6gI2/ SllACLTYWOFZMczxZKPflirR8xl85X4ZsWtURxv18xDT8ztkh2ErclAEQNHH4B6KEz8R KHJq/oAXzxUlDoZRMEK1How/97yF6qcjC1MelZhukmgIYElLEQzhSK3Y/6hRtVdaX0EV 3vBQGk3Xet4ViTmmhTu0CYgpAP5F8wlrW4XI6TCoC/5siWiDG5gC5xzSBGLKsjgswcPZ 05SF9qPdLL7Qxc7cnOftGeR9zdErdGdAX5wuon+5Fe2beczbuQeOzdc+u1Z4Bmvsr6bO 8MIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=FFQKtCicsoRb4jBcWE+kkWFD8gMvEX1ULgtaJcmBLyw=; b=pXQBx6ZF2mE9p2qi/hYCmMk5dBniGmlSwG3U7488TI0pjPz3KHz/Qz/O0aC5JIA1Yn 8holSbR+6o9JxManqGYsJIv5soDYeqISooAqTKqvW+WJxT0qyL9p3PtL1LyzX0a6Td8R U9GXKYaArQFRyR4ckkPV09ZvIEzXvXyQsDhrtVYsbo8lEFntlR7+F6xHebAIXV/0GJ5k tQNiO/qu9n7251nARpnDdc3VkJOSVifpx4IBP+i3unWBi204zFsrYMCiaKXNIu5jGLyX bhnljrSxRGZ6sF3nBcQ0hrbWg7R5Bi9FtCr/+Vpe2JPuyH9pM0N9gk5LutNZ+wmi45ye HuCw== X-Gm-Message-State: AOAM530mZI5yaZLL8Y/ZW9wfwfrdRPjIPGicPJEfeaXZVFCGxbybmJzA B7x9tk+VRxUyPgeTQsxRXBAum7c8E3i4EjLxmc8DPQ== X-Google-Smtp-Source: ABdhPJwEnf76kghB83MZCvOcklnM3QjIzyqjoxjczjR5LrpD/y55RQxaUou/OJNCm19FPqMyQOV+rFJTF7zWeWM7Lig= X-Received: by 2002:a2e:9ed9:: with SMTP id h25mr13179989ljk.40.1631593224970; Mon, 13 Sep 2021 21:20:24 -0700 (PDT) MIME-Version: 1.0 References: <20210907033000.GA88160@shbuild999.sh.intel.com> <20210912111756.4158-1-hdanton@sina.com> <20210912132914.GA56674@shbuild999.sh.intel.com> <20210914021322.GE56674@shbuild999.sh.intel.com> In-Reply-To: <20210914021322.GE56674@shbuild999.sh.intel.com> From: Shakeel Butt Date: Mon, 13 Sep 2021 21:20:12 -0700 Message-ID: Subject: Re: [memcg] 45208c9105: aim7.jobs-per-min -14.0% regression To: Feng Tang Cc: Tejun Heo , Hillf Danton , LKML , Xing Zhengjun , Linux MM Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=QD7ueQYq; spf=pass (imf13.hostedemail.com: domain of shakeelb@google.com designates 209.85.208.172 as permitted sender) smtp.mailfrom=shakeelb@google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: tqy8z9cke4yzoscp885jrb16xk1ncm9q X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: C2EF0102FC70 X-HE-Tag: 1631593226-581240 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000014, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: wi On Mon, Sep 13, 2021 at 7:13 PM Feng Tang wrote: > > Hi Shakeel, > > On Mon, Sep 13, 2021 at 01:13:57PM -0700, Shakeel Butt wrote: > > On Mon, Sep 13, 2021 at 1:10 PM Tejun Heo wrote: > > > > > > On Mon, Sep 13, 2021 at 01:09:11PM -0700, Shakeel Butt wrote: > > > > Thanks a lot for the explanation. Are there any concerns to call > > > > cgroup_rstat_flush_irqsafe(root_mem_cgroup->css.cgroup) in system_w= q? > > > > This will be called every 2 seconds, so, we can assume the updated > > > > tree would be small most of the time. > > > > > > I can't think of a reason why this would be problematic. > > > > > > > Thanks again. > > > > Feng, is it possible to re-run these benchmarks with > > queue_work(system_wq) instead of queue_work(system_unbound_wq)? > > I just run the patch twice, and there was no obvious change, the > hotspot is still the spinlock. > > Thanks, > Feng > > $git-diff aa48e47e > diff --git a/mm/memcontrol.c b/mm/memcontrol.c > index 4d8c9af..fa9cae9 100644 > --- a/mm/memcontrol.c > +++ b/mm/memcontrol.c > @@ -683,7 +683,7 @@ void __mod_memcg_lruvec_state(struct lruvec *lruvec, = enum node_stat_item idx, > /* Update lruvec */ > __this_cpu_add(pn->lruvec_stats_percpu->state[idx], val); > if (!(__this_cpu_inc_return(stats_flush_threshold) % MEMCG_CHARGE= _BATCH)) > - queue_work(system_unbound_wq, &stats_flush_work); > + queue_work(system_wq, &stats_flush_work); > } > > > 7e1c0d6f58207e7e aa48e47e3906c332eaf1e5d7b58 1638eee6432c1a5175685a7945a > ---------------- --------------------------- --------------------------- > \ | \ | \ > 648.62 +243.0% 2224 =C2=B1 3% +246.1% 2244 = aim7.cpu > 588139 -13.4% 509421 -13.5% 508738 = aim7.jobs-per-min > 196.05 -13.4% 169.81 -13.5% 169.58 = aim7.jobs-per-min-per-task > 3.93 =C2=B1 3% +62.8 66.70 +63.4 67.37 = pp.child.native_queued_spin_lock_slowpath > 3.96 =C2=B1 4% +62.8 66.76 +63.5 67.46 = pp.child._raw_spin_lock_irqsave > 3.66 =C2=B1 5% +62.9 66.54 +63.6 67.22 = pp.child.lock_page_lruvec_irqsave > 0.00 +0.1 0.10 =C2=B1 5% +0.1 0.09 = pp.self.queue_work_on > 0.00 +0.2 0.18 =C2=B1 5% +0.2 0.22 = pp.self.cgroup_rstat_flush_locked > 0.00 +0.6 0.60 =C2=B1 7% +0.7 0.72 = =C2=B1 3% pp.self.mem_cgroup_css_rstat_flush > 0.19 +0.7 0.86 =C2=B1 5% +0.7 0.84 = =C2=B1 4% pp.self.cgroup_rstat_updated > 3.92 =C2=B1 3% +62.8 66.70 +63.4 67.37 = pp.self.native_queued_spin_lock_slowpath > > Thanks Feng, the spinlock in lock_page_lruvec_irqsave() is unrelated and not taken in __mod_memcg_lruvec_state(). If removing queue_work() from __mod_memcg_lruvec_state() removes the regression then I was expecting some improvement by replacing system_unbound_wq with system_wq. Anyways thanks for your help. I will run some benchmarks as well (the page fault ones from will-it-scale).