From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03C43C25B46 for ; Mon, 23 Oct 2023 18:26:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7DC3B6B0141; Mon, 23 Oct 2023 14:26:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 78C966B0142; Mon, 23 Oct 2023 14:26:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 653EA6B0143; Mon, 23 Oct 2023 14:26:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 563F36B0141 for ; Mon, 23 Oct 2023 14:26:39 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1313C140A7E for ; Mon, 23 Oct 2023 18:26:39 +0000 (UTC) X-FDA: 81377556918.19.B88BE21 Received: from mail-ej1-f54.google.com (mail-ej1-f54.google.com [209.85.218.54]) by imf27.hostedemail.com (Postfix) with ESMTP id 399AC4001A for ; Mon, 23 Oct 2023 18:26:36 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=sHmVaA8t; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of yosryahmed@google.com designates 209.85.218.54 as permitted sender) smtp.mailfrom=yosryahmed@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1698085597; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oE+REpypGDMAev/H3GE245B5o6iKa1zk/siXumc4sBI=; b=npMLc9Cp4kAtncdukJe5GW34hlW1AFKdzl1UwFLANF6kRgHzUmyQK9KXlVt2AG1Zwzji3q 8y2xWqyyT1yX0Z8DhVP0UNJ8lIED0AxZT946YxqNKG4k0rL4yz9ZDONkZJUAuWQ9Qpm/7B El2krnkMGfdH+iJtP8uhwJzx2GcH20A= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=sHmVaA8t; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of yosryahmed@google.com designates 209.85.218.54 as permitted sender) smtp.mailfrom=yosryahmed@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1698085597; a=rsa-sha256; cv=none; b=ZgxDayfo5NZjq2NpzzVRCZNrx/+MoDSzJetrvGkCZjHrTyz/1XB/0eG6mLovVOSPiD3FMj 0TCCtbhVwc9/9I1nL6PKXIE89a03gBtRZgFf+R9+0hTS6KYEE96tRxQG1uOBrq2w21FlzM SELVksHHna8VHSdjNi4/zkSpAieKlNI= Received: by mail-ej1-f54.google.com with SMTP id a640c23a62f3a-9ad8a822508so546724066b.0 for ; Mon, 23 Oct 2023 11:26:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1698085595; x=1698690395; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=oE+REpypGDMAev/H3GE245B5o6iKa1zk/siXumc4sBI=; b=sHmVaA8tWj2o5+6s6Imx/BnA1NAus4F0Axv0YzXXago/Cm/s3+aqZx6nY/0gk39EFe ua3Whbr47wVaTiQ3CARK5+LHHqw4A53cE8NPZDaq+PW6gqxKA0d1Qhq6e1Gpjz+t/rwZ SsuOVDafpU9hY1+cMFvK8uv/p8/4ayNYJgfiNXLimB4M+QN94zQ1JxOlMji0q3tMOTaU s4cvVaRDpzucrcH7bZXBRLNvVLZ0DBaqzgCOYJZWpR8qE9lH+ouLUbtP+D6Yg65D9H4O kpse2MAVwJkR34qYfrRBWs9QDhb3OHUuM+fHOY7nOk4oMkzPTkvMtAFaPNLO8gL4TjZE HbhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1698085595; x=1698690395; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oE+REpypGDMAev/H3GE245B5o6iKa1zk/siXumc4sBI=; b=wOPrIosmf9KHpme8v8GhBErpotFOS8+H+TbwY6LcPIufMAdR5Asm9bBqOgLIsjOHdY OqzKIyVrgNb1PVl5N5tu7oytIP3cKM/RQruYgvy8mm9oYzFlaGwhZeEvcS8zwaexmyIK L1ZgdIwxkVeZHcCP8ADmRjhucojTk/+VPuH+iUfCZ76x9t5ZwwAcXSakE8+3F49bFvP+ fV2LMcH5ha8fjE8aHaPAi6sM3zhdS7dNj9O5HR47qMh+ewGQcTwxGuUGkozrUBQbs3Rx ee7kid/etShCSyQ4mFVlZ3qUwSAPrSUs+jKgpsFZOPt+2fZuq39qcIsVMf76Hta1D2+J Nfxg== X-Gm-Message-State: AOJu0Ywvrk10Jq3s8NpVkhRUasWWoKx4eWGn8dygtP/RQVe5eBkM28iW dN8b6YRkqWUs4sAwp7EpQNGhCIrcnuJZgicohrR0HQ== X-Google-Smtp-Source: AGHT+IG9GvP+J6p+a0G+WPAQIu946WVihLIDsq32dpZ9WN0/LYIVUoqcHZK5j4U7bjqT28mUlI/UmNUtjdVKdyLr1mU= X-Received: by 2002:a17:907:7295:b0:9bf:30e8:5bf9 with SMTP id dt21-20020a170907729500b009bf30e85bf9mr8656842ejc.4.1698085595352; Mon, 23 Oct 2023 11:26:35 -0700 (PDT) MIME-Version: 1.0 References: <20231010032117.1577496-4-yosryahmed@google.com> <202310202303.c68e7639-oliver.sang@intel.com> In-Reply-To: From: Yosry Ahmed Date: Mon, 23 Oct 2023 11:25:55 -0700 Message-ID: Subject: Re: [PATCH v2 3/5] mm: memcg: make stats flushing threshold per-memcg To: Feng Tang Cc: Shakeel Butt , "Sang, Oliver" , "oe-lkp@lists.linux.dev" , lkp , "cgroups@vger.kernel.org" , "linux-mm@kvack.org" , "Huang, Ying" , "Yin, Fengwei" , Andrew Morton , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Ivan Babrou , Tejun Heo , =?UTF-8?Q?Michal_Koutn=C3=BD?= , Waiman Long , "kernel-team@cloudflare.com" , Wei Xu , Greg Thelen , "linux-kernel@vger.kernel.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Stat-Signature: qtrm6inxyhcsh5xkme9zdqq1p6xnonzr X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 399AC4001A X-HE-Tag: 1698085596-761747 X-HE-Meta: U2FsdGVkX1+//NBMXSBtsIDo9VhifDV77mJcx3Deg5uM+yJewELuHvcqux4IvIf9eMyKhK1dEyVJ9MP4CFivfCa2n00tFxV9xEQhPRxbQKoPCP38kMmYm8eUZRfcx6K4QLreqBNJDjvRGhJ2oOTopSDM5tXd5a55QaokjQ87oHsFRzwai+Q0XkivT7P0NsepXhDnYJxO+8RfiSFV6hzNG/bb8VHrrg0FQJmoIqQYX+e5dS1IcjTL2azG42qYtYETMxpXqX0hHxluDfLA+lQHuIg1pFFvMDSy6ZsBzO8s4bezOLjqtpMt7Afa/exFNcDl9n05XiFKyrDGDGHsEjj7V8NswDGX+079nDb3G4W6XLx9ify78IgW65In6OUFhyprvCTdRO4QahRQGyMcwACKSjLKebHs/sks5sHytYNmRPX8C3Z1z5bxlcC8ARr4DjFoD/r3yV1/0nqF6+yNqCq1/Y4O7ILxmmPLsUwgRGYgXorA0QsGnk//K1SQ9hppscWdBCWIfM5XdzRPxZg+2yk+rQrv1tGBfAkt+4O9SBSILeh2SHoI75bqe0gmiUDKICVe9cMDYkg74Gl36nFnPzQ0tgnfEy/JXe4WVU2OpAxRzdvTYcndbDgwRkHNnTKLLwpfJosjZAvFSHKh0UlNzmiJpAvfH8xUt4Fml5sdjw9C39KOPV4kqLlz8IqBUIEoFk/DgrckPx/vesxi+XfWXZRB/mweFAIr+qAmzwHWBaej4+lboi7cA+uWYdNCPx4rmXB0LCiXpyqDflAA3aOfpebqpS2+QnQo4fqixkFSKrbPddopJWUszaCxoXsXcUHYiZoAH48RD7T7gI9F5ljrWRfesNTFbrruqEI0N3a3C2hprl0GFjljFOkj6jqDxXspwYPWSiFNl2SzY1MjSfrdVNK6kXzM9Xo5Q43o04a6DH+tENh0zgwypoBkGg4Qtq4J3i7wEGx9GI1OVXcdWcWsqO0 2OZudQuz iGxh44ux6yQxSaSPNAPWZR16WUfJDMVLxTXTCyI6uzAKRQ35aFx0ZMDMeuCZMqCgvF0umjIXS7Ol9d4f53ylb6VQeH6kwJR91xJb90YrJJKgMRyzab9pt//2hixDGczC+J/fvMIAGfSkzW6vywZFvM8ddHmq20DZ8zu0FqZ383h9ISIuOpforRB27nSzqcO8x1NWi0LPyPnoDPXjPwmO698IPMwE4UyRu2sCU6r+zYlNRyfk+XcuiZxIgOfnSHcazwhKBTceL2FpUXdM553fayGfJQlCijEWbZOOOmvqnIcYAyt8z0Kqo/+Gi2GbWICI5CKSwFNEx1I5SJG4S+U0D5Est6tZqFXcG9gB4cBy4UG8ng5dddX9PRBR5Z6x9TpKM3cbmaYgMakS2UV1e+m0fSQFA5atFewz8vyreLHwN7q84Itk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Sun, Oct 22, 2023 at 6:34=E2=80=AFPM Feng Tang wro= te: > > On Sat, Oct 21, 2023 at 01:42:58AM +0800, Yosry Ahmed wrote: > > On Fri, Oct 20, 2023 at 10:23=E2=80=AFAM Shakeel Butt wrote: > > > > > > On Fri, Oct 20, 2023 at 9:18=E2=80=AFAM kernel test robot wrote: > > > > > > > > > > > > > > > > Hello, > > > > > > > > kernel test robot noticed a -25.8% regression of will-it-scale.per_= thread_ops on: > > > > > > > > > > > > commit: 51d74c18a9c61e7ee33bc90b522dd7f6e5b80bb5 ("[PATCH v2 3/5] m= m: memcg: make stats flushing threshold per-memcg") > > > > url: https://github.com/intel-lab-lkp/linux/commits/Yosry-Ahmed/mm-= memcg-change-flush_next_time-to-flush_last_time/20231010-112257 > > > > base: https://git.kernel.org/cgit/linux/kernel/git/akpm/mm.git mm-e= verything > > > > patch link: https://lore.kernel.org/all/20231010032117.1577496-4-yo= sryahmed@google.com/ > > > > patch subject: [PATCH v2 3/5] mm: memcg: make stats flushing thresh= old per-memcg > > > > > > > > testcase: will-it-scale > > > > test machine: 104 threads 2 sockets (Skylake) with 192G memory > > > > parameters: > > > > > > > > nr_task: 100% > > > > mode: thread > > > > test: fallocate1 > > > > cpufreq_governor: performance > > > > > > > > > > > > In addition to that, the commit also has significant impact on the = following tests: > > > > > > > > +------------------+-----------------------------------------------= ----------------+ > > > > | testcase: change | will-it-scale: will-it-scale.per_thread_ops -3= 0.0% regression | > > > > | test machine | 104 threads 2 sockets (Skylake) with 192G memo= ry | > > > > | test parameters | cpufreq_governor=3Dperformance = | > > > > | | mode=3Dthread = | > > > > | | nr_task=3D50% = | > > > > | | test=3Dfallocate1 = | > > > > +------------------+-----------------------------------------------= ----------------+ > > > > > > > > > > Yosry, I don't think 25% to 30% regression can be ignored. Unless > > > there is a quick fix, IMO this series should be skipped for the > > > upcoming kernel open window. > > > > I am currently looking into it. It's reasonable to skip the next merge > > window if a quick fix isn't found soon. > > > > I am surprised by the size of the regression given the following: > > 1.12 =C4=85 5% +1.4 2.50 =C4=85 2% > > perf-profile.self.cycles-pp.__mod_memcg_lruvec_state > > > > IIUC we are only spending 1% more time in __mod_memcg_lruvec_state(). > > Yes, this is kind of confusing. And we have seen similar cases before, > espcially for micro benchmark like will-it-scale, stressng, netperf > etc, the change to those functions in hot path was greatly amplified > in the final benchmark score. > > In a netperf case, https://lore.kernel.org/lkml/20220619150456.GB34471@xs= ang-OptiPlex-9020/ > the affected functions have around 10% change in perf's cpu-cycles, > and trigger 69% regression. IIRC, micro benchmarks are very sensitive > to those statistics update, like memcg's and vmstat. > Thanks for clarifying. I am still trying to reproduce locally but I am running into some quirks with tooling. I may have to run a modified version of the fallocate test manually. Meanwhile, I noticed that the patch was tested without the fixlet that I posted [1] for it, understandably. Would it be possible to get some numbers with that fixlet? It should reduce the total number of contended atomic operations, so it may help. [1]https://lore.kernel.org/lkml/CAJD7tkZDarDn_38ntFg5bK2fAmFdSe+Rt6DKOZA7Sg= s_kERoVA@mail.gmail.com/ I am also wondering if aligning the stats_updates atomic will help. Right now it may share a cacheline with some items of the events_pending array. The latter may be dirtied during a flush and unnecessarily dirty the former, but the chances are slim to be honest. If it's easy to test such a diff, that would be nice, but I don't expect a lot of difference: diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 7cbc7d94eb65..a35fce653262 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -646,7 +646,7 @@ struct memcg_vmstats { unsigned long events_pending[NR_MEMCG_EVENTS]; /* Stats updates since the last flush */ - atomic64_t stats_updates; + atomic64_t stats_updates ____cacheline_aligned_in_smp; }; /*