Re: [linux-next:master] [memcg] 70a64b7919: will-it-scale.per_process_ops -11.9% regression

[Oliver Sang <oliver.sang@intel.com>]

hi, Shakeel,

On Thu, May 23, 2024 at 09:47:30AM -0700, Shakeel Butt wrote:
> On Thu, May 23, 2024 at 03:48:40PM +0800, Oliver Sang wrote:
> > hi, Shakeel,
> > 
> > On Tue, May 21, 2024 at 09:18:19PM -0700, Shakeel Butt wrote:
> > > On Tue, May 21, 2024 at 10:43:16AM +0800, Oliver Sang wrote:
> > > > hi, Shakeel,
> > > > 
> > > [...]
> > > > 
> > > > we reported regression on a 2-node Skylake server. so I found a 1-node Skylake
> > > > desktop (we don't have 1 node server) to check.
> > > > 
> > > 
> > > Please try the following patch on both single node and dual node
> > > machines:
> > 
> > 
> > the regression is partially recovered by applying your patch.
> > (but one even more regression case as below)
> > 
> > details:
> > 
> > since you mentioned the whole patch-set behavior last time, I applied the
> > patch upon
> >   a94032b35e5f9 memcg: use proper type for mod_memcg_state
> > 
> > below fd2296741e2686ed6ecd05187e4 = a94032b35e5f9 + patch
> > 
> 
> Thanks a lot Oliver. I have couple of questions and requests:

you are welcome!

> 
> 1. What is the baseline kernel you are using? Is it linux-next or linus?
> If linux-next, which one specifically?

base is just 59142d87ab03b, which is in current linux-next/master,
and is already merged into linus/master now.

linux$ git rev-list linux-next/master | grep 59142d87ab03b
59142d87ab03b8ff969074348f65730d465f42ee

linux$ git rev-list linus/master | grep 59142d87ab03b
59142d87ab03b8ff969074348f65730d465f42ee


the data for it is the first column in the tables we supplied.

I just applied your patch upon a94032b35e5f9, so:

linux$ git log --oneline --graph fd2296741e2686ed6ecd05187e4
* fd2296741e268 fix for 70a64b7919 from Shakeel  <----- your fix patch
* a94032b35e5f9 memcg: use proper type for mod_memcg_state   <--- patch-set tip, I believe
* acb5fe2f1aff0 memcg: warn for unexpected events and stats
* 4715c6a753dcc mm: cleanup WORKINGSET_NODES in workingset
* 0667c7870a186 memcg: cleanup __mod_memcg_lruvec_state
* ff48c71c26aae memcg: reduce memory for the lruvec and memcg stats
* aab6103b97f1c mm: memcg: account memory used for memcg vmstats and lruvec stats
* 70a64b7919cbd memcg: dynamically allocate lruvec_stats   <--- we reported this as 'fbc' in original report
* 59142d87ab03b memcg: reduce memory size of mem_cgroup_events_index   <--- base


> 
> 2. What is the cgroup hierarchy where the workload is running? Is it
> running in the root cgroup?

Our test system uses systemd from the distribution (debian-12). The workload is
automatically assigned to a specific cgroup by systemd which is in the
sub-hierarchy of root, so it is not directly running in the root cgroup.

> 
> 3. For the followup experiments when needed, can you please remove the
> whole series (including 59142d87ab03b8ff) for the base numbers.

I cannot understand this very well, if the patch is to fix the regression
cause by this series, seems to me the best way is to apply this patch on top
of the series. anything I misunderstood here?

anyway, I could do that, do you mean such like v6.9, which doesn't include this
serial yet? I could use it as base, then apply your patch onto it. then check
the diff between v6.9 and v6.9+patch.

but I still have some concern that, what a big improvement show in this test
cannot guarantee there will be same improvement if comparing the series and
the series+patch

> 
> 4. My experiment [1] on Cooper Lake (2 node) and Skylake (1 node) shows
> significant improvement but I noticed that I am directly running
> page_fault2_processes with -t equal nr_cpus but you are running through
> runtest.py. Also it seems like lkp has modified runtest.py. I will try
> to run the same setup as yours to repro.
> 
> 
> [1] https://lore.kernel.org/all/20240523034824.1255719-1-shakeel.butt@linux.dev
> 
> thanks,
> Shakeel