RE: [PATCH net-next 1/2] net: Keep sk->sk_forward_alloc as a proper size

["Zhang, Cathy" <cathy.zhang@intel.com>]

> -----Original Message-----
> From: Shakeel Butt <shakeelb@google.com>
> Sent: Friday, May 12, 2023 1:07 PM
> To: Zhang, Cathy <cathy.zhang@intel.com>
> Cc: Eric Dumazet <edumazet@google.com>; Linux MM <linux-
> mm@kvack.org>; Cgroups <cgroups@vger.kernel.org>; Paolo Abeni
> <pabeni@redhat.com>; davem@davemloft.net; kuba@kernel.org;
> Brandeburg@google.com; Brandeburg, Jesse <jesse.brandeburg@intel.com>;
> Srinivas, Suresh <suresh.srinivas@intel.com>; Chen, Tim C
> <tim.c.chen@intel.com>; You, Lizhen <lizhen.you@intel.com>;
> eric.dumazet@gmail.com; netdev@vger.kernel.org
> Subject: Re: [PATCH net-next 1/2] net: Keep sk->sk_forward_alloc as a proper
> size
> 
> On Fri, May 12, 2023 at 03:23:45AM +0000, Zhang, Cathy wrote:
> > Remove the invalid mail addr added unintentionally.
> >
> 
> Sorry that was my buggy script.
> 
> [...]
> > >
> > > Hi Shakeel,
> > >
> > > Run with the temp change you provided,  the output shows it comes to
> > > drain_stock_1(), Here is the call trace:
> > >
> > >      8.96%  mc-worker        [kernel.vmlinux]            [k] page_counter_cancel
> > >             |
> > >              --8.95%--page_counter_cancel
> > >                        |
> > >                         --8.95%--page_counter_uncharge
> > >                                   drain_stock_1
> > >                                   __refill_stock
> > >                                   refill_stock
> > >                                   |
> > >                                    --8.88%--try_charge_memcg
> > >                                              mem_cgroup_charge_skmem
> > >                                              |
> > >                                               --8.87%--__sk_mem_raise_allocated
> > >                                                         __sk_mem_schedule
> > >                                                         |
> > >                                                         |--5.37%--tcp_try_rmem_schedule
> > >                                                         |          tcp_data_queue
> > >                                                         |          tcp_rcv_established
> > >                                                         |          tcp_v4_do_rcv
> >
> 
> Thanks a lot. This tells us that one or both of following scenarios are
> happening:
> 
> 1. In the softirq recv path, the kernel is processing packets from multiple
> memcgs.
> 
> 2. The process running on the CPU belongs to memcg which is different from
> the memcgs whose packets are being received on that CPU.

Thanks for sharing the points, Shakeel! Is there any trace records you want to
collect?

> 
> BTW have you seen this performance issue when you run the client and
> server on different machines? I am wondering if RFS would be good enough
> for such scenario and we only need to worry about the same machine case.

The same test scenario is tried between two machines which are on the same
network but with network topology, both of the two machines are with 1Gbit NIC.
There is no hot paths observed, but it might because that the systems are
not so busy, for memtier shows the average transfer speed is down to a few
tenth of original on the same machine.