Re: [PATCH] Add LD_NUMA_REPLICATION for glibc

Re: [PATCH] Add LD_NUMA_REPLICATION for glibc

[Huang Shijie]

On Fri, Sep 03, 2021 at 08:28:57AM +0200, Florian Weimer wrote:
> * Huang Shijie via Libc-alpha:
> 
> > This patch adds LD_NUMA_REPLICATION which influences the linkage of shared libraries at run time.
> >
> > If LD_NUMA_REPLICATION is set for program foo like this:
> > 	#LD_NUMA_REPLICATION=1 ./foo
> >
> > At the time ld.so mmaps the shared libraries, it will uses
> > 	mmap(, c->prot | PROT_WRITE, MAP_COPY | MAP_FILE | MAP_POPULATE,)
> > for them, and the mmap will trigger COW(copy on write) for the shared
> > libraries at the NUMA node which the program `foo` runs. After the
> > COW, the foo will have a copy of the shared library segment(mmap
> > covered) which belong to the same NUMA node.
> >
> > So when enable LD_NUMA_REPLICATION, it will consume more memory,
> > but it will reduce the remote-access in NUMA.
> 
> I think the kernel could do this in a much better way, avoiding
> duplicating the pages within the same NUMA node.
I think maybe only the per-NUMA-node page cache can avoid it..

> 
> The other issue is the temporary RWX mapping, which does not
> interoperate well with some security hardening features.
> 
Thanks for the feedback.
I CC more people who may  have interesting about this.


Thanks
Huang Shijie

RE: [PATCH] Add LD_NUMA_REPLICATION for glibc

[Song Bao Hua (Barry Song)]

> -----Original Message-----
> From: Huang Shijie [mailto:shijie@os.amperecomputing.com]
> Sent: Saturday, September 4, 2021 3:16 AM
> To: Florian Weimer <fweimer@redhat.com>
> Cc: Huang Shijie via Libc-alpha <libc-alpha@sourceware.org>;
> carlos@systemhalted.org; zwang@amperecomputing.com;
> patches@amperecomputing.com; akpm@linux-foundation.org;
> torvalds@linux-foundation.org; viro@zeniv.linux.org.uk; willy@infradead.org;
> Song Bao Hua (Barry Song) <song.bao.hua@hisilicon.com>; linux-mm@kvack.org
> Subject: Re: [PATCH] Add LD_NUMA_REPLICATION for glibc
> 
> On Fri, Sep 03, 2021 at 08:28:57AM +0200, Florian Weimer wrote:
> > * Huang Shijie via Libc-alpha:
> >
> > > This patch adds LD_NUMA_REPLICATION which influences the linkage of shared
> libraries at run time.
> > >
> > > If LD_NUMA_REPLICATION is set for program foo like this:
> > > 	#LD_NUMA_REPLICATION=1 ./foo
> > >
> > > At the time ld.so mmaps the shared libraries, it will uses
> > > 	mmap(, c->prot | PROT_WRITE, MAP_COPY | MAP_FILE | MAP_POPULATE,)
> > > for them, and the mmap will trigger COW(copy on write) for the shared
> > > libraries at the NUMA node which the program `foo` runs. After the
> > > COW, the foo will have a copy of the shared library segment(mmap
> > > covered) which belong to the same NUMA node.
> > >
> > > So when enable LD_NUMA_REPLICATION, it will consume more memory,
> > > but it will reduce the remote-access in NUMA.
> >
> > I think the kernel could do this in a much better way, avoiding
> > duplicating the pages within the same NUMA node.
> I think maybe only the per-NUMA-node page cache can avoid it..
> 
> >
> > The other issue is the temporary RWX mapping, which does not
> > interoperate well with some security hardening features.
> >
> Thanks for the feedback.
> I CC more people who may  have interesting about this.
> 

Hi Shijie,
Thanks!

Could we have some benchmark data for this patch? I assume it
will benefit frontend-bound large workset which might face
relatively more icache miss?

> 
> Thanks
> Huang Shijie

Thanks
Barry

Re: [PATCH] Add LD_NUMA_REPLICATION for glibc

[Huang Shijie]

Hi Barry,
On Fri, Sep 03, 2021 at 10:16:14PM +0000, Song Bao Hua (Barry Song) wrote:
> Could we have some benchmark data for this patch? I assume it
> will benefit frontend-bound large workset which might face
> relatively more icache miss?
I did not find a proper program for the benchmark test..

Thanks
Huang Shijie