LSFMMBPF proposal [MM]: Eliminate vmap/vmalloc lock contention

LSFMMBPF proposal [MM]: Eliminate vmap/vmalloc lock contention

[Uladzislau Rezki]

Hello, LSF.

Title: Introduce a per-cpu-vmap-cache to eliminate a vmap lock contention

Description:
 Currently the vmap code is not scaled to number of CPU cores in a system
 because a global vmap space is protected by a single spinlock. Such approach
 has a clear bottleneck if many CPUs simultaneously access to one resource.

 In this talk i would like to describe a drawback, show some data related
 to contentions and places where those occur in a code. Apart of that i
 would like to share ideas how to eliminate it providing a few approaches
 and compare them.

Requirements:
 * It should be a per-cpu approach;
 * Search of freed ptrs should not interfere with other freeing(as much as we can);
 *   - offload allocated areas(buzy ones) per-cpu;
 * Cache ready sized objects or merge them into one big per-cpu-space(split on demand);
 * Lazily-freed areas either drained per-cpu individually or by one CPU for all;
 * Prefetch a fixed size in front and allocate per-cpu 

Goals:
 * Implement a per-cpu way of allocation to eliminate a contention.

Thanks!

--
Uladzislau Rezki

Re: LSFMMBPF proposal [MM]: Eliminate vmap/vmalloc lock contention

[Lorenzo Stoakes]

On Tue, Feb 28, 2023 at 05:42:43PM +0100, Uladzislau Rezki wrote:
> Hello, LSF.
>
> Title: Introduce a per-cpu-vmap-cache to eliminate a vmap lock contention
>
> Description:
>  Currently the vmap code is not scaled to number of CPU cores in a system
>  because a global vmap space is protected by a single spinlock. Such approach
>  has a clear bottleneck if many CPUs simultaneously access to one resource.
>
>  In this talk i would like to describe a drawback, show some data related
>  to contentions and places where those occur in a code. Apart of that i
>  would like to share ideas how to eliminate it providing a few approaches
>  and compare them.
>
> Requirements:
>  * It should be a per-cpu approach;
>  * Search of freed ptrs should not interfere with other freeing(as much as we can);
>  *   - offload allocated areas(buzy ones) per-cpu;
>  * Cache ready sized objects or merge them into one big per-cpu-space(split on demand);
>  * Lazily-freed areas either drained per-cpu individually or by one CPU for all;
>  * Prefetch a fixed size in front and allocate per-cpu
>
> Goals:
>  * Implement a per-cpu way of allocation to eliminate a contention.
>
> Thanks!
>
> --
> Uladzislau Rezki
>

This is a really interesting topic and I'd really like this to go ahead as
there's certainly been a lot of discussion regarding vmalloc locking of late. I
for one would certainly attend it.

Re: [Lsf-pc] LSFMMBPF proposal [MM]: Eliminate vmap/vmalloc lock contention

[Michal Hocko]

On Tue 28-02-23 17:42:43, Uladzislau Rezki wrote:
> Hello, LSF.
> 
> Title: Introduce a per-cpu-vmap-cache to eliminate a vmap lock contention
> 
> Description:
>  Currently the vmap code is not scaled to number of CPU cores in a system
>  because a global vmap space is protected by a single spinlock. Such approach
>  has a clear bottleneck if many CPUs simultaneously access to one resource.
> 
>  In this talk i would like to describe a drawback, show some data related
>  to contentions and places where those occur in a code. Apart of that i
>  would like to share ideas how to eliminate it providing a few approaches
>  and compare them.

It's been some time since you brough this up. Has there been any
progress on the topic? Do you still find it important to discuss it at
LSFMM?

> Requirements:
>  * It should be a per-cpu approach;
>  * Search of freed ptrs should not interfere with other freeing(as much as we can);
>  *   - offload allocated areas(buzy ones) per-cpu;
>  * Cache ready sized objects or merge them into one big per-cpu-space(split on demand);
>  * Lazily-freed areas either drained per-cpu individually or by one CPU for all;
>  * Prefetch a fixed size in front and allocate per-cpu 
> 
> Goals:
>  * Implement a per-cpu way of allocation to eliminate a contention.
> 
> Thanks!
> 
> --
> Uladzislau Rezki
> _______________________________________________
> Lsf-pc mailing list
> Lsf-pc@lists.linux-foundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/lsf-pc

-- 
Michal Hocko
SUSE Labs

Re: [Lsf-pc] LSFMMBPF proposal [MM]: Eliminate vmap/vmalloc lock contention

[Uladzislau Rezki]

On Fri, Apr 21, 2023 at 02:03:43PM +0200, Michal Hocko wrote:
> On Tue 28-02-23 17:42:43, Uladzislau Rezki wrote:
> > Hello, LSF.
> > 
> > Title: Introduce a per-cpu-vmap-cache to eliminate a vmap lock contention
> > 
> > Description:
> >  Currently the vmap code is not scaled to number of CPU cores in a system
> >  because a global vmap space is protected by a single spinlock. Such approach
> >  has a clear bottleneck if many CPUs simultaneously access to one resource.
> > 
> >  In this talk i would like to describe a drawback, show some data related
> >  to contentions and places where those occur in a code. Apart of that i
> >  would like to share ideas how to eliminate it providing a few approaches
> >  and compare them.
> 
> It's been some time since you brough this up. Has there been any
> progress on the topic? Do you still find it important to discuss it at
> LSFMM?
> 
The idea about sequence was/is:

1) Give an overview on the proposal;
2) Submit patches to address the problem;
3) Start a discussion over lkml with people who are interested in it;
4) Send out a complete solution.

--
Uladzislau Rezki

Re: [Lsf-pc] LSFMMBPF proposal [MM]: Eliminate vmap/vmalloc lock contention

[Michal Hocko]

On Mon 24-04-23 09:44:00, Uladzislau Rezki wrote:
> On Fri, Apr 21, 2023 at 02:03:43PM +0200, Michal Hocko wrote:
> > On Tue 28-02-23 17:42:43, Uladzislau Rezki wrote:
> > > Hello, LSF.
> > > 
> > > Title: Introduce a per-cpu-vmap-cache to eliminate a vmap lock contention
> > > 
> > > Description:
> > >  Currently the vmap code is not scaled to number of CPU cores in a system
> > >  because a global vmap space is protected by a single spinlock. Such approach
> > >  has a clear bottleneck if many CPUs simultaneously access to one resource.
> > > 
> > >  In this talk i would like to describe a drawback, show some data related
> > >  to contentions and places where those occur in a code. Apart of that i
> > >  would like to share ideas how to eliminate it providing a few approaches
> > >  and compare them.
> > 
> > It's been some time since you brough this up. Has there been any
> > progress on the topic? Do you still find it important to discuss it at
> > LSFMM?
> > 
> The idea about sequence was/is:
> 
> 1) Give an overview on the proposal;
> 2) Submit patches to address the problem;
> 3) Start a discussion over lkml with people who are interested in it;
> 4) Send out a complete solution.

Thanks for the clarification. The usual LSFMM format is strongly
discussion focused. Long presentations are usually discouraged and they
should only introduce people to the underlying problem to kick of a
discussion.

That being said, IMO it would be helpful to have some material on the
mailing list before any discussion could be productive.

Thanks!
-- 
Michal Hocko
SUSE Labs

Re: [Lsf-pc] LSFMMBPF proposal [MM]: Eliminate vmap/vmalloc lock contention

[Uladzislau Rezki]

On Mon, Apr 24, 2023 at 10:55:20AM +0200, Michal Hocko wrote:
> On Mon 24-04-23 09:44:00, Uladzislau Rezki wrote:
> > On Fri, Apr 21, 2023 at 02:03:43PM +0200, Michal Hocko wrote:
> > > On Tue 28-02-23 17:42:43, Uladzislau Rezki wrote:
> > > > Hello, LSF.
> > > > 
> > > > Title: Introduce a per-cpu-vmap-cache to eliminate a vmap lock contention
> > > > 
> > > > Description:
> > > >  Currently the vmap code is not scaled to number of CPU cores in a system
> > > >  because a global vmap space is protected by a single spinlock. Such approach
> > > >  has a clear bottleneck if many CPUs simultaneously access to one resource.
> > > > 
> > > >  In this talk i would like to describe a drawback, show some data related
> > > >  to contentions and places where those occur in a code. Apart of that i
> > > >  would like to share ideas how to eliminate it providing a few approaches
> > > >  and compare them.
> > > 
> > > It's been some time since you brough this up. Has there been any
> > > progress on the topic? Do you still find it important to discuss it at
> > > LSFMM?
> > > 
> > The idea about sequence was/is:
> > 
> > 1) Give an overview on the proposal;
> > 2) Submit patches to address the problem;
> > 3) Start a discussion over lkml with people who are interested in it;
> > 4) Send out a complete solution.
> 
> Thanks for the clarification. The usual LSFMM format is strongly
> discussion focused. Long presentations are usually discouraged and they
> should only introduce people to the underlying problem to kick of a
> discussion.
> 
I have not posted yet any RFC and have not kicked it yet. Though people
are aware the problem.

>
> That being said, IMO it would be helpful to have some material on the
> mailing list before any discussion could be productive.
> 
This is what i have so far:

wget ftp://vps418301.ovh.net/incoming/Fix_a_vmalloc_lock_contention_in_SMP_env.pdf

I can, of course, move it forward over lkml only. If you are fully
booked or there other reason then please just withdraw my proposal
from your conference.

--
Uladzislau Rezki

Re: [Lsf-pc] LSFMMBPF proposal [MM]: Eliminate vmap/vmalloc lock contention

[Lorenzo Stoakes]

On Mon, Apr 24, 2023 at 12:08:22PM +0200, Uladzislau Rezki wrote:
> On Mon, Apr 24, 2023 at 10:55:20AM +0200, Michal Hocko wrote:
> > On Mon 24-04-23 09:44:00, Uladzislau Rezki wrote:
> > > On Fri, Apr 21, 2023 at 02:03:43PM +0200, Michal Hocko wrote:
> > > > On Tue 28-02-23 17:42:43, Uladzislau Rezki wrote:
> > > > > Hello, LSF.
> > > > >
> > > > > Title: Introduce a per-cpu-vmap-cache to eliminate a vmap lock contention
> > > > >
> > > > > Description:
> > > > >  Currently the vmap code is not scaled to number of CPU cores in a system
> > > > >  because a global vmap space is protected by a single spinlock. Such approach
> > > > >  has a clear bottleneck if many CPUs simultaneously access to one resource.
> > > > >
> > > > >  In this talk i would like to describe a drawback, show some data related
> > > > >  to contentions and places where those occur in a code. Apart of that i
> > > > >  would like to share ideas how to eliminate it providing a few approaches
> > > > >  and compare them.
> > > >
> > > > It's been some time since you brough this up. Has there been any
> > > > progress on the topic? Do you still find it important to discuss it at
> > > > LSFMM?
> > > >
> > > The idea about sequence was/is:
> > >
> > > 1) Give an overview on the proposal;
> > > 2) Submit patches to address the problem;
> > > 3) Start a discussion over lkml with people who are interested in it;
> > > 4) Send out a complete solution.
> >
> > Thanks for the clarification. The usual LSFMM format is strongly
> > discussion focused. Long presentations are usually discouraged and they
> > should only introduce people to the underlying problem to kick of a
> > discussion.
> >
> I have not posted yet any RFC and have not kicked it yet. Though people
> are aware the problem.
>
> >
> > That being said, IMO it would be helpful to have some material on the
> > mailing list before any discussion could be productive.
> >
> This is what i have so far:
>
> wget ftp://vps418301.ovh.net/incoming/Fix_a_vmalloc_lock_contention_in_SMP_env.pdf
>
> I can, of course, move it forward over lkml only. If you are fully
> booked or there other reason then please just withdraw my proposal
> from your conference.

For what it's worth I'm definitely interested in attending this session if
it goes ahead. I am sure Baoquan, if he's attending LSF/MM, would be too! (cc'd).

Re: [Lsf-pc] LSFMMBPF proposal [MM]: Eliminate vmap/vmalloc lock contention

[Baoquan He]

On 04/24/23 at 11:22am, Lorenzo Stoakes wrote:
> On Mon, Apr 24, 2023 at 12:08:22PM +0200, Uladzislau Rezki wrote:
> > On Mon, Apr 24, 2023 at 10:55:20AM +0200, Michal Hocko wrote:
> > > On Mon 24-04-23 09:44:00, Uladzislau Rezki wrote:
> > > > On Fri, Apr 21, 2023 at 02:03:43PM +0200, Michal Hocko wrote:
> > > > > On Tue 28-02-23 17:42:43, Uladzislau Rezki wrote:
> > > > > > Hello, LSF.
> > > > > >
> > > > > > Title: Introduce a per-cpu-vmap-cache to eliminate a vmap lock contention
> > > > > >
> > > > > > Description:
> > > > > >  Currently the vmap code is not scaled to number of CPU cores in a system
> > > > > >  because a global vmap space is protected by a single spinlock. Such approach
> > > > > >  has a clear bottleneck if many CPUs simultaneously access to one resource.
> > > > > >
> > > > > >  In this talk i would like to describe a drawback, show some data related
> > > > > >  to contentions and places where those occur in a code. Apart of that i
> > > > > >  would like to share ideas how to eliminate it providing a few approaches
> > > > > >  and compare them.
> > > > >
> > > > > It's been some time since you brough this up. Has there been any
> > > > > progress on the topic? Do you still find it important to discuss it at
> > > > > LSFMM?
> > > > >
> > > > The idea about sequence was/is:
> > > >
> > > > 1) Give an overview on the proposal;
> > > > 2) Submit patches to address the problem;
> > > > 3) Start a discussion over lkml with people who are interested in it;
> > > > 4) Send out a complete solution.
> > >
> > > Thanks for the clarification. The usual LSFMM format is strongly
> > > discussion focused. Long presentations are usually discouraged and they
> > > should only introduce people to the underlying problem to kick of a
> > > discussion.
> > >
> > I have not posted yet any RFC and have not kicked it yet. Though people
> > are aware the problem.
> >
> > >
> > > That being said, IMO it would be helpful to have some material on the
> > > mailing list before any discussion could be productive.
> > >
> > This is what i have so far:
> >
> > wget ftp://vps418301.ovh.net/incoming/Fix_a_vmalloc_lock_contention_in_SMP_env.pdf
> >
> > I can, of course, move it forward over lkml only. If you are fully
> > booked or there other reason then please just withdraw my proposal
> > from your conference.
> 
> For what it's worth I'm definitely interested in attending this session if
> it goes ahead. I am sure Baoquan, if he's attending LSF/MM, would be too! (cc'd).

Thanks for CC. 

Yes, I am very interested in this topic, have read it from the beginning.
Unfortunately, I can't attend LSF/MM this year because of some reasons.
I look forward to learning the decision or conclusion of the sessin, and
reviewing Uladzislau's RFC or formal patchset later.  Wish the session a
great success, and you guys a pleasant meeting and discussion.

Thanks
Baoquan

Re: LSFMMBPF proposal [MM]: Eliminate vmap/vmalloc lock contention

[Uladzislau Rezki]

>
> Hello, LSF.
>
> Title: Introduce a per-cpu-vmap-cache to eliminate a vmap lock contention
>
> Description:
>  Currently the vmap code is not scaled to number of CPU cores in a system
>  because a global vmap space is protected by a single spinlock. Such approach
>  has a clear bottleneck if many CPUs simultaneously access to one resource.
>
>  In this talk i would like to describe a drawback, show some data related
>  to contentions and places where those occur in a code. Apart of that i
>  would like to share ideas how to eliminate it providing a few approaches
>  and compare them.
>
> Requirements:
>  * It should be a per-cpu approach;
>  * Search of freed ptrs should not interfere with other freeing(as much as we can);
>  *   - offload allocated areas(buzy ones) per-cpu;
>  * Cache ready sized objects or merge them into one big per-cpu-space(split on demand);
>  * Lazily-freed areas either drained per-cpu individually or by one CPU for all;
>  * Prefetch a fixed size in front and allocate per-cpu
>
> Goals:
>  * Implement a per-cpu way of allocation to eliminate a contention.
>
Thank you for giving a time to talk about this topic on the LSFMMBPF.
The slides can
be accessed here:

wget ftp://vps418301.ovh.net/incoming/Mitigate_a_vmalloc_lock_contention_in_SMP_env_v1.pdf

There were some issues with a camera, but i was told in the end you
could see me.2

Thank you again!

-- 
Uladzislau Rezki