Re: [PATCH 0/3] further damage-control lack of clone scalability

[Mateusz Guzik <mjguzik@gmail.com>]

On Sun, Nov 23, 2025 at 4:00 PM Matthew Wilcox <willy@infradead.org> wrote:
>
> On Sun, Nov 23, 2025 at 07:30:51AM +0100, Mateusz Guzik wrote:
> > When spawning and killing threads in separate processes in parallel the
> > primary bottleneck on the stock kernel is pidmap_lock, largely because
> > of a back-to-back acquire in the common case.
> >
> > Benchmark code at the end.
> >
> > With this patchset alloc_pid() only takes the lock once and consequently
> > alleviates the problem. While scalability improves, the lock remains the
> > primary bottleneck by a large margin.
> >
> > I believe idr is a poor choice for the task at hand to begin with, but
> > sorting out that out beyond the scope of this patchset. At the same time
> > any replacement would be best evaluated against a state where the
> > above relock problem is fixed.
>
> Good news!  The IDR is deprecated.  Bad news!  I'm not 100% sure that
> the XArray is quite appropriate for this usecase.  I am opposed to
> introducing more IDR APIs.  Have you looked at converting to the XArray?
> Or do you have a better data structure in mind than the XArray?
>

Hi Willy,

in other responses I outlined what I suspect would be a viable long
term solution, very much not idr-based.

However, that's not something I'm likely to implement anytime soon and
I doubt there is someone willing to pick up the matter.

Whatever the long term solution and who/when implements it, the
current code avoidably loses out on some performance because of at
least two lock acquires on each fork instead of one.

At the moment it is structured in a way which makes it possible to
take the lock once with minor effort.

Leaving this in the current state results in a minor risk that someone
will make changes which turn fixing the scalability issue into a
massive problem.

With my patch as-is I can suffer some pain and avoid modifying
idr_prealloc, but at the same time I don't think the modification at
hand is particularly problematic. Notably it does not change any of
the internals.

So the question is if by "opposed to introducing more IDR APIs" you
mean you are just not fond of it, or is it a straight up NAK. Per the
explanation above, I think the change is tolerable in its own right
and I provided reasoning why I'm adding it in the first place.

If the latter, I'll see about massaging this to drop locks and retry
memory alloc.