[MAINTAINERS SUMMIT] Merge tree too flat?

[MAINTAINERS SUMMIT] Merge tree too flat?

[Jiri Kosina]

On last year's maintainer summit, there was a session where Linus asked 
the participants to bring up topics / questions [1].

I myself was the one raising the concern of the merge tree feeling a 
little bit too flat, and there seemed to be general agreement on that.

Checking the git repository as of now, it seems like we have not changed 
anything in that respect over the past year, and a lot of things are going 
directly to Linus although they could be cascaded a little bit better, 
contributing to better load-balancing on all maintainer levels (including 
the top-level one, of course).

I'd like to propose this as a proper discussion topic slot this year, by 
e.g. looking at:

- the actual numbers and current merge graph

- is this really something that could improve maintainer load (on all 
  levels) and throughput if done properly?

- how could we motivate maintainers to change the process and delegate 
  more into proper hierarchical sub-trees?

- potentially identify particular trees and changes that could be made in
  the merge graph/path to improve the process

[1] https://lwn.net/Articles/952146/

-- 
Jiri Kosina
SUSE Labs

Re: [MAINTAINERS SUMMIT] Merge tree too flat?

[Steven Rostedt]

On Wed, 5 Jun 2024 00:03:45 +0200 (CEST)
Jiri Kosina <jikos@kernel.org> wrote:

> On last year's maintainer summit, there was a session where Linus asked 
> the participants to bring up topics / questions [1].
> 
> I myself was the one raising the concern of the merge tree feeling a 
> little bit too flat, and there seemed to be general agreement on that.
> 
> Checking the git repository as of now, it seems like we have not changed 
> anything in that respect over the past year, and a lot of things are going 
> directly to Linus although they could be cascaded a little bit better, 
> contributing to better load-balancing on all maintainer levels (including 
> the top-level one, of course).

Note, the git tree may hide some hierarchy too. For example, I'm starting
to do pulls from Daniel Bristot for his rtla tooling. But since I have a
topic branch for his work, I just do a fetch from his pull request, and
then copy his pull request cover letter (with some tweaks) directly to Linus.

Although it may look like I'm pulling in Daniel's patches, I'm not. I
review them, but I'm more of a middle man between him and Linus.

> 
> I'd like to propose this as a proper discussion topic slot this year, by 
> e.g. looking at:
> 
> - the actual numbers and current merge graph

Now one thing I have started doing at Linus's request, is to make more
topic branches. This way if there's something that he doesn't like, he can
give it a line-item-veto, and I don't have to rebase everything.

This actually makes things easier to maintain. Perhaps maintainers should just
start splitting up their work? This may increase Linus's pull requests, but
it may also make it it easier for him to do those pulls. It is similar to
breaking up a change into many patches. Yes, there may be more patches to
review, but those patches are much easier to review than one big one. I
think the same can be said for pull requests.

The sad part is, the more topic branches you have, it will make the tree
look even flatter.

> 
> - is this really something that could improve maintainer load (on all 
>   levels) and throughput if done properly?

Topic branches.

> 
> - how could we motivate maintainers to change the process and delegate 
>   more into proper hierarchical sub-trees?

Now with more topic branches, it may be even easier to get others to take
over. Daniel does changes for rtla, and Masami Hiramatsu does all the probe
logic. Perhaps the first step is just to have maintainers split up their
work.

-- Steve

> 
> - potentially identify particular trees and changes that could be made in
>   the merge graph/path to improve the process
> 
> [1] https://lwn.net/Articles/952146/
> 

Re: [MAINTAINERS SUMMIT] Merge tree too flat?

[Jiri Kosina]

On Tue, 4 Jun 2024, Steven Rostedt wrote:

> Note, the git tree may hide some hierarchy too. For example, I'm starting
> to do pulls from Daniel Bristot for his rtla tooling. But since I have a
> topic branch for his work, I just do a fetch from his pull request, and
> then copy his pull request cover letter (with some tweaks) directly to Linus.

Right; that's exactly how it should be done in my view.

But if Daniel's tree has always fed into yours (no matter whether the 'git 
merge' way or 'apply patch' way), in doesn't really decrease the net 
effort one level above.

> > - is this really something that could improve maintainer load (on all 
> >   levels) and throughput if done properly?
> 
> Topic branches.

Oh, if you check hid.git, you'll see I am a huge fan of topic branches :)

We have a topic branch pretty much for every driver (and we have quite a 
number of them), plus one for patches to core.

And it has worked pretty well for many years already, the only downside I 
can imagine being that the ultimate 'for-linus' (or whatever else 
for-upstream) branch/tag is seeing quite a lot of driver-specific branches 
being merged. But I don't think that's a big problem.

> > - how could we motivate maintainers to change the process and delegate 
> >   more into proper hierarchical sub-trees?
> 
> Now with more topic branches, it may be even easier to get others to take
> over. Daniel does changes for rtla, and Masami Hiramatsu does all the probe
> logic. Perhaps the first step is just to have maintainers split up their
> work.

Agreed.

-- 
Jiri Kosina
SUSE Labs

Re: [MAINTAINERS SUMMIT] Merge tree too flat?

[Steven Rostedt]

On Wed, 5 Jun 2024 00:34:04 +0200 (CEST)
Jiri Kosina <jikos@kernel.org> wrote:

> On Tue, 4 Jun 2024, Steven Rostedt wrote:
> 
> > Note, the git tree may hide some hierarchy too. For example, I'm starting
> > to do pulls from Daniel Bristot for his rtla tooling. But since I have a
> > topic branch for his work, I just do a fetch from his pull request, and
> > then copy his pull request cover letter (with some tweaks) directly to Linus.  
> 
> Right; that's exactly how it should be done in my view.
> 
> But if Daniel's tree has always fed into yours (no matter whether the 'git 
> merge' way or 'apply patch' way), in doesn't really decrease the net 
> effort one level above.

I disagree. I use to take his patches and pull them in. But having him do
it, and also write the pull request, makes my job so much easier.

Note, I review his work, but not some much as if I were to review his
patches. I look at it at a different level when it's a pull request. I
trust Daniel enough to not go through his work with a fine comb, but
instead just look to make sure the general idea is sound.

How would imagine this being different than having a hierarchy. I would be
doing the same thing if I had done a merge. Actually, it is a merge, but I
do a "fast-forward" merge since he's the only one modifying the topic
branch. And it would look funny if I had done a merge, and then send the
same merge to Linus, where we have two merges back to back. Hence why I do
a fast-forward merge and use his pull request commit message in the one I
send to Linus.

-- Steve

Re: [MAINTAINERS SUMMIT] Merge tree too flat?

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1264 bytes --]

On Tue, Jun 04, 2024 at 06:45:06PM -0400, Steven Rostedt wrote:
> Jiri Kosina <jikos@kernel.org> wrote:

> > Right; that's exactly how it should be done in my view.

> > But if Daniel's tree has always fed into yours (no matter whether the 'git 
> > merge' way or 'apply patch' way), in doesn't really decrease the net 
> > effort one level above.

> I disagree. I use to take his patches and pull them in. But having him do
> it, and also write the pull request, makes my job so much easier.

> Note, I review his work, but not some much as if I were to review his
> patches. I look at it at a different level when it's a pull request. I
> trust Daniel enough to not go through his work with a fine comb, but
> instead just look to make sure the general idea is sound.

I don't think the mechanics of how patches get moved about has a huge
impact on the effort involved - trust and delegation make much more of
a difference.  I've got several areas where other people are reviewing
large volumes of patches before I ever see them, those take me almost no
time compared to what comes to me directly because of the level of trust
I place in these reviewers.  These differences aren't really obvious in
the git history but they're very real.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [MAINTAINERS SUMMIT] Merge tree too flat?

[Wolfram Sang]

[-- Attachment #1: Type: text/plain, Size: 1330 bytes --]


> I don't think the mechanics of how patches get moved about has a huge
> impact on the effort involved

50:50, in my case.

> - trust and delegation make much more of a difference.  I've got
> several areas where other people are reviewing large volumes of
> patches before I ever see them,

This is also true for me. When I called out for per-driver maintainers,
this made the flood of patches bearable. All of the driver maintainers
prefer to review only, though, and let me handle the rest. This is
totally fine with me. I'd likely lose some reviewers if I force to them
to provide me with separate branches.

The other part is: a few months ago Andi Shyti took over the maintenance
of the I2C controller patches. He updates patchwork, handles
dependencies, decides on for-current and for-next, writes pull
requests... That frees so much time that I have actually time left to
work on the core code again and give high-level guidance how to tackle
problems. So, this really helps as well.

That basically boils down to what I said last year at the Maintainer
Summit: In some parts of the Kernel, there is no flexibility to redesign
the hierarchy. The limited amount of people interested in maintaining
and their needs already shape the workflow. And for me, this is also the
primary reason why the merge-tree looks so flat.


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [MAINTAINERS SUMMIT] Merge tree too flat?

[Sasha Levin]

On Wed, Jun 05, 2024 at 12:03:45AM +0200, Jiri Kosina wrote:
>On last year's maintainer summit, there was a session where Linus asked
>the participants to bring up topics / questions [1].
>
>I myself was the one raising the concern of the merge tree feeling a
>little bit too flat, and there seemed to be general agreement on that.
>
>Checking the git repository as of now, it seems like we have not changed
>anything in that respect over the past year, and a lot of things are going
>directly to Linus although they could be cascaded a little bit better,
>contributing to better load-balancing on all maintainer levels (including
>the top-level one, of course).

I'm not sure we should be pushing for a more hierarchial tree. Yes, it's
flat, but is it the issue we're trying to address?

With heirarchy, we're asking folks to layer one on top of the other, but
for most subsystems this doesn't work as well (outside, maybe, drivers
that fall under a single subsystem).

I'd argue that what we want is more co-maintainer groups where several
folks share the burden. This, in turn, makes the tree look flat (all of
"x86" is one maintainer group, for example).

I'd also point to the LWN article you've linked: the flat model isn't
called out as an issue either by Linus or by the rest of the group
(which is what I recall from the discussion). Yes, it's flat, but the
solution isn't to make it a pyramid.

-- 
Thanks,
Sasha

Re: [MAINTAINERS SUMMIT] Merge tree too flat?

[Jiri Kosina]

On Tue, 4 Jun 2024, Sasha Levin wrote:

> I'm not sure we should be pushing for a more hierarchial tree. Yes, it's
> flat, but is it the issue we're trying to address?

That's one of the postulated questions :)

Distributing the merge load more evenly (by eliminating "hot spots" where 
a lot of merges are happening all the time) might help with the overload 
in some areas.

> I'd argue that what we want is more co-maintainer groups where several 
> folks share the burden. This, in turn, makes the tree look flat (all of 
> "x86" is one maintainer group, for example).

tip tree is a great example of this being implemented by a load of topic 
branches. Not much to be improved in that particular area, I'd say :)

Thanks,

-- 
Jiri Kosina
SUSE Labs