linus-next: improving functional testing for to-be-merged pull requests

linus-next: improving functional testing for to-be-merged pull requests

[Sasha Levin]

Hi folks,

The linux-next tree we all know and love is widely used by the kernel
community for integration work. It offers several advantages:

	1. Early detection of conflicts between matinainer trees

	2. Catching most new build errors/warnings

However, it faces significant testing challenges:

	1. Contains a mix of "ready-to-go" code and experimental additions

	2. A single "bad" piece of code can affect testing of everything else

	3. Low barrier of entry, encouraging inclusion over exclusion

	4. While linux-next offers early conflict resolution and
	identifies build issues, it is very difficult to actually test
	due to the abundance of runtime issues it tends to have

These factors combine to make linux-next a valuable tool for integration
but problematic for comprehensive testing.

During the Maintainer's Summit, Linus Torvalds expressed concerns about
the quality of testing that code receives before he pulls it. The
subsequent discussion side-tracked to the testability of linux-next, but
we didn't directly address Linus's original concern about pre-pull
testing quality.

In an attempt to address the concerns, we're trying out a new "linus-next"
tree is being created and maintained with the following characteristics:

	1. Composed of pull requests sent directly to Linus

	2. Contains branches destined for imminent inclusion by Linus

	3. Higher code quality expectation (these are pull requests that
	maintainers expect Linus to pull)

	4. Continuous tree (not daily tags like in linux-next),
	facilitating easier bisection

The linus-next tree aims to provide a more stable and testable
integration point compared to linux-next, addressing the runtime issues
that make testing linux-next challenging and focusing on code that's
about to be pulled by Linus.

linus-next is (expected to be) particularly effective before the merge
window opens, as maintainers tend to send their pull requests early,
allowing for more thorough testing of to-be-merged changes.

We also want to avoid altering the existing workflow. In particular:

	1. No increase in latency. If anything, the expectation is that
	the cadence of merges would be improved given that Linus will
	need to do less builds and tests.

	2. Require "sign up" for the tree like linux-next does. Instead,
	pull requests are monitored and grabbed directly from the
	mailing list.

Tree location: `git://git.kernel.org/pub/scm/linux/kernel/git/sashal/linus-next.git linus-next`

Current testing:
   - LKFT: https://qa-reports.linaro.org/lkft/sashal-linus-next/
   - KernelCI: https://t.ly/KEW7F

Feedback and suggestions for improving usability are welcome!

-- 
Thanks,
Sasha

Re: linus-next: improving functional testing for to-be-merged pull requests

[Matthieu Baerts]

Hi Sasha,

On 21/10/2024 18:07, Sasha Levin wrote:

(...)

> In an attempt to address the concerns, we're trying out a new "linus-next"
> tree is being created and maintained with the following characteristics:
> 
>     1. Composed of pull requests sent directly to Linus
> 
>     2. Contains branches destined for imminent inclusion by Linus
> 
>     3. Higher code quality expectation (these are pull requests that
>     maintainers expect Linus to pull)

That's a good idea! Thank you for putting this in place!

If you don't mind, I have some questions below.

>     4. Continuous tree (not daily tags like in linux-next),
>     facilitating easier bisection

What will happen when a pull request is rejected?

(...)

> We also want to avoid altering the existing workflow. In particular:
> 
>     1. No increase in latency. If anything, the expectation is that
>     the cadence of merges would be improved given that Linus will
>     need to do less builds and tests.
> 
>     2. Require "sign up" for the tree like linux-next does. Instead,
>     pull requests are monitored and grabbed directly from the
>     mailing list.

Out of curiosity: is it done automatically? Will it email someone when a
conflict is found?

(...)

> Current testing:
>   - LKFT: https://qa-reports.linaro.org/lkft/sashal-linus-next/
>   - KernelCI: https://t.ly/KEW7F

That's great to have more tests being executed! Who is going to monitor
the results? This task can quickly take time if this person also has to
check for false positives and flaky tests.

Are the maintainers supposed to regularly monitor the results for the
tests they are responsible for? Or will they be (automatically?) emailed
when there is a regression?

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: linus-next: improving functional testing for to-be-merged pull requests

[Bart Van Assche]

On 10/21/24 9:07 AM, Sasha Levin wrote:
> Current testing:
>    - LKFT: https://qa-reports.linaro.org/lkft/sashal-linus-next/
>    - KernelCI: https://t.ly/KEW7F

Hi Sasha,

Is blktests included in any of the above? If not, please consider 
including it. During the past few years we have noticed that the
test failures reported by this test suite are most of the time caused
by kernel bugs. Sometimes issues in the tests are discovered but this
is rare. See also https://github.com/osandov/blktests/.

Thanks,

Bart.

Re: linus-next: improving functional testing for to-be-merged pull requests

[Sasha Levin]

On Mon, Oct 21, 2024 at 10:24:53AM -0700, Bart Van Assche wrote:
>On 10/21/24 9:07 AM, Sasha Levin wrote:
>>Current testing:
>>   - LKFT: https://qa-reports.linaro.org/lkft/sashal-linus-next/
>>   - KernelCI: https://t.ly/KEW7F
>
>Hi Sasha,
>
>Is blktests included in any of the above? If not, please consider 
>including it. During the past few years we have noticed that the
>test failures reported by this test suite are most of the time caused
>by kernel bugs. Sometimes issues in the tests are discovered but this
>is rare. See also https://github.com/osandov/blktests/.

Hey Bart,

I don't plan on doing any tests on my own, but rather have our existing
CI infra (kernelci, LKFT, etc) deal with the actual testing part of
things.

AFAIK KernelCI if working on adding blktests support!

Thanks!

-- 
Thanks,
Sasha

Re: linus-next: improving functional testing for to-be-merged pull requests

[Sasha Levin]

On Mon, Oct 21, 2024 at 07:18:38PM +0200, Matthieu Baerts wrote:
>Hi Sasha,
>
>On 21/10/2024 18:07, Sasha Levin wrote:
>
>(...)
>
>> In an attempt to address the concerns, we're trying out a new "linus-next"
>> tree is being created and maintained with the following characteristics:
>>
>>     1. Composed of pull requests sent directly to Linus
>>
>>     2. Contains branches destined for imminent inclusion by Linus
>>
>>     3. Higher code quality expectation (these are pull requests that
>>     maintainers expect Linus to pull)
>
>That's a good idea! Thank you for putting this in place!

Thank you!

>If you don't mind, I have some questions below.
>
>>     4. Continuous tree (not daily tags like in linux-next),
>>     facilitating easier bisection
>
>What will happen when a pull request is rejected?

My mental playbook is:

1. If a pull request is just ignored, ping it in case it was forgotten.
2. If we have an explicit NACK, just revert the merge commit.

>(...)
>
>> We also want to avoid altering the existing workflow. In particular:
>>
>>     1. No increase in latency. If anything, the expectation is that
>>     the cadence of merges would be improved given that Linus will
>>     need to do less builds and tests.
>>
>>     2. Require "sign up" for the tree like linux-next does. Instead,
>>     pull requests are monitored and grabbed directly from the
>>     mailing list.
>
>Out of curiosity: is it done automatically? Will it email someone when a
>conflict is found?

So it's 80% automatic now: my scripts monitor emails using lei, parse
relevant ones and manage to extract the pull instructions out of them,
and then most of those pull requests just merge cleanly.

There are some with conflicts, but since Linus insists on having an
explanation for merge conflicts, those pull requsts contain those
instructions within them. In those cases I manually followed the
instructions to resolve the conflicts (which were trivial so far).

I'll likely send a mail out *only* if I see a non-trivial merge conflict
without an explanation in the body.

>(...)
>
>> Current testing:
>>   - LKFT: https://qa-reports.linaro.org/lkft/sashal-linus-next/
>>   - KernelCI: https://t.ly/KEW7F
>
>That's great to have more tests being executed! Who is going to monitor
>the results? This task can quickly take time if this person also has to
>check for false positives and flaky tests.
>
>Are the maintainers supposed to regularly monitor the results for the
>tests they are responsible for? Or will they be (automatically?) emailed
>when there is a regression?

I'm not sure about this part. While I look at it in and will likely send
a mail out if I see something fishy, the only change in workflow that I
hope will happen here is Linus looking at a dashboard or two before he
begins his daily merge session.

-- 
Thanks,
Sasha

Re: linus-next: improving functional testing for to-be-merged pull requests

[Luis Chamberlain]

On Mon, Oct 21, 2024 at 01:30:06PM -0400, Sasha Levin wrote:
> On Mon, Oct 21, 2024 at 10:24:53AM -0700, Bart Van Assche wrote:
> > On 10/21/24 9:07 AM, Sasha Levin wrote:
> > > Current testing:
> > >   - LKFT: https://qa-reports.linaro.org/lkft/sashal-linus-next/
> > >   - KernelCI: https://t.ly/KEW7F
> > 
> > Hi Sasha,
> > 
> > Is blktests included in any of the above? If not, please consider
> > including it. During the past few years we have noticed that the
> > test failures reported by this test suite are most of the time caused
> > by kernel bugs. Sometimes issues in the tests are discovered but this
> > is rare. See also https://github.com/osandov/blktests/.
> 
> Hey Bart,
> 
> I don't plan on doing any tests on my own, but rather have our existing
> CI infra (kernelci, LKFT, etc) deal with the actual testing part of
> things.
> 
> AFAIK KernelCI if working on adding blktests support!

We can automate blktests with kdevops today, if that's desirable let
me know.

  Luis

Re: linus-next: improving functional testing for to-be-merged pull requests

[Liam R. Howlett]

* Sasha Levin <sashal@kernel.org> [241021 12:32]:
> Hi folks,
> 
> The linux-next tree we all know and love is widely used by the kernel
> community for integration work. It offers several advantages:
> 
> 	1. Early detection of conflicts between matinainer trees
> 
> 	2. Catching most new build errors/warnings

Would it be difficult to catch branches that change things outside their
scope without correct SOB/RB/Acks?  Asking for a friend...

> 
> However, it faces significant testing challenges:
> 
> 	1. Contains a mix of "ready-to-go" code and experimental additions
> 
> 	2. A single "bad" piece of code can affect testing of everything else
> 
> 	3. Low barrier of entry, encouraging inclusion over exclusion
> 
> 	4. While linux-next offers early conflict resolution and
> 	identifies build issues, it is very difficult to actually test
> 	due to the abundance of runtime issues it tends to have
> 
> These factors combine to make linux-next a valuable tool for integration
> but problematic for comprehensive testing.
> 
> During the Maintainer's Summit, Linus Torvalds expressed concerns about
> the quality of testing that code receives before he pulls it. The
> subsequent discussion side-tracked to the testability of linux-next, but
> we didn't directly address Linus's original concern about pre-pull
> testing quality.
> 
> In an attempt to address the concerns, we're trying out a new "linus-next"

The names are really close, could it be something that's more than one
character different?

linus-pulled, linux-pending, linux-pr or something?  They are also the
same length, which adds to the parsing challenge on both typing and
reading.  I'm thinking I'll get emails about the wrong branch or send
them with the wrong branch specified - especially pre-coffee.

> tree is being created and maintained with the following characteristics:
> 
> 	1. Composed of pull requests sent directly to Linus
> 
> 	2. Contains branches destined for imminent inclusion by Linus
> 
> 	3. Higher code quality expectation (these are pull requests that
> 	maintainers expect Linus to pull)
> 
> 	4. Continuous tree (not daily tags like in linux-next),
> 	facilitating easier bisection
> 
> The linus-next tree aims to provide a more stable and testable
> integration point compared to linux-next, addressing the runtime issues
> that make testing linux-next challenging and focusing on code that's
> about to be pulled by Linus.

What about the people who send them late?  I know people get told not to
do that, but some people still do.  Those late pull requests would
potentially invalidate a lot of the testing in this scenario.

For example, if there was a late mm change, then anything using mm could
be affected.  That's a large subset.

Is there any cut-off time for testing?

Thanks,
Liam

Re: linus-next: improving functional testing for to-be-merged pull requests

[Sasha Levin]

On Mon, Oct 21, 2024 at 02:36:39PM -0400, Liam R. Howlett wrote:
>* Sasha Levin <sashal@kernel.org> [241021 12:32]:
>> Hi folks,
>>
>> The linux-next tree we all know and love is widely used by the kernel
>> community for integration work. It offers several advantages:
>>
>> 	1. Early detection of conflicts between matinainer trees
>>
>> 	2. Catching most new build errors/warnings
>
>Would it be difficult to catch branches that change things outside their
>scope without correct SOB/RB/Acks?  Asking for a friend...

Up to the guy in charge... I don't want to attempt and monitor a policy
that won't be enforced :)

If Linus wants to add this to the workflow (which is doable), then an
explicit ack would be great.

>> However, it faces significant testing challenges:
>>
>> 	1. Contains a mix of "ready-to-go" code and experimental additions
>>
>> 	2. A single "bad" piece of code can affect testing of everything else
>>
>> 	3. Low barrier of entry, encouraging inclusion over exclusion
>>
>> 	4. While linux-next offers early conflict resolution and
>> 	identifies build issues, it is very difficult to actually test
>> 	due to the abundance of runtime issues it tends to have
>>
>> These factors combine to make linux-next a valuable tool for integration
>> but problematic for comprehensive testing.
>>
>> During the Maintainer's Summit, Linus Torvalds expressed concerns about
>> the quality of testing that code receives before he pulls it. The
>> subsequent discussion side-tracked to the testability of linux-next, but
>> we didn't directly address Linus's original concern about pre-pull
>> testing quality.
>>
>> In an attempt to address the concerns, we're trying out a new "linus-next"
>
>The names are really close, could it be something that's more than one
>character different?
>
>linus-pulled, linux-pending, linux-pr or something?  They are also the
>same length, which adds to the parsing challenge on both typing and
>reading.  I'm thinking I'll get emails about the wrong branch or send
>them with the wrong branch specified - especially pre-coffee.

Maybe... I didn't think about the name too much because my thinking is
that it'll mostly be a behind-the-scenes for most folks outside of a
very small group.

>> tree is being created and maintained with the following characteristics:
>>
>> 	1. Composed of pull requests sent directly to Linus
>>
>> 	2. Contains branches destined for imminent inclusion by Linus
>>
>> 	3. Higher code quality expectation (these are pull requests that
>> 	maintainers expect Linus to pull)
>>
>> 	4. Continuous tree (not daily tags like in linux-next),
>> 	facilitating easier bisection
>>
>> The linus-next tree aims to provide a more stable and testable
>> integration point compared to linux-next, addressing the runtime issues
>> that make testing linux-next challenging and focusing on code that's
>> about to be pulled by Linus.
>
>What about the people who send them late?  I know people get told not to
>do that, but some people still do.  Those late pull requests would
>potentially invalidate a lot of the testing in this scenario.
>
>For example, if there was a late mm change, then anything using mm could
>be affected.  That's a large subset.
>
>Is there any cut-off time for testing?

I think that the answer here is the same as whatever Linus considers as
"too late".

My cron scripts will pick up the pull request quickly and send it out to
be tested, but it's up to Linus to decide if he wants to sit on it for
another day/week/month or not.

Again, not enforcing policy, that's up to the penguin in charge...

-- 
Thanks,
Sasha

Re: linus-next: improving functional testing for to-be-merged pull requests

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 4560 bytes --]

On Mon, Oct 21, 2024 at 12:07:13PM -0400, Sasha Levin wrote:

> During the Maintainer's Summit, Linus Torvalds expressed concerns about
> the quality of testing that code receives before he pulls it. The
> subsequent discussion side-tracked to the testability of linux-next, but
> we didn't directly address Linus's original concern about pre-pull
> testing quality.

> In an attempt to address the concerns, we're trying out a new "linus-next"
> tree is being created and maintained with the following characteristics:

> 	1. Composed of pull requests sent directly to Linus

> 	2. Contains branches destined for imminent inclusion by Linus

> 	3. Higher code quality expectation (these are pull requests that
> 	maintainers expect Linus to pull)

This feels kind of late for thorough testing, especially outside of the
merge window where it's mostly fixes.

> 	4. Continuous tree (not daily tags like in linux-next),
> 	facilitating easier bisection

Is this a pressing problem?  I routinely bisect -next, you have to base
things on Linus' tree (or pending-fixes) but otherwise it's not
especially problematic.  It is a bit helpful to remember commits that
you've run a given test on before to speed things up a bit but that's
not super essential.  The downside of not redoing the merge is that
once problematic stuff ends up in history it doesn't go away which can
itself be disruptive whenever there's multiple issues going on, you get
a window you have to skip through when bisecting.

> The linus-next tree aims to provide a more stable and testable
> integration point compared to linux-next, addressing the runtime issues
> that make testing linux-next challenging and focusing on code that's
> about to be pulled by Linus.

> linus-next is (expected to be) particularly effective before the merge
> window opens, as maintainers tend to send their pull requests early,
> allowing for more thorough testing of to-be-merged changes.

How many people send pull requests in advance, as opposed to getting
their tree stable, waiting for further testing, applying any fixes from
that testing and then sending a pull request after the merge window has
opened?

One big concern I have with this is that for non-fix changes the testing
coverage will be very heavily loaded to the end of the development cycle
and the merge window, with a lot of changes in flight at the same time.
That means it's going to be harder to effectively run more expensive
tests, and we'll potentially be a month or two on from when the changes
were landed by the time we actually get round to running whatever extra
tests get run here.

I do think it's useful to have coverage of specifically things actually
sent in pull requests, but I'm not sure we'd get much more thorough
testing than we already get via pending-fixes and -next.

> We also want to avoid altering the existing workflow. In particular:
> 
> 	1. No increase in latency. If anything, the expectation is that
> 	the cadence of merges would be improved given that Linus will
> 	need to do less builds and tests.

For fixes we do routinely have things landing within hours of the pull
request being sent, that's pretty aggressive for CI.

> 	2. Require "sign up" for the tree like linux-next does. Instead,
> 	pull requests are monitored and grabbed directly from the
> 	mailing list.

Like I mentioned above what's the plan for handling bad changes?  If an
issue is found in linus-next (BTW, I forsee confusion being caused by
typos there).  If a problem is found and then fixed with an incremental
patch which is sent in another pull request then the fix will eventually
land but if the offending patch is rebased out or the pull request just
dropped entirely then presumably someone will have to do something
manual.  How will that happen, and on what sort of schedule?

I do wonder if it's worth splitting the -next merge further and having
another pending-fixes type split where maintainers can differentiate
between changes where they're reasonably happy with the testing and
those that are bleeding edge.  It is volunteering Stephen to do even
more merges, and it does rely on people identifying what's stable, so
I'm not sure how useful it'd be but seems like it would mesh well with
the flow where people are doing their own stabalisation prior to sending
pull requests.  That's a different thing to increasing the coverage of
PRs like you're proposing here but it's another angle on the "-next is
too unstable" issue.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: linus-next: improving functional testing for to-be-merged pull requests

[Linus Torvalds]

On Mon, 21 Oct 2024 at 12:44, Sasha Levin <sashal@kernel.org> wrote:
>
> On Mon, Oct 21, 2024 at 02:36:39PM -0400, Liam R. Howlett wrote:
> >
> >Would it be difficult to catch branches that change things outside their
> >scope without correct SOB/RB/Acks?  Asking for a friend...
>
> Up to the guy in charge... I don't want to attempt and monitor a policy
> that won't be enforced :)
>
> If Linus wants to add this to the workflow (which is doable), then an
> explicit ack would be great.

Has this been a huge deal?  There are things like ABI changes
(function renames, dropping or adding arguments etc etc) that will
inevitably end up tree-wide, and honestly, it would often be totally
frustrating trying to get acks from every maintainer.

Is it hugely common? No. But still, I'm not convinced there is a sane
model for "outside their scope".

I mean, sure, if the change comes from an individual driver or
filesystem or something like that, then it generally has no business
touching anything else. So it *can* be an issue, but I don't see how
to sanely automate this kind of thing without it becoming a potential
huge annoyance.

Maybe other people don't mind, but I personally hate false positives
that you are then supposed to ignore when appropriate. It just makes
people ignore the things they *should* care about.

                 Linus

Re: linus-next: improving functional testing for to-be-merged pull requests

[Paul E. McKenney]

On Mon, Oct 21, 2024 at 12:07:13PM -0400, Sasha Levin wrote:
> Hi folks,
> 
> The linux-next tree we all know and love is widely used by the kernel
> community for integration work. It offers several advantages:
> 
> 	1. Early detection of conflicts between matinainer trees
> 
> 	2. Catching most new build errors/warnings
> 
> However, it faces significant testing challenges:
> 
> 	1. Contains a mix of "ready-to-go" code and experimental additions
> 
> 	2. A single "bad" piece of code can affect testing of everything else
> 
> 	3. Low barrier of entry, encouraging inclusion over exclusion
> 
> 	4. While linux-next offers early conflict resolution and
> 	identifies build issues, it is very difficult to actually test
> 	due to the abundance of runtime issues it tends to have
> 
> These factors combine to make linux-next a valuable tool for integration
> but problematic for comprehensive testing.
> 
> During the Maintainer's Summit, Linus Torvalds expressed concerns about
> the quality of testing that code receives before he pulls it. The
> subsequent discussion side-tracked to the testability of linux-next, but
> we didn't directly address Linus's original concern about pre-pull
> testing quality.

I have to ask...

Wouldn't more people testing -next result in more pressure to fix
linux-next problems quickly?  Or perhaps more pressure for people to
avoid linux-next?  But this later would also apply to a new linus-next.
Unless Linus were to start rejecting pull requests that had not been
in linu[sx]-next for "long enough", whatever that might be.  ;-)

							Thanx, Paul

> In an attempt to address the concerns, we're trying out a new "linus-next"
> tree is being created and maintained with the following characteristics:
> 
> 	1. Composed of pull requests sent directly to Linus
> 
> 	2. Contains branches destined for imminent inclusion by Linus
> 
> 	3. Higher code quality expectation (these are pull requests that
> 	maintainers expect Linus to pull)
> 
> 	4. Continuous tree (not daily tags like in linux-next),
> 	facilitating easier bisection
> 
> The linus-next tree aims to provide a more stable and testable
> integration point compared to linux-next, addressing the runtime issues
> that make testing linux-next challenging and focusing on code that's
> about to be pulled by Linus.
> 
> linus-next is (expected to be) particularly effective before the merge
> window opens, as maintainers tend to send their pull requests early,
> allowing for more thorough testing of to-be-merged changes.
> 
> We also want to avoid altering the existing workflow. In particular:
> 
> 	1. No increase in latency. If anything, the expectation is that
> 	the cadence of merges would be improved given that Linus will
> 	need to do less builds and tests.
> 
> 	2. Require "sign up" for the tree like linux-next does. Instead,
> 	pull requests are monitored and grabbed directly from the
> 	mailing list.
> 
> Tree location: `git://git.kernel.org/pub/scm/linux/kernel/git/sashal/linus-next.git linus-next`
> 
> Current testing:
>   - LKFT: https://qa-reports.linaro.org/lkft/sashal-linus-next/
>   - KernelCI: https://t.ly/KEW7F
> 
> Feedback and suggestions for improving usability are welcome!
> 
> -- 
> Thanks,
> Sasha
> 

Re: linus-next: improving functional testing for to-be-merged pull requests

[Kees Cook]

On October 21, 2024 9:07:13 AM PDT, Sasha Levin <sashal@kernel.org> wrote:
>In an attempt to address the concerns, we're trying out a new "linus-next"
>tree is being created and maintained with the following characteristics:
>
>	1. Composed of pull requests sent directly to Linus
>
>	2. Contains branches destined for imminent inclusion by Linus

But this means hours or a day or 2 at most.

>	3. Higher code quality expectation (these are pull requests that
>	maintainers expect Linus to pull)

Are people putting things in linux-next that they don't expect to send to Linus? That seems like the greater problem.

>	4. Continuous tree (not daily tags like in linux-next),
>	facilitating easier bisection

I'm not sure how useful that is given the very small time window to find bugs.

>The linus-next tree aims to provide a more stable and testable
>integration point compared to linux-next,

Why not just use linux-next? I don't understand how this is any different except that it provides very little time to do testing and will need manual conflict resolutions that have already been done in linux-next.

How about this, instead: no one sends -rc1 PRs to Linus that didn't go through -next. Just have a bot that replies to all PRs with a health check, and Linus can pull it if he thinks it looks good. 

For example, for a given PR, the bot can report:

- Were the patches CCed to a mailing list?
- A histogram of how long the patches were in next (to show bake times)
- Are any patches associated with test failures? (0day and many other CIs are already running tests against -next; parse those reports)

We could have a real pre-submit checker! :)

-Kees

-- 
Kees Cook

Re: linus-next: improving functional testing for to-be-merged pull requests

[Christoph Hellwig]

On Mon, Oct 21, 2024 at 09:54:53PM -0700, Kees Cook wrote:
> >	1. Composed of pull requests sent directly to Linus
> >
> >	2. Contains branches destined for imminent inclusion by Linus
> 
> But this means hours or a day or 2 at most.

Yeah.

> 
> >	3. Higher code quality expectation (these are pull requests that
> >	maintainers expect Linus to pull)
> 
> Are people putting things in linux-next that they don't expect to send to Linus? That seems like the greater problem.

They shouldn't.  If they do we do indeed have a problem.

> >	4. Continuous tree (not daily tags like in linux-next),
> >	facilitating easier bisection
> 
> I'm not sure how useful that is given the very small time window to find bugs.

Same.

> >The linus-next tree aims to provide a more stable and testable
> >integration point compared to linux-next,
> 
> Why not just use linux-next? I don't understand how this is any
> different except that it provides very little time to do testing and
> will need manual conflict resolutions that have already been done in
> linux-next.

Exactly!

> How about this, instead: no one sends -rc1 PRs to Linus that didn't go
> through -next. Just have a bot that replies to all PRs with a health
> check, and Linus can pull it if he thinks it looks good. 

Not just -rc1, otherwise agreed.

> For example, for a given PR, the bot can report:
> 
> - Were the patches CCed to a mailing list?
> - A histogram of how long the patches were in next (to show bake times)
> - Are any patches associated with test failures? (0day and many other
> CIs are already running tests against -next; parse those reports)
> 
> We could have a real pre-submit checker! :)

That would be very useful.  Items 1 and 2 should be trivial, 3 would
require a bit of work but would still be very useful.

Re: linus-next: improving functional testing for to-be-merged pull requests

[Geert Uytterhoeven]

On Tue, Oct 22, 2024 at 6:55 AM Kees Cook <kees@kernel.org> wrote:
> On October 21, 2024 9:07:13 AM PDT, Sasha Levin <sashal@kernel.org> wrote:
> >In an attempt to address the concerns, we're trying out a new "linus-next"
> >tree is being created and maintained with the following characteristics:
> >
> >       1. Composed of pull requests sent directly to Linus
> >
> >       2. Contains branches destined for imminent inclusion by Linus
>
> But this means hours or a day or 2 at most.
>
> >       3. Higher code quality expectation (these are pull requests that
> >       maintainers expect Linus to pull)
>
> Are people putting things in linux-next that they don't expect to send to Linus? That seems like the greater problem.

Exactly.

> >       4. Continuous tree (not daily tags like in linux-next),
> >       facilitating easier bisection
>
> I'm not sure how useful that is given the very small time window to find bugs.
>
> >The linus-next tree aims to provide a more stable and testable
> >integration point compared to linux-next,
>
> Why not just use linux-next? I don't understand how this is any different except that it provides very little time to do testing and will need manual conflict resolutions that have already been done in linux-next.

And many actual issues have been seen/worked around in linux-next
a few weeks before.  In addition, this might make people laxer
w.r.t. linux-next, thus increasing Stephen's work load, and making
linux-next worse for actual testing.

> How about this, instead: no one sends -rc1 PRs to Linus that didn't go through -next. Just have a bot that replies to all PRs with a health check, and Linus can pull it if he thinks it looks good.
>
> For example, for a given PR, the bot can report:
>
> - Were the patches CCed to a mailing list?

This check would be good to have for linux-next, too.

> - A histogram of how long the patches were in next (to show bake times)
> - Are any patches associated with test failures? (0day and many other CIs are already running tests against -next; parse those reports)
>
> We could have a real pre-submit checker! :)

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: linus-next: improving functional testing for to-be-merged pull requests

[Steven Rostedt]

On Mon, 21 Oct 2024 23:48:34 -0700
Christoph Hellwig <hch@infradead.org> wrote:

> > How about this, instead: no one sends -rc1 PRs to Linus that didn't go
> > through -next. Just have a bot that replies to all PRs with a health
> > check, and Linus can pull it if he thinks it looks good.   
> 
> Not just -rc1, otherwise agreed.

You mean have everything go into linux-next before going to Linus after -rc1?

I'm one that doesn't do this. That's because my code in linux-next
after -rc1 is for the next merge window, and the code I send to Linus
is only fixes for code I sent before -rc1. I tend to keep an "urgent"
and "core" branch. My "core" branch is everything I plan to send in the
next merge window and goes into linux-next (via being pulled into my
for-next branch). After I send my pull request to Linus, and he pulls
it in the merge window, that "core" branch becomes my "urgent" branch.

But when I find a bug that's in Linus's tree, I put the fix on top of
"urgent", run it through my test suite (takes 8 hours or so), then send
a pull request to Linus. My "urgent" branch doesn't go into linux-next
as it doesn't have changes that should affect others work, which is
what I think linux-next is mostly for. I also find known bugs in
Linus's tree to be high priority to be fixed (I stop what I'm doing to
get the fix out ASAP).

Now, if there was better testing from linux-next, maybe it would be
worth the time to push my urgent branch there for a bit. But so far I
haven't seen the benefit of doing that.

> 
> > For example, for a given PR, the bot can report:
> > 
> > - Were the patches CCed to a mailing list?
> > - A histogram of how long the patches were in next (to show bake times)
> > - Are any patches associated with test failures? (0day and many other
> > CIs are already running tests against -next; parse those reports)
> > 
> > We could have a real pre-submit checker! :)  
> 
> That would be very useful.  Items 1 and 2 should be trivial, 3 would
> require a bit of work but would still be very useful.

If there was more feedback to going into linux-next, that would be good.

-- Steve

Re: linus-next: improving functional testing for to-be-merged pull requests

[Benjamin Tissoires]

On Oct 21 2024, Kees Cook wrote:
> 
> 
> On October 21, 2024 9:07:13 AM PDT, Sasha Levin <sashal@kernel.org> wrote:
> >In an attempt to address the concerns, we're trying out a new "linus-next"
> >tree is being created and maintained with the following characteristics:
> >
> >	1. Composed of pull requests sent directly to Linus
> >
> >	2. Contains branches destined for imminent inclusion by Linus
> 
> But this means hours or a day or 2 at most.

Yeah :/

> 
> >	3. Higher code quality expectation (these are pull requests that
> >	maintainers expect Linus to pull)
> 
> Are people putting things in linux-next that they don't expect to send to Linus? That seems like the greater problem.
> 
> >	4. Continuous tree (not daily tags like in linux-next),
> >	facilitating easier bisection
> 
> I'm not sure how useful that is given the very small time window to find bugs.

I think there is some value for this tree, but the target is not what
Sasha explained (or what I understood at least). The linus-next (or
whatever linus-pr) is seems to me to be solely targetting Linus, for
running pre-merge checks.

And that is *very* interesting for him, so yes, I'd vote for a plus one
on this.

IMO this new tree shouldn't be advertised as a:
"please run as many possible tests on this before Linus pulls it",
but as a:
"we are gathering all PR, run a dedicated sets of tests on it, and if
they passes, Linus will look at your PR. If not, your PR will be
automatically put on hold".

Think of it as a pre-merge testing, and if a PR fails, the maintainer
has to sort it out instead of having Linus to sort it out.

> 
> >The linus-next tree aims to provide a more stable and testable
> >integration point compared to linux-next,
> 
> Why not just use linux-next? I don't understand how this is any different except that it provides very little time to do testing and will need manual conflict resolutions that have already been done in linux-next.

Agree. Gathering general tests should not come *after* the PR has been
sent, but before.

It could be interesting IMO to have a linux-current-fixes tree which
gather trees that are meant to be sent to this current cycle (fixes only
basically).

In my small subsystem we sometime gather fixes that are not urgent but
could go into the current cycle but are not send them right away. They
could be integrated in such a tree, but OTOH, they are tested in
linux-next already, so it's more of a matter for testers to decide if
they want to have a "current tree + upcoming fixes" or not.

> 
> How about this, instead: no one sends -rc1 PRs to Linus that didn't go through -next. Just have a bot that replies to all PRs with a health check, and Linus can pull it if he thinks it looks good. 
> 
> For example, for a given PR, the bot can report:
> 
> - Were the patches CCed to a mailing list?
> - A histogram of how long the patches were in next (to show bake times)
> - Are any patches associated with test failures? (0day and many other CIs are already running tests against -next; parse those reports)
> 
> We could have a real pre-submit checker! :)

As long as it only helps making an educated guess (or catch obvious
failures) I'm all for it.
In the same way stable receives emails from testers before Greg tags the
new tree.

Cheers,
Benjamin

Re: linus-next: improving functional testing for to-be-merged pull requests

[Thorsten Leemhuis]

On 21.10.24 23:41, Mark Brown wrote:
> On Mon, Oct 21, 2024 at 12:07:13PM -0400, Sasha Levin wrote:
>
>> In an attempt to address the concerns, we're trying out a new "linus-next"
>> tree is being created and maintained with the following characteristics:

BTW, in case anyone cares: I fully agree with what Kees wrote earlier
today elsewhere in this thread, e.g. things like "improve -next instead"
and "pre-merge bot":
http://lore.kernel.org/all/792F4759-EA33-48B8-9AD0-FA14FA69E86E%40kernel.org

Regarding that bot: a few of the CI folks and a developer or two told me
they want regzbot to react to PRs for Linus as well, so it can send
mails like "hey Linus, just so you know, this PR contains changes that
cause the following regressions not yet fixed". I think I like the idea,
but well, quite a few other improvements around regzbot and its use have
a much higher priority currently.

>> 	4. Continuous tree (not daily tags like in linux-next),
>> 	facilitating easier bisection
>
> Is this a pressing problem?  I routinely bisect -next, you have to base
> things on Linus' tree (or pending-fixes) but otherwise it's not
> especially problematic.

I wonder if part of this is a "don't know how to do that" aka "lack of
documentation" problem. I've recently seen some good guide or mailing
list post how to bisect -next somewhere, but I think it wasn't in our
Documentation/ directory. I need to search where that was (Mark, I might
misremember, but wasn't it you who posted it somewhere?) and could work
towards upstreaming that or some other guide. And don't worry, due to
the different target audience it would be much shorter text than other
documents I contributed. ;-)

Ciao, Thorsten

Re: linus-next: improving functional testing for to-be-merged pull requests

[Matthieu Baerts]

Hi Sasha,

Thank you for your replies!

On 21/10/2024 19:36, Sasha Levin wrote:
> On Mon, Oct 21, 2024 at 07:18:38PM +0200, Matthieu Baerts wrote:
>> On 21/10/2024 18:07, Sasha Levin wrote:

(...)

>>>     4. Continuous tree (not daily tags like in linux-next),
>>>     facilitating easier bisection
>>
>> What will happen when a pull request is rejected?
> 
> My mental playbook is:
> 
> 1. If a pull request is just ignored, ping it in case it was forgotten.
> 2. If we have an explicit NACK, just revert the merge commit.

Hopefully these reverts will be exceptional, because they can quickly be
hard to manage!

>> (...)
>>
>>> We also want to avoid altering the existing workflow. In particular:
>>>
>>>     1. No increase in latency. If anything, the expectation is that
>>>     the cadence of merges would be improved given that Linus will
>>>     need to do less builds and tests.
>>>
>>>     2. Require "sign up" for the tree like linux-next does. Instead,
>>>     pull requests are monitored and grabbed directly from the
>>>     mailing list.
>>
>> Out of curiosity: is it done automatically? Will it email someone when a
>> conflict is found?
> 
> So it's 80% automatic now: my scripts monitor emails using lei, parse
> relevant ones and manage to extract the pull instructions out of them,
> and then most of those pull requests just merge cleanly.
> 
> There are some with conflicts, but since Linus insists on having an
> explanation for merge conflicts, those pull requsts contain those
> instructions within them. In those cases I manually followed the
> instructions to resolve the conflicts (which were trivial so far).
> 
> I'll likely send a mail out *only* if I see a non-trivial merge conflict
> without an explanation in the body.

OK, thank you!

>> (...)
>>
>>> Current testing:
>>>   - LKFT: https://qa-reports.linaro.org/lkft/sashal-linus-next/
>>>   - KernelCI: https://t.ly/KEW7F
>>
>> That's great to have more tests being executed! Who is going to monitor
>> the results? This task can quickly take time if this person also has to
>> check for false positives and flaky tests.
>>
>> Are the maintainers supposed to regularly monitor the results for the
>> tests they are responsible for? Or will they be (automatically?) emailed
>> when there is a regression?
> 
> I'm not sure about this part. While I look at it in and will likely send
> a mail out if I see something fishy, the only change in workflow that I
> hope will happen here is Linus looking at a dashboard or two before he
> begins his daily merge session.

OK, thank you! I find these dashboards not so easy to read: there are
many tests, and it is not always clear what they are doing or how
important they are. Yes it is possible to find the history and check if
a test is known as being unstable, but there are no indicators to show
that directly, nor a global one saying "OK to pull".

What I want to say is that I hope these dashboards will help, and not
just to say "look, we are running tests", but nobody is actually looking
at the results :)

Cheers,
Matt
-- 
Sponsored by the NGI0 Core fund.

Re: linus-next: improving functional testing for to-be-merged pull requests

[Sasha Levin]

On Mon, Oct 21, 2024 at 11:48:34PM -0700, Christoph Hellwig wrote:
>On Mon, Oct 21, 2024 at 09:54:53PM -0700, Kees Cook wrote:
>> >	1. Composed of pull requests sent directly to Linus
>> >
>> >	2. Contains branches destined for imminent inclusion by Linus
>>
>> But this means hours or a day or 2 at most.
>
>Yeah.

During the -rc cycles, sure.

However, folks have been consistently sending content for the next
release early - usually a week or two before the merge window even
opens.

For that matter, we've already seen pull requests destined for 6.13
getting pulled into linus-next.

>>
>> >	3. Higher code quality expectation (these are pull requests that
>> >	maintainers expect Linus to pull)
>>
>> Are people putting things in linux-next that they don't expect to send to Linus? That seems like the greater problem.
>
>They shouldn't.  If they do we do indeed have a problem.

Not in the sense that it's not expected to be sent to Linus, but more in
the sense that folks are shoving things in -next before they passed all
the "local" tests a maintainer can run.

We end up with content that is destined to Linus, but is immature.

>> >	4. Continuous tree (not daily tags like in linux-next),
>> >	facilitating easier bisection
>>
>> I'm not sure how useful that is given the very small time window to find bugs.
>
>Same.
>
>> >The linus-next tree aims to provide a more stable and testable
>> >integration point compared to linux-next,
>>
>> Why not just use linux-next? I don't understand how this is any
>> different except that it provides very little time to do testing and
>> will need manual conflict resolutions that have already been done in
>> linux-next.
>
>Exactly!

We had multiple issues just this release cycle that would have been
caught by this tree and not by linux-next.

>> How about this, instead: no one sends -rc1 PRs to Linus that didn't go
>> through -next. Just have a bot that replies to all PRs with a health
>> check, and Linus can pull it if he thinks it looks good.
>
>Not just -rc1, otherwise agreed.
>
>> For example, for a given PR, the bot can report:
>>
>> - Were the patches CCed to a mailing list?
>> - A histogram of how long the patches were in next (to show bake times)
>> - Are any patches associated with test failures? (0day and many other
>> CIs are already running tests against -next; parse those reports)
>>
>> We could have a real pre-submit checker! :)
>
>That would be very useful.  Items 1 and 2 should be trivial, 3 would
>require a bit of work but would still be very useful.

We could add a report for the above, but:

1. Linus consistently pulls patches that haven't seen the light of day.
2. Linus explicitly objected to making a linux-next a must have.

So unless these results would be actually used, what's the point in
writing all of that?

-- 
Thanks,
Sasha

Re: linus-next: improving functional testing for to-be-merged pull requests

[Vlastimil Babka]

On 10/22/24 10:12, Steven Rostedt wrote:
> On Mon, 21 Oct 2024 23:48:34 -0700
> Christoph Hellwig <hch@infradead.org> wrote:
> 
>> > How about this, instead: no one sends -rc1 PRs to Linus that didn't go
>> > through -next. Just have a bot that replies to all PRs with a health
>> > check, and Linus can pull it if he thinks it looks good.   
>> 
>> Not just -rc1, otherwise agreed.
> 
> You mean have everything go into linux-next before going to Linus after -rc1?
> 
> I'm one that doesn't do this. That's because my code in linux-next
> after -rc1 is for the next merge window, and the code I send to Linus
> is only fixes for code I sent before -rc1. I tend to keep an "urgent"
> and "core" branch. My "core" branch is everything I plan to send in the
> next merge window and goes into linux-next (via being pulled into my
> for-next branch). After I send my pull request to Linus, and he pulls
> it in the merge window, that "core" branch becomes my "urgent" branch.
> 
> But when I find a bug that's in Linus's tree, I put the fix on top of
> "urgent", run it through my test suite (takes 8 hours or so), then send
> a pull request to Linus. My "urgent" branch doesn't go into linux-next
> as it doesn't have changes that should affect others work, which is
> what I think linux-next is mostly for. I also find known bugs in
> Linus's tree to be high priority to be fixed (I stop what I'm doing to
> get the fix out ASAP).
> 
> Now, if there was better testing from linux-next, maybe it would be
> worth the time to push my urgent branch there for a bit. But so far I
> haven't seen the benefit of doing that.

I've seen somewhere mentioned (but can't now find where it was) that -next
has a pending-fixes branch [1] that's exactly intended to collect hotfix
branches from maintainer trees separately from the true "for next merge
window" branches. And that one of the points from the maintainers summit was
that this would be made more widely known and expected of maintainers to
use. Can anyone confirm?

[1]
https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/log/?h=pending-fixes

>> 
>> > For example, for a given PR, the bot can report:
>> > 
>> > - Were the patches CCed to a mailing list?
>> > - A histogram of how long the patches were in next (to show bake times)
>> > - Are any patches associated with test failures? (0day and many other
>> > CIs are already running tests against -next; parse those reports)
>> > 
>> > We could have a real pre-submit checker! :)  
>> 
>> That would be very useful.  Items 1 and 2 should be trivial, 3 would
>> require a bit of work but would still be very useful.
> 
> If there was more feedback to going into linux-next, that would be good.
> 
> -- Steve
> 

Re: linus-next: improving functional testing for to-be-merged pull requests

[Sasha Levin]

On Mon, Oct 21, 2024 at 11:48:34PM -0700, Christoph Hellwig wrote:
>On Mon, Oct 21, 2024 at 09:54:53PM -0700, Kees Cook wrote:
>> For example, for a given PR, the bot can report:
>>
>> - Were the patches CCed to a mailing list?
>> - A histogram of how long the patches were in next (to show bake times)
>> - Are any patches associated with test failures? (0day and many other
>> CIs are already running tests against -next; parse those reports)
>>
>> We could have a real pre-submit checker! :)
>
>That would be very useful.  Items 1 and 2 should be trivial, 3 would
>require a bit of work but would still be very useful.

So I've ended up writing something hacky that does (1) and (2) above, and
ran it on the existing (already merged) PRs on the mailing list to give
it a spin.

The script tripped on the very first PR it looked at:
https://lore.kernel.org/all/20241021171728.274997-1-pbonzini@redhat.com/

And in particular, this commit: afa9b48f327c ("KVM: arm64: Shave a few
bytes from the EL2 idmap code")

(sorry, not trying to pick on anyone/anything, just an example...)

The commit can't be found on lore.kernel.org, it was never in -next, and
yet Linus pulled it promptly without questioning anything.

So yes, we could argue that a bot sending out this information would
have informed Linus of this issue, but somehow I suspect that Linus is
not an idiot and he has enough scripts in place already to alert him of
these issues, which he chooses to ignore in some cases.

-- 
Thanks,
Sasha

Re: linus-next: improving functional testing for to-be-merged pull requests

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 734 bytes --]

On Tue, Oct 22, 2024 at 06:52:51AM -0400, Sasha Levin wrote:
> On Mon, Oct 21, 2024 at 11:48:34PM -0700, Christoph Hellwig wrote:

> The script tripped on the very first PR it looked at:
> https://lore.kernel.org/all/20241021171728.274997-1-pbonzini@redhat.com/

> And in particular, this commit: afa9b48f327c ("KVM: arm64: Shave a few
> bytes from the EL2 idmap code")

> (sorry, not trying to pick on anyone/anything, just an example...)

> The commit can't be found on lore.kernel.org, it was never in -next, and
> yet Linus pulled it promptly without questioning anything.

That was on the list at least, but buried in the replies to a thread
rather than posted separately:

  https://lore.kernel.org/86msjc56mi.wl-maz@kernel.org

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: linus-next: improving functional testing for to-be-merged pull requests

[James Bottomley]

On Tue, 2024-10-22 at 04:12 -0400, Steven Rostedt wrote:
> On Mon, 21 Oct 2024 23:48:34 -0700
> Christoph Hellwig <hch@infradead.org> wrote:
> 
> > > How about this, instead: no one sends -rc1 PRs to Linus that
> > > didn't go through -next. Just have a bot that replies to all PRs
> > > with a health check, and Linus can pull it if he thinks it looks
> > > good.   
> > 
> > Not just -rc1, otherwise agreed.
> 
> You mean have everything go into linux-next before going to Linus
> after -rc1?

I think that's a good goal, yes.  Most developers tend to send a fixes
pull around once a week (or less) after the merge window, which means
it could be soaked in -next.

> I'm one that doesn't do this. That's because my code in linux-next
> after -rc1 is for the next merge window, and the code I send to Linus
> is only fixes for code I sent before -rc1. I tend to keep an "urgent"
> and "core" branch. My "core" branch is everything I plan to send in
> the next merge window and goes into linux-next (via being pulled into
> my for-next branch). After I send my pull request to Linus, and he
> pulls it in the merge window, that "core" branch becomes my "urgent"
> branch.
> 
> But when I find a bug that's in Linus's tree, I put the fix on top of
> "urgent", run it through my test suite (takes 8 hours or so), then
> send a pull request to Linus. My "urgent" branch doesn't go into
> linux-next as it doesn't have changes that should affect others work,
> which is what I think linux-next is mostly for.

That's not necessarily a safe assumption.  The reason we put scsi-fixes
into -next is precisely to pick up if this happens.  I admit it happens
very rarely because of the compact and local nature of the fixes, but
the signal rate isn't zero.

>  I also find known bugs in Linus's tree to be high priority to be
> fixed (I stop what I'm doing to get the fix out ASAP).
> 
> Now, if there was better testing from linux-next, maybe it would be
> worth the time to push my urgent branch there for a bit. But so far I
> haven't seen the benefit of doing that.

It does at least get 0day running over it, although that can also be
configured for your branches separately as well.  It's also not unwise
to have some pause time before fixing and sending ... the fix you first
thought of isn't always the best one (or even sometimes an actual fix).

Regards,

James

Re: linus-next: improving functional testing for to-be-merged pull requests

[Jiri Kosina]

On Mon, 21 Oct 2024, Paul E. McKenney wrote:

> I have to ask...
> 
> Wouldn't more people testing -next result in more pressure to fix
> linux-next problems quickly?

I believe I brought up pretty much exactly this at this year's maintainer 
summit.

From the discussion it turned out the many people believe that this 
investing into this is probably not worth it, as it will require much more 
continous, never-ending effort (for which there are probably not enough 
resources) than just dealing with the fallout once during the -rc1+ phase.

-- 
Jiri Kosina
SUSE Labs

Re: linus-next: improving functional testing for to-be-merged pull requests

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 2122 bytes --]

On Tue, Oct 22, 2024 at 07:51:03AM -0400, James Bottomley wrote:
> On Tue, 2024-10-22 at 04:12 -0400, Steven Rostedt wrote:
> > Christoph Hellwig <hch@infradead.org> wrote:

> > But when I find a bug that's in Linus's tree, I put the fix on top of
> > "urgent", run it through my test suite (takes 8 hours or so), then
> > send a pull request to Linus. My "urgent" branch doesn't go into
> > linux-next as it doesn't have changes that should affect others work,
> > which is what I think linux-next is mostly for.

> That's not necessarily a safe assumption.  The reason we put scsi-fixes
> into -next is precisely to pick up if this happens.  I admit it happens
> very rarely because of the compact and local nature of the fixes, but
> the signal rate isn't zero.

Yeah, just yesterday there was a cross architecture build break
introduced by a fix in the selftests.

The whole reason pending-fixes exists is that we had a disruptive
problem with fixes that hadn't been given enough coverage at some point,
IIRC in the case that immediately inspired it was that there was a
dependency on changes that were only in -next that hadn't been noticed
which is why we've now got the fixes tested independently of any new
development.  

> >  I also find known bugs in Linus's tree to be high priority to be
> > fixed (I stop what I'm doing to get the fix out ASAP).

> > Now, if there was better testing from linux-next, maybe it would be
> > worth the time to push my urgent branch there for a bit. But so far I
> > haven't seen the benefit of doing that.

> It does at least get 0day running over it, although that can also be
> configured for your branches separately as well.  It's also not unwise
> to have some pause time before fixing and sending ... the fix you first
> thought of isn't always the best one (or even sometimes an actual fix).

I'm also covering it in my CI and do occasionally find issues there, as
does KernelCI.  There's probably others I'm less aware of.  TBH half the
reason I cover it is that it also cuts down the number of bisect steps
for -next a little.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: linus-next: improving functional testing for to-be-merged pull requests

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 921 bytes --]

On Tue, Oct 22, 2024 at 11:10:36AM +0200, Thorsten Leemhuis wrote:

> I wonder if part of this is a "don't know how to do that" aka "lack of
> documentation" problem. I've recently seen some good guide or mailing
> list post how to bisect -next somewhere, but I think it wasn't in our
> Documentation/ directory. I need to search where that was (Mark, I might
> misremember, but wasn't it you who posted it somewhere?) and could work
> towards upstreaming that or some other guide. And don't worry, due to
> the different target audience it would be much shorter text than other
> documents I contributed. ;-)

I don't recall anything specific, though it's plausible I said something
in a thread that was basically just the bit about bisecting between
mainline<->pending-fixes<->-next rather than between -next versions.
You're right that we should have that in the documentation somewhere,
I'll look at sending a patch.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: linus-next: improving functional testing for to-be-merged pull requests

[Paul E. McKenney]

On Tue, Oct 22, 2024 at 02:06:46PM +0200, Jiri Kosina wrote:
> On Mon, 21 Oct 2024, Paul E. McKenney wrote:
> 
> > I have to ask...
> > 
> > Wouldn't more people testing -next result in more pressure to fix
> > linux-next problems quickly?
> 
> I believe I brought up pretty much exactly this at this year's maintainer 
> summit.
> 
> >From the discussion it turned out the many people believe that this 
> investing into this is probably not worth it, as it will require much more 
> continous, never-ending effort (for which there are probably not enough 
> resources) than just dealing with the fallout once during the -rc1+ phase.

Thank you for the response and the information!

But why won't this same issues apply just as forcefully to a new
linus-next tree?

Full disclosure:  Testing and tracking down bugs in -next can be a bit of
a hassle, to be sure, but I expect to continue to do so.  For one thing,
dealing with -next is way easier than testing patches on the various
mailing lists.

							Thanx, Paul

Re: linus-next: improving functional testing for to-be-merged pull requests

[Sasha Levin]

On Tue, Oct 22, 2024 at 07:22:12AM -0700, Paul E. McKenney wrote:
>On Tue, Oct 22, 2024 at 02:06:46PM +0200, Jiri Kosina wrote:
>> On Mon, 21 Oct 2024, Paul E. McKenney wrote:
>>
>> > I have to ask...
>> >
>> > Wouldn't more people testing -next result in more pressure to fix
>> > linux-next problems quickly?
>>
>> I believe I brought up pretty much exactly this at this year's maintainer
>> summit.
>>
>> >From the discussion it turned out the many people believe that this
>> investing into this is probably not worth it, as it will require much more
>> continous, never-ending effort (for which there are probably not enough
>> resources) than just dealing with the fallout once during the -rc1+ phase.
>
>Thank you for the response and the information!
>
>But why won't this same issues apply just as forcefully to a new
>linus-next tree?
>
>Full disclosure:  Testing and tracking down bugs in -next can be a bit of
>a hassle, to be sure, but I expect to continue to do so.  For one thing,
>dealing with -next is way easier than testing patches on the various
>mailing lists.

I'm not trying to change the workflow here, I think all this amounts to
is just a quality of life improvement for Linus who could ideally merge
faster if he knows that the pull requests he's looking at will build and
won't brick his laptop.

If we start playing games around "must spend X days in linus-next", then
yes - I agree with you.

-- 
Thanks,
Sasha

Re: linus-next: improving functional testing for to-be-merged pull requests

[Paul E. McKenney]

On Tue, Oct 22, 2024 at 10:36:13AM -0400, Sasha Levin wrote:
> On Tue, Oct 22, 2024 at 07:22:12AM -0700, Paul E. McKenney wrote:
> > On Tue, Oct 22, 2024 at 02:06:46PM +0200, Jiri Kosina wrote:
> > > On Mon, 21 Oct 2024, Paul E. McKenney wrote:
> > > 
> > > > I have to ask...
> > > >
> > > > Wouldn't more people testing -next result in more pressure to fix
> > > > linux-next problems quickly?
> > > 
> > > I believe I brought up pretty much exactly this at this year's maintainer
> > > summit.
> > > 
> > > >From the discussion it turned out the many people believe that this
> > > investing into this is probably not worth it, as it will require much more
> > > continous, never-ending effort (for which there are probably not enough
> > > resources) than just dealing with the fallout once during the -rc1+ phase.
> > 
> > Thank you for the response and the information!
> > 
> > But why won't this same issues apply just as forcefully to a new
> > linus-next tree?
> > 
> > Full disclosure:  Testing and tracking down bugs in -next can be a bit of
> > a hassle, to be sure, but I expect to continue to do so.  For one thing,
> > dealing with -next is way easier than testing patches on the various
> > mailing lists.
> 
> I'm not trying to change the workflow here, I think all this amounts to
> is just a quality of life improvement for Linus who could ideally merge
> faster if he knows that the pull requests he's looking at will build and
> won't brick his laptop.

I don't understand how creating yet another tree will have that result,
but you have more direct experience with this aspect of the process than
I do.  I do hope that it works out.

> If we start playing games around "must spend X days in linus-next", then
> yes - I agree with you.

I did see the later email indicating that Linus was dead-set against
such a requirement, so there is that.  On the other hand, the subsequent
discussion of publicly documenting which commits had less testing seems
like it might work.  For but one example, I would be suspicious of
commits coming from someone arguing that appearing frequently on that
list is a non-problem.  ;-)

(Yes, yes, there are always exceptions.)

							Thanx, Paul

Re: linus-next: improving functional testing for to-be-merged pull requests

[Sasha Levin]

On Tue, Oct 22, 2024 at 12:50:50PM +0100, Mark Brown wrote:
>On Tue, Oct 22, 2024 at 06:52:51AM -0400, Sasha Levin wrote:
>> On Mon, Oct 21, 2024 at 11:48:34PM -0700, Christoph Hellwig wrote:
>
>> The script tripped on the very first PR it looked at:
>> https://lore.kernel.org/all/20241021171728.274997-1-pbonzini@redhat.com/
>
>> And in particular, this commit: afa9b48f327c ("KVM: arm64: Shave a few
>> bytes from the EL2 idmap code")
>
>> (sorry, not trying to pick on anyone/anything, just an example...)
>
>> The commit can't be found on lore.kernel.org, it was never in -next, and
>> yet Linus pulled it promptly without questioning anything.
>
>That was on the list at least, but buried in the replies to a thread
>rather than posted separately:
>
>  https://lore.kernel.org/86msjc56mi.wl-maz@kernel.org

If folks are interested in collaborating around -next analysis and maybe
adding a bot/dashboard/etc, I've pushed my scripts to
https://git.kernel.org/pub/scm/linux/kernel/git/sashal/next-analysis.git/

Right now they're enough to generate a histogram showing how many
patches spent N days in -next:

$ git log --format="%H" --no-merges origin/master..next/master | shuf | head -n500 | xargs histo.sh ~/next-analysis/db/
Days in -next:
----------------------------------------
  0  |
  1  | ████████████████████ (40)
  2  | ███████████████████████████████ (62)
  3  | █████████████ (27)
  4  | █████████████ (27)
  5  | ███████████████████ (39)
  6  | ██████████████ (28)
  7  | ████████ (16)
  8  | ██████████████████ (37)
  9  | ██████ (12)
10  | █████████████████████ (42)
11  | █████████████████████████ (50)
12  | ██████ (12)
13  | █████ (10)
14+ | █████████████████████████████████████████████████ (98)

As well as find commits that did not appear on lore:

$ git log --format="%H" --no-merges origin/master..next/master | shuf | head -n100 | xargs no-lore.sh ~/next-analysis/db/
c88414f56c37f XArray: Prevent node leaks in xas_alloc()

-- 
Thanks,
Sasha

Re: linus-next: improving functional testing for to-be-merged pull requests

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 342 bytes --]

On Tue, Oct 22, 2024 at 10:47:01AM -0400, Sasha Levin wrote:

> If folks are interested in collaborating around -next analysis and maybe
> adding a bot/dashboard/etc, I've pushed my scripts to
> https://git.kernel.org/pub/scm/linux/kernel/git/sashal/next-analysis.git/

Those look interesting, it'd be good to get some licensing on the repo.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: linus-next: improving functional testing for to-be-merged pull requests

[Kees Cook]

On Tue, Oct 22, 2024 at 04:12:43AM -0400, Steven Rostedt wrote:
> On Mon, 21 Oct 2024 23:48:34 -0700
> Christoph Hellwig <hch@infradead.org> wrote:
> 
> > > How about this, instead: no one sends -rc1 PRs to Linus that didn't go
> > > through -next. Just have a bot that replies to all PRs with a health
> > > check, and Linus can pull it if he thinks it looks good.   
> > 
> > Not just -rc1, otherwise agreed.
> 
> You mean have everything go into linux-next before going to Linus after -rc1?

It seems like the most useful signal would be for the initial PR for the
merge window. After that it becomes a lot of fixes that didn't get tons
of soak time in -next, etc.

-- 
Kees Cook

Re: linus-next: improving functional testing for to-be-merged pull requests

[Guenter Roeck]

On 10/22/24 12:33, Kees Cook wrote:
> On Tue, Oct 22, 2024 at 04:12:43AM -0400, Steven Rostedt wrote:
>> On Mon, 21 Oct 2024 23:48:34 -0700
>> Christoph Hellwig <hch@infradead.org> wrote:
>>
>>>> How about this, instead: no one sends -rc1 PRs to Linus that didn't go
>>>> through -next. Just have a bot that replies to all PRs with a health
>>>> check, and Linus can pull it if he thinks it looks good.
>>>
>>> Not just -rc1, otherwise agreed.
>>
>> You mean have everything go into linux-next before going to Linus after -rc1?
> 
> It seems like the most useful signal would be for the initial PR for the
> merge window. After that it becomes a lot of fixes that didn't get tons
> of soak time in -next, etc.
> 

FWIW, there is also the pending-fixes sub-branch of linux-next, but from my
experience the build and test failures observed in it are pretty much
ignored, just like the build and test failures in linux-next itself.

Guenter

Re: linus-next: improving functional testing for to-be-merged pull requests

[Christoph Hellwig]

On Tue, Oct 22, 2024 at 04:12:43AM -0400, Steven Rostedt wrote:
> You mean have everything go into linux-next before going to Linus after -rc1?
> 
> I'm one that doesn't do this. That's because my code in linux-next
> after -rc1 is for the next merge window, and the code I send to Linus
> is only fixes for code I sent before -rc1. I tend to keep an "urgent"
> and "core" branch. My "core" branch is everything I plan to send in the
> next merge window and goes into linux-next (via being pulled into my
> for-next branch). After I send my pull request to Linus, and he pulls
> it in the merge window, that "core" branch becomes my "urgent" branch.

You can easily have two branches in linux-next.  Many trees do that.
It is also a really nice warning about self-conflicts.

Re: linus-next: improving functional testing for to-be-merged pull requests

[Christoph Hellwig]

On Tue, Oct 22, 2024 at 05:37:38AM -0400, Sasha Levin wrote:
> We could add a report for the above, but:
> 
> 1. Linus consistently pulls patches that haven't seen the light of day.
> 2. Linus explicitly objected to making a linux-next a must have.
> 
> So unless these results would be actually used, what's the point in
> writing all of that?

Yes, without Linus caring we're not going to get our process worked out.
Not sure how a tree that probably won't have much better latency than
linux-next is going to fix that, though.

Re: linus-next: improving functional testing for to-be-merged pull requests

[Steven Rostedt]

On Tue, 22 Oct 2024 22:47:09 -0700
Christoph Hellwig <hch@infradead.org> wrote:

> On Tue, Oct 22, 2024 at 04:12:43AM -0400, Steven Rostedt wrote:
> > You mean have everything go into linux-next before going to Linus after -rc1?
> > 
> > I'm one that doesn't do this. That's because my code in linux-next
> > after -rc1 is for the next merge window, and the code I send to Linus
> > is only fixes for code I sent before -rc1. I tend to keep an "urgent"
> > and "core" branch. My "core" branch is everything I plan to send in the
> > next merge window and goes into linux-next (via being pulled into my
> > for-next branch). After I send my pull request to Linus, and he pulls
> > it in the merge window, that "core" branch becomes my "urgent" branch.  
> 
> You can easily have two branches in linux-next.  Many trees do that.
> It is also a really nice warning about self-conflicts.

I actually do have several branches in linux-next. But they are all
topic branches. My urgent branches usually mirror them (by naming
convention). My scripts pull my for-next branches together and then I
push them up.

I did push urgent branches to linux-next some time back, but never
found any advantage in doing so, so I stopped doing it. As the code in
my urgent branches are just fixing the stuff already in Linus's tree,
they seldom ever have any effect on other subsystems. My new work does
benefit from being in linux-next. But since I don't find more testing
in linux-next for things that are already in Linus's tree, I still
don't see how its worth the time to put my urgent work there.

To put it this way. The bugs I'm fixing was for code in linux-next
where the bugs were never found. They only appeared when they went into
Linus's tree. So why put the fixes in linux-next, if it didn't catch
the bugs I fixed in the first place?

-- Steve

Re: linus-next: improving functional testing for to-be-merged pull requests

[Geert Uytterhoeven]

Hi Steven,

On Wed, Oct 23, 2024 at 10:20 AM Steven Rostedt <rostedt@goodmis.org> wrote:
> On Tue, 22 Oct 2024 22:47:09 -0700
> Christoph Hellwig <hch@infradead.org> wrote:
> > On Tue, Oct 22, 2024 at 04:12:43AM -0400, Steven Rostedt wrote:
> > > You mean have everything go into linux-next before going to Linus after -rc1?
> > >
> > > I'm one that doesn't do this. That's because my code in linux-next
> > > after -rc1 is for the next merge window, and the code I send to Linus
> > > is only fixes for code I sent before -rc1. I tend to keep an "urgent"
> > > and "core" branch. My "core" branch is everything I plan to send in the
> > > next merge window and goes into linux-next (via being pulled into my
> > > for-next branch). After I send my pull request to Linus, and he pulls
> > > it in the merge window, that "core" branch becomes my "urgent" branch.
> >
> > You can easily have two branches in linux-next.  Many trees do that.
> > It is also a really nice warning about self-conflicts.
>
> I actually do have several branches in linux-next. But they are all
> topic branches. My urgent branches usually mirror them (by naming
> convention). My scripts pull my for-next branches together and then I
> push them up.
>
> I did push urgent branches to linux-next some time back, but never
> found any advantage in doing so, so I stopped doing it. As the code in
> my urgent branches are just fixing the stuff already in Linus's tree,
> they seldom ever have any effect on other subsystems. My new work does
> benefit from being in linux-next. But since I don't find more testing
> in linux-next for things that are already in Linus's tree, I still
> don't see how its worth the time to put my urgent work there.
>
> To put it this way. The bugs I'm fixing was for code in linux-next
> where the bugs were never found. They only appeared when they went into
> Linus's tree. So why put the fixes in linux-next, if it didn't catch
> the bugs I fixed in the first place?

Hmmm...

Your arguments sound very similar to those being used in recent
discussions about not posting patches for public review...

Please follow the process! ;-)

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: linus-next: improving functional testing for to-be-merged pull requests

[Steven Rostedt]

On Wed, 23 Oct 2024 10:36:20 +0200
Geert Uytterhoeven <geert@linux-m68k.org> wrote:

> > To put it this way. The bugs I'm fixing was for code in linux-next
> > where the bugs were never found. They only appeared when they went into
> > Linus's tree. So why put the fixes in linux-next, if it didn't catch
> > the bugs I fixed in the first place?  
> 
> Hmmm...
> 
> Your arguments sound very similar to those being used in recent
> discussions about not posting patches for public review...
> 
> Please follow the process! ;-)

What process?

Note, I probably post everything to mailing lists more than anyone
else (besides stable). All my commits come from mailing lists. Even
things I change myself. I always send out the change to a list. Then I
use patchwork to pull it into my tree.

After the changes are tested, I send out the patches *again* with my
[for-next] tags in the subject. If it's a fix for Linus, it goes out as
a "[for-linus]" tag. These emails automatically update my patchwork
status.

No change goes into Linus's tree from me that hasn't been sent out
publicly.

But pushing to linux-next for a day or two, what does that give me?

-- Steve

Re: linus-next: improving functional testing for to-be-merged pull requests

[Geert Uytterhoeven]

Hi Steven,

On Wed, Oct 23, 2024 at 11:19 AM Steven Rostedt <rostedt@goodmis.org> wrote:
> On Wed, 23 Oct 2024 10:36:20 +0200
> Geert Uytterhoeven <geert@linux-m68k.org> wrote:
>
> > > To put it this way. The bugs I'm fixing was for code in linux-next
> > > where the bugs were never found. They only appeared when they went into
> > > Linus's tree. So why put the fixes in linux-next, if it didn't catch
> > > the bugs I fixed in the first place?
> >
> > Hmmm...
> >
> > Your arguments sound very similar to those being used in recent
> > discussions about not posting patches for public review...
> >
> > Please follow the process! ;-)
>
> What process?
>
> Note, I probably post everything to mailing lists more than anyone
> else (besides stable). All my commits come from mailing lists. Even
> things I change myself. I always send out the change to a list. Then I
> use patchwork to pull it into my tree.
>
> After the changes are tested, I send out the patches *again* with my
> [for-next] tags in the subject. If it's a fix for Linus, it goes out as
> a "[for-linus]" tag. These emails automatically update my patchwork
> status.
>
> No change goes into Linus's tree from me that hasn't been sent out
> publicly.

All good!

> But pushing to linux-next for a day or two, what does that give me?

It may catch issues you missed, e.g. compile failures on obscure
architectures or configs, e.g. due to indirect includes.

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: linus-next: improving functional testing for to-be-merged pull requests

[Vlastimil Babka]

On 10/23/24 10:20, Steven Rostedt wrote:
> To put it this way. The bugs I'm fixing was for code in linux-next
> where the bugs were never found. They only appeared when they went into
> Linus's tree. So why put the fixes in linux-next, if it didn't catch
> the bugs I fixed in the first place?

The fix might be in a different part of the code, one that's stressed by
-next testing even if the code with the original bug wasn't. So I don't
think you can always assume that -next not catching the original bug means
it can't catch a bug in the fix?

Re: linus-next: improving functional testing for to-be-merged pull requests

[Dan Carpenter]

On Wed, Oct 23, 2024 at 05:19:14AM -0400, Steven Rostedt wrote:
> But pushing to linux-next for a day or two, what does that give me?
> 

Two days probably gives you 80% of the testing that people do on linux-next.
I wouldn't run linux-next on real systems but it generally boots and runs LTP
okay these days.

regards,
dan carpenter

Re: linus-next: improving functional testing for to-be-merged pull requests

[Thorsten Leemhuis]

On 23.10.24 11:32, Vlastimil Babka wrote:
> On 10/23/24 10:20, Steven Rostedt wrote:
>> To put it this way. The bugs I'm fixing was for code in linux-next
>> where the bugs were never found. They only appeared when they went into
>> Linus's tree. So why put the fixes in linux-next, if it didn't catch
>> the bugs I fixed in the first place?
> 
> The fix might be in a different part of the code, one that's stressed by
> -next testing even if the code with the original bug wasn't. So I don't
> think you can always assume that -next not catching the original bug means
> it can't catch a bug in the fix?

+1 to this and the "compile failures on obscure architectures or
configs" from Geert.

A -fixes branch in -next also makes a few things easier for regression
tracking:

* I only have to check *one* place instead of one or two hundred to see
if a regression fix is heading towards mainline or stuck somewhere.

* I can see if people accidentally queued regression fix for the current
or the next cycle when it should be the former (some subsystems make
this hard or impossible).

Ciao, Thorsten

Re: linus-next: improving functional testing for to-be-merged pull requests

[James Bottomley]

On Wed, 2024-10-23 at 04:20 -0400, Steven Rostedt wrote:
[...]
> I did push urgent branches to linux-next some time back, but never
> found any advantage in doing so, so I stopped doing it. As the code
> in my urgent branches are just fixing the stuff already in Linus's
> tree, they seldom ever have any effect on other subsystems. My new
> work does benefit from being in linux-next. But since I don't find
> more testing in linux-next for things that are already in Linus's
> tree, I still don't see how its worth the time to put my urgent work
> there.

What do you mean "push" to linux-next?  You just give Stephen a list of
branches and he pulls.  I do have a single for-next tag that he pulls
so I merge both fixes and collecting merge window stuff into that, but
I do this so I can see immediately if we have an internal conflict,
which is the most common problem.

If you don't want any work at all, just name your fixes branch and tell
Stephen and then he'll tell you if there's a problem.  In this model
you don't have to do *anything*.

Regards,

James

Re: linus-next: improving functional testing for to-be-merged pull requests

[Paul E. McKenney]

On Tue, Oct 22, 2024 at 10:50:04PM -0700, Christoph Hellwig wrote:
> On Tue, Oct 22, 2024 at 05:37:38AM -0400, Sasha Levin wrote:
> > We could add a report for the above, but:
> > 
> > 1. Linus consistently pulls patches that haven't seen the light of day.
> > 2. Linus explicitly objected to making a linux-next a must have.
> > 
> > So unless these results would be actually used, what's the point in
> > writing all of that?
> 
> Yes, without Linus caring we're not going to get our process worked out.
> Not sure how a tree that probably won't have much better latency than
> linux-next is going to fix that, though.

If I recall correctly, one thing Linus asked us to do earlier this year
(ARM Summit) is to CC him on -next failures.  I have been failing to do
this, so I will post myself a note or something to remind me.

After all, if Linus doesn't know of a problem with a set of commits,
how does he know not to pull it?

							Thanx, Paul

Re: linus-next: improving functional testing for to-be-merged pull requests

[Paul E. McKenney]

On Wed, Oct 23, 2024 at 05:19:14AM -0400, Steven Rostedt wrote:

[ . . . ]

> But pushing to linux-next for a day or two, what does that give me?

If the stars align, I will run some serious rcutorture on it.  Not that
the stars have been aligning much over the past few weeks because I
have been putting most of my testing effort into chasing an RCU bug.
And not that rcutorture and friends are likely to find bugs in your part
of the Linux kernel.

Nevertheless, this is one of my concerns with a separate tree.  Right now,
my testing priority is (1) RCU, (2) -next, (3) mainline.  If we add
another tree, either I choose not to test that tree on the one hand or
it detracts from my testing -next and/or mainline on the other.

							Thanx, Paul

Re: linus-next: improving functional testing for to-be-merged pull requests

[Guenter Roeck]

On 10/23/24 10:47, Paul E. McKenney wrote:
> On Tue, Oct 22, 2024 at 10:50:04PM -0700, Christoph Hellwig wrote:
>> On Tue, Oct 22, 2024 at 05:37:38AM -0400, Sasha Levin wrote:
>>> We could add a report for the above, but:
>>>
>>> 1. Linus consistently pulls patches that haven't seen the light of day.
>>> 2. Linus explicitly objected to making a linux-next a must have.
>>>
>>> So unless these results would be actually used, what's the point in
>>> writing all of that?
>>
>> Yes, without Linus caring we're not going to get our process worked out.
>> Not sure how a tree that probably won't have much better latency than
>> linux-next is going to fix that, though.
> 
> If I recall correctly, one thing Linus asked us to do earlier this year
> (ARM Summit) is to CC him on -next failures.  I have been failing to do
> this, so I will post myself a note or something to remind me.
> 
> After all, if Linus doesn't know of a problem with a set of commits,
> how does he know not to pull it?
> 

As of right now:

Build results:
	total: 158 pass: 144 fail: 14
Failed builds:
	arm:axm55xx_defconfig
	csky:defconfig
	i386:allyesconfig
	i386:allmodconfig
	microblaze:defconfig
	mips:allmodconfig
	mips:mtx1_defconfig
	mips:db1xxx_defconfig
	openrisc:allmodconfig
	parisc:allmodconfig
	parisc64:generic-64bit_defconfig
	sh:defconfig
	sh:shx3_defconfig
	xtensa:allmodconfig
Qemu test results:
	total: 556 pass: 528 fail: 28
Unit test results:
	pass: 463943 fail: 0

And that is a good day. Sometimes dozens of builds and hundreds
of boot tests fail. Analyzing those failures would be a full-time job.
Who do you expect would or should do that ?

Guenter

Re: linus-next: improving functional testing for to-be-merged pull requests

[Linus Torvalds]

On Wed, 23 Oct 2024 at 10:47, Paul E. McKenney <paulmck@kernel.org> wrote:
>
> > Yes, without Linus caring we're not going to get our process worked out.
> > Not sure how a tree that probably won't have much better latency than
> > linux-next is going to fix that, though.
>
> If I recall correctly, one thing Linus asked us to do earlier this year
> (ARM Summit) is to CC him on -next failures.

Yes. I definitely care about failures in linux-next, but I often don't
_know_ about them unless I'm told.

The linux-next automation sends notifications to the owners of the
trees, but not to me.

And that is as it should be - I don't want to be spammed by everything
that is found in linux-next.

So failures should normally be fixed by the owners of the trees when
they get detected, and those trees should not even make it as pull
requests until the problem has been resolved.

After all, that's kind of the point of linux-next - finding problems
*before* they become pull requests.

But that does mean that if some failure has been detected in
linux-next, and the maintainer has *not* corrected it, I don't even
know about it. And most maintainers are very good about this, and
point to things like the resolution reports (which are not actual
problems, just heads-up).

So automation that says "this tree does not actually work" would very
much be appreciated. I'd prefer them to be the same kind of "before
the pull request has even been sent" situation, of course, but if
problems _remain_ in linux-next, and pr-bot sees the pull request, I'd
actually like automation that says "Oh, this tree has these issues:
..."

IOW, I very much do care. linux-next has improved our build test
coverage and is a big deal. The more "problem coverage" it has, even
outside of just build issues, the better.

And yes, I know some people do functional testing on linux-next
already. The message at the maintainer summit was a bit mixed with
some people saying linux-next tends to work even for that, others
saying it's often too broken to be useful.

              Linus

Re: linus-next: improving functional testing for to-be-merged pull requests

[Linus Torvalds]

On Wed, 23 Oct 2024 at 11:05, Guenter Roeck <linux@roeck-us.net> wrote:
>
> And that is a good day. Sometimes dozens of builds and hundreds
> of boot tests fail. Analyzing those failures would be a full-time job.
> Who do you expect would or should do that ?

Yeah, this is the problem. I think it's only useful if some automation
(not humans! That would make people burn out immediately) can actually
pinpoint the trees that introduced the failures.

And I think that would be absolutely lovely. But I suspect the testing
requirements then have latencies long enough that getting to that
point might not be entirely realistic.

              Linus

Re: linus-next: improving functional testing for to-be-merged pull requests

[Paul E. McKenney]

On Wed, Oct 23, 2024 at 11:06:59AM -0700, Linus Torvalds wrote:
> On Wed, 23 Oct 2024 at 10:47, Paul E. McKenney <paulmck@kernel.org> wrote:
> >
> > > Yes, without Linus caring we're not going to get our process worked out.
> > > Not sure how a tree that probably won't have much better latency than
> > > linux-next is going to fix that, though.
> >
> > If I recall correctly, one thing Linus asked us to do earlier this year
> > (ARM Summit) is to CC him on -next failures.
> 
> Yes. I definitely care about failures in linux-next, but I often don't
> _know_ about them unless I'm told.
> 
> The linux-next automation sends notifications to the owners of the
> trees, but not to me.

OK, then I guess I was inadvertently doing the right thing by forgetting
to CC you on all -next issues my testing has located.  ;-)

[ . . . ]

> And yes, I know some people do functional testing on linux-next
> already. The message at the maintainer summit was a bit mixed with
> some people saying linux-next tends to work even for that, others
> saying it's often too broken to be useful.

Functional testing?

Me, I do rcutorture *stress* testing on -next, and it usually passes.
Yes, there is the occasional spectacular exception, like the version
last month where rcutorture found the better part of ten bugs.  Two of
which were ugly heisenbugs that are still being chased down.

(Full disclosure: One of those bugs is in RCU, but that bug is already
in mainline.  Much of my time over the past two weeks has gone into
moving it from a once-per-year heisenbug to more than ten failures
per hour.  This bug does not happen without lots of CPU hotplug, as in
a 50-millisecond delay between successive CPU-hotplug operations.)

What I do is to carry patches against -next, and this week I am down
from two to one of them.  But most of the time, my list of -next-fix
patches is of length zero.

Now, I freely admit that rcutorture doesn't hammer all that much of
the kernel: mostly just RCU, timers, the scheduler, and CPU hotplug.
So I could easily imagine that -next testing of other parts of the kernel
might be more failure-prone.  Except that the correct answer to that is
*more* -next testing, not less of it.

							Thanx, Paul

Re: linus-next: improving functional testing for to-be-merged pull requests

[Geert Uytterhoeven]

Hi Linus,

On Wed, Oct 23, 2024 at 8:10 PM Linus Torvalds
<torvalds@linux-foundation.org> wrote:
> On Wed, 23 Oct 2024 at 11:05, Guenter Roeck <linux@roeck-us.net> wrote:
> > And that is a good day. Sometimes dozens of builds and hundreds
> > of boot tests fail. Analyzing those failures would be a full-time job.
> > Who do you expect would or should do that ?
>
> Yeah, this is the problem. I think it's only useful if some automation
> (not humans! That would make people burn out immediately) can actually
> pinpoint the trees that introduced the failures.
>
> And I think that would be absolutely lovely. But I suspect the testing
> requirements then have latencies long enough that getting to that
> point might not be entirely realistic.

At least for the build failures, zero day does a good job, and does
inform the author and appropriate maintainer.
Unfortunately maintainers can still ignore any reported issues...

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: linus-next: improving functional testing for to-be-merged pull requests

[Linus Torvalds]

On Wed, 23 Oct 2024 at 11:37, Paul E. McKenney <paulmck@kernel.org> wrote:
>
> Functional testing?

Yes. Things like the stuff that Guenter does every -rc, and that
invariably finds several issues every single merge window.

Like actually booting up the kernel on multiple different odd
architectures. But also much more complex things than just running a
torture test for a very small piece of the kernel.

The same way linux-next does the build testing on many different
architectures, and that often finds things that developers missed
simply because the bulk of developers are still purely on x86-64.

              Linus

Re: linus-next: improving functional testing for to-be-merged pull requests

[Paul E. McKenney]

On Wed, Oct 23, 2024 at 12:24:05PM -0700, Linus Torvalds wrote:
> On Wed, 23 Oct 2024 at 11:37, Paul E. McKenney <paulmck@kernel.org> wrote:
> >
> > Functional testing?
> 
> Yes. Things like the stuff that Guenter does every -rc, and that
> invariably finds several issues every single merge window.
> 
> Like actually booting up the kernel on multiple different odd
> architectures. But also much more complex things than just running a
> torture test for a very small piece of the kernel.
> 
> The same way linux-next does the build testing on many different
> architectures, and that often finds things that developers missed
> simply because the bulk of developers are still purely on x86-64.

Fair point, my testing is still strictly x86-64.

							Thanx, Paul

Re: linus-next: improving functional testing for to-be-merged pull requests

[Theodore Ts'o]

On Wed, Oct 23, 2024 at 11:06:59AM -0700, Linus Torvalds wrote:
> 
> So automation that says "this tree does not actually work" would very
> much be appreciated. I'd prefer them to be the same kind of "before
> the pull request has even been sent" situation, of course, but if
> problems _remain_ in linux-next, and pr-bot sees the pull request, I'd
> actually like automation that says "Oh, this tree has these issues:
> ..."

As I think I mentioned at the Maintainer's Summit, for the past 2 or 3
months, I've been running xfstests autmation for ext4, xfs, f2fs, and
btrfs on the fs-next branch of the linux-next tree, and it's actually
been quite stable.  That's because most of the fs developers are
pretty careful about running tests *before* publishing their branches
to linux-next, so it hasn't had the boot failures whichhas caused fs
testing on the full linux-next branch to be more painful.  Yes, I know
that solution doesn't scale but it works for us.  :-)

An example of how this has worked out is a regression which popped up
a few days ago, causing generic/564 to fail for ext4, xfs, f2fs, and
btrfs, and which was fixed by reverting "vfs: Fix implicit conversion
problem when testing overflow case".  I expect Christian will be
dropping that commit (or replacing it with a fixed patch) from the vfs
tree long before he sends a pull request to you.

       	    	    	      	     	- Ted

Re: linus-next: improving functional testing for to-be-merged pull requests

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1521 bytes --]

On Wed, Oct 23, 2024 at 11:06:59AM -0700, Linus Torvalds wrote:

> And yes, I know some people do functional testing on linux-next
> already. The message at the maintainer summit was a bit mixed with
> some people saying linux-next tends to work even for that, others
> saying it's often too broken to be useful.

It very much depends on what you're trying to get out of the testing -
-next does work well most of the time, but it will absolutely just blow
up catastrophically on you from time time to time so you have to be
prepared to cope with loosing some or all of your coverage sometimes.
Usually anything major gets fixed fairly promptly, but sometimes you'll
be missing coverage for extended periods especially if it's something
like a more niche platform that's been broken or there's some problem
getting people to actually apply the fixes.  Submaintainer trees that
people don't want to add to -next can be an issue too.

You're also going to run into issues that are nothing to do with
whatever you're actually working on yourself and need to consider what
you're covering based on your tolerance for dealing with that.  The rate
of change can also be an issue if the tests you're intersted in are
expensive.  OTOH if you're doing things that are likely to be affected
by changes in a broad set of trees (eg, maintaining some embedded
platform where you care about all the various subsystems breaking
platform specific drivers) it can be a lot easier to cover -next rather
than all the individual subsystems.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: linus-next: improving functional testing for to-be-merged pull requests

[Paul E. McKenney]

On Wed, Oct 23, 2024 at 10:24:29PM +0100, Mark Brown wrote:
> On Wed, Oct 23, 2024 at 11:06:59AM -0700, Linus Torvalds wrote:
> 
> > And yes, I know some people do functional testing on linux-next
> > already. The message at the maintainer summit was a bit mixed with
> > some people saying linux-next tends to work even for that, others
> > saying it's often too broken to be useful.
> 
> It very much depends on what you're trying to get out of the testing -
> -next does work well most of the time, but it will absolutely just blow
> up catastrophically on you from time time to time so you have to be
> prepared to cope with loosing some or all of your coverage sometimes.
> Usually anything major gets fixed fairly promptly, but sometimes you'll
> be missing coverage for extended periods especially if it's something
> like a more niche platform that's been broken or there's some problem
> getting people to actually apply the fixes.  Submaintainer trees that
> people don't want to add to -next can be an issue too.
> 
> You're also going to run into issues that are nothing to do with
> whatever you're actually working on yourself and need to consider what
> you're covering based on your tolerance for dealing with that.  The rate
> of change can also be an issue if the tests you're intersted in are
> expensive.  OTOH if you're doing things that are likely to be affected
> by changes in a broad set of trees (eg, maintaining some embedded
> platform where you care about all the various subsystems breaking
> platform specific drivers) it can be a lot easier to cover -next rather
> than all the individual subsystems.

You said it much better than I did, thank you!

For me, -next is a convenient point to test much of what will be going in.
Yes, it can be frustrating, finding problems just as someone else fixed
them, finding problems irrelevant to any of my use cases, and so on.
But sometimes I find something that would have been quite painful to
deal with later in process that others don't find.  Overall, it is well
worth the my effort.

							Thanx, Paul

Re: linus-next: improving functional testing for to-be-merged pull requests

[Michael Ellerman]

Steven Rostedt <rostedt@goodmis.org> writes:
> On Wed, 23 Oct 2024 10:36:20 +0200
> Geert Uytterhoeven <geert@linux-m68k.org> wrote:
>
>> > To put it this way. The bugs I'm fixing was for code in linux-next
>> > where the bugs were never found. They only appeared when they went into
>> > Linus's tree. So why put the fixes in linux-next, if it didn't catch
>> > the bugs I fixed in the first place?  
>> 
>> Hmmm...
>> 
>> Your arguments sound very similar to those being used in recent
>> discussions about not posting patches for public review...
>> 
>> Please follow the process! ;-)
>
> What process?
>
...
>
> But pushing to linux-next for a day or two, what does that give me?

Several thousand build tests, across pretty much every architecture.

And a few hundred boot tests, lots virtualised, but some on real HW.

A single character typo in an #ifdef your testing doesn't cover can
break the build for lots of people ...

cheers

Re: linus-next: improving functional testing for to-be-merged pull requests

[Steven Rostedt]

On Thu, 24 Oct 2024 14:59:16 +1100
Michael Ellerman <mpe@ellerman.id.au> wrote:

> > But pushing to linux-next for a day or two, what does that give me?  
> 
> Several thousand build tests, across pretty much every architecture.
> 
> And a few hundred boot tests, lots virtualised, but some on real HW.
> 
> A single character typo in an #ifdef your testing doesn't cover can
> break the build for lots of people ...

I use to get that from the zero-day bot from intel. Does that not run
anymore? I noticed that I don't get notifications anymore.

So yeah, my work flow is to push to my "urgent" branch, and wait a day
for the zero-day bot. But if that's no longer the case, then maybe I
can ask Stephen to take my urgent branches.

Now the issue here is that I push to my urgent branches *before* I run
my tests. That's because my tests take 8 - 13 hours and I want the
zero-day bot to get a head start. When a bug was in my code, both my
tests and the zero-day bot would catch it. I would check to see which
one found it first. Sometimes it was my tests, sometimes the bot.

Now the issue is, if a bug is found. I don't change the branch I pushed
up. Thus, if its broken, it will stay broken until I get around to
fixing it. Depending on how badly it is broken, it may be several days
before its fixed. I would imagine if this branch is now in next, it
would break next and burden Stephen to revert my branch until its
fixed. Or it will change my workflow to have to either push after it
passes my tests or revert what I pushed first. I'm guessing I would
have to now push after my tests pass (if zero-day is no longer working).

As for the linus-next (linus-pr), this is a case that could work with
my workflow. I would happily sign up, and when I do my git pull, have
a bot pull my code into this tree (along with all the other pull
request for the day) run all these tests, and then Linus can pull after
they all passed.

-- Steve

Re: linus-next: improving functional testing for to-be-merged pull requests

[Guenter Roeck]

On 10/23/24 22:01, Steven Rostedt wrote:
> On Thu, 24 Oct 2024 14:59:16 +1100
> Michael Ellerman <mpe@ellerman.id.au> wrote:
> 
>>> But pushing to linux-next for a day or two, what does that give me?
>>
>> Several thousand build tests, across pretty much every architecture.
>>
>> And a few hundred boot tests, lots virtualised, but some on real HW.
>>
>> A single character typo in an #ifdef your testing doesn't cover can
>> break the build for lots of people ...
> 
> I use to get that from the zero-day bot from intel. Does that not run
> anymore? I noticed that I don't get notifications anymore.
> 

I do get notifications, so it is still running.
Its configuration is (still) at https://github.com/intel/lkp-tests.git,
so you can check yourself if your current repositories and branches are
listed (and send a pull request to update it if needed). I see

repo/linux/rostedt-kconfig:owner: Steven Rostedt <rostedt@goodmis.org>
repo/linux/rostedt-ktest:owner: Steven Rostedt <rostedt@goodmis.org>
repo/linux/rostedt-trace:owner: Steven Rostedt <rostedt@goodmis.org>

so at least some testing should still happen. I did notice though
that "notify_build_success_branch:" is only set in one of the files,
so you might not get notified if a build was successful in the other
two.

Guenter

Re: linus-next: improving functional testing for to-be-merged pull requests

[Steven Rostedt]

On Wed, 23 Oct 2024 22:16:31 -0700
Guenter Roeck <linux@roeck-us.net> wrote:
> 
> I do get notifications, so it is still running.
> Its configuration is (still) at https://github.com/intel/lkp-tests.git,
> so you can check yourself if your current repositories and branches are
> listed (and send a pull request to update it if needed). I see
> 
> repo/linux/rostedt-kconfig:owner: Steven Rostedt <rostedt@goodmis.org>
> repo/linux/rostedt-ktest:owner: Steven Rostedt <rostedt@goodmis.org>
> repo/linux/rostedt-trace:owner: Steven Rostedt <rostedt@goodmis.org>
> 
> so at least some testing should still happen. I did notice though
> that "notify_build_success_branch:" is only set in one of the files,
> so you might not get notified if a build was successful in the other
> two.
> 

Thanks for the update!

Yeah, I started using topic branches (requested by Linus) and I didn't
update the success notifications. That explains why I don't receive
them anymore.

Now I have to ask. What's the benefit of pushing to linux-next over
waiting for the zero-day bot?

-- Steve

Re: linus-next: improving functional testing for to-be-merged pull requests

[Geert Uytterhoeven]

Hi Steven,

On Thu, Oct 24, 2024 at 8:49 AM Steven Rostedt <rostedt@goodmis.org> wrote:
> On Wed, 23 Oct 2024 22:16:31 -0700
> Guenter Roeck <linux@roeck-us.net> wrote:
> > I do get notifications, so it is still running.
> > Its configuration is (still) at https://github.com/intel/lkp-tests.git,
> > so you can check yourself if your current repositories and branches are
> > listed (and send a pull request to update it if needed). I see
> >
> > repo/linux/rostedt-kconfig:owner: Steven Rostedt <rostedt@goodmis.org>
> > repo/linux/rostedt-ktest:owner: Steven Rostedt <rostedt@goodmis.org>
> > repo/linux/rostedt-trace:owner: Steven Rostedt <rostedt@goodmis.org>
> >
> > so at least some testing should still happen. I did notice though
> > that "notify_build_success_branch:" is only set in one of the files,
> > so you might not get notified if a build was successful in the other
> > two.
>
> Thanks for the update!
>
> Yeah, I started using topic branches (requested by Linus) and I didn't
> update the success notifications. That explains why I don't receive
> them anymore.
>
> Now I have to ask. What's the benefit of pushing to linux-next over
> waiting for the zero-day bot?

On Thu, Oct 24, 2024 at 5:59 AM Michael Ellerman <mpe@ellerman.id.au> wrote:
> Several thousand build tests, across pretty much every architecture.
>
> And a few hundred boot tests, lots virtualised, but some on real HW.
>
> A single character typo in an #ifdef your testing doesn't cover can
> break the build for lots of people ...

Or a missing "static" for a dummy function.
Or a plain 64-bit division.
Or ...

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: linus-next: improving functional testing for to-be-merged pull requests

[Steven Rostedt]

On Thu, 24 Oct 2024 09:01:15 +0200
Geert Uytterhoeven <geert@linux-m68k.org> wrote:


> On Thu, Oct 24, 2024 at 5:59 AM Michael Ellerman <mpe@ellerman.id.au> wrote:
> > Several thousand build tests, across pretty much every architecture.
> >
> > And a few hundred boot tests, lots virtualised, but some on real HW.
> >
> > A single character typo in an #ifdef your testing doesn't cover can
> > break the build for lots of people ...  
> 
> Or a missing "static" for a dummy function.
> Or a plain 64-bit division.
> Or ...

Note, my fixes code seldom adds dummy functions. I like to try to keep
them as small as possible. That's not always the case, so maybe I could
push it. But it will change my workflow quite a bit or burden Stephen
with broken branches.

I'm still not convinced it's worth it.

We are not talking about new development code. We are talking about bug
fixes for code that is in Linus's tree. The zero day bot and my tests
appear to find most issues. Bugs that happened on my fixes patches are
usually other use cases. For instance, cpu hotplug while tracing from
rtla. That's not coverage I get from linux-next.

-- Steve

Re: linus-next: improving functional testing for to-be-merged pull requests

[Christoph Hellwig]

On Thu, Oct 24, 2024 at 05:21:39AM -0400, Steven Rostedt wrote:
> them as small as possible. That's not always the case, so maybe I could
> push it. But it will change my workflow quite a bit or burden Stephen
> with broken branches.
> 
> I'm still not convinced it's worth it.
> 
> We are not talking about new development code. We are talking about bug
> fixes for code that is in Linus's tree. The zero day bot and my tests
> appear to find most issues. Bugs that happened on my fixes patches are
> usually other use cases. For instance, cpu hotplug while tracing from
> rtla. That's not coverage I get from linux-next.

Seriously, just add your damn fixes tree to linux-next.  If your fixes
are as perfect as you claim that one time setup is all you need to do,
and you have less work than you spent arguing on this thread already.

If it catches bugs eventually you will need to do more work, but save
others from deadling with your regression.  There is no downside for
you.

Re: linus-next: improving functional testing for to-be-merged pull requests

[Steven Rostedt]

On Thu, 24 Oct 2024 02:24:53 -0700
Christoph Hellwig <hch@infradead.org> wrote:

> On Thu, Oct 24, 2024 at 05:21:39AM -0400, Steven Rostedt wrote:
> > them as small as possible. That's not always the case, so maybe I could
> > push it. But it will change my workflow quite a bit or burden Stephen
> > with broken branches.
> > 
> > I'm still not convinced it's worth it.
> > 
> > We are not talking about new development code. We are talking about bug
> > fixes for code that is in Linus's tree. The zero day bot and my tests
> > appear to find most issues. Bugs that happened on my fixes patches are
> > usually other use cases. For instance, cpu hotplug while tracing from
> > rtla. That's not coverage I get from linux-next.  
> 
> Seriously, just add your damn fixes tree to linux-next.  If your fixes
> are as perfect as you claim that one time setup is all you need to do,
> and you have less work than you spent arguing on this thread already.
> 
> If it catches bugs eventually you will need to do more work, but save
> others from deadling with your regression.  There is no downside for
> you.

So basically, all I need to do to satisfy your request is to add fixes
branch that I push to that is pushed after it passes my tests (and not
the urgent branch that is still being tested and may have bugs) and
then have that be added to linux-next?

Now I have been batching changes and not send a pull request right
after my tests pass. I've been sending a pull request at most now once
a week. So I could have this branch hold the code that's already been
tested.

-- Steve

Re: linus-next: improving functional testing for to-be-merged pull requests

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 683 bytes --]

On Thu, Oct 24, 2024 at 05:49:09AM -0400, Steven Rostedt wrote:

> So basically, all I need to do to satisfy your request is to add fixes
> branch that I push to that is pushed after it passes my tests (and not
> the urgent branch that is still being tested and may have bugs) and
> then have that be added to linux-next?

> Now I have been batching changes and not send a pull request right
> after my tests pass. I've been sending a pull request at most now once
> a week. So I could have this branch hold the code that's already been
> tested.

Yes, that's what pretty much everyone is doing here.  Generally we find
very few issues this way but it's certainly a non-zero number.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: linus-next: improving functional testing for to-be-merged pull requests

[Christoph Hellwig]

On Thu, Oct 24, 2024 at 12:08:28PM +0100, Mark Brown wrote:
> On Thu, Oct 24, 2024 at 05:49:09AM -0400, Steven Rostedt wrote:
> 
> > So basically, all I need to do to satisfy your request is to add fixes
> > branch that I push to that is pushed after it passes my tests (and not
> > the urgent branch that is still being tested and may have bugs) and
> > then have that be added to linux-next?
> 
> > Now I have been batching changes and not send a pull request right
> > after my tests pass. I've been sending a pull request at most now once
> > a week. So I could have this branch hold the code that's already been
> > tested.
> 
> Yes, that's what pretty much everyone is doing here.  Generally we find
> very few issues this way but it's certainly a non-zero number.

*nod*

Re: linus-next: improving functional testing for to-be-merged pull requests

[Guenter Roeck]

On 10/23/24 23:49, Steven Rostedt wrote:
> On Wed, 23 Oct 2024 22:16:31 -0700
> Guenter Roeck <linux@roeck-us.net> wrote:
>>
>> I do get notifications, so it is still running.
>> Its configuration is (still) at https://github.com/intel/lkp-tests.git,
>> so you can check yourself if your current repositories and branches are
>> listed (and send a pull request to update it if needed). I see
>>
>> repo/linux/rostedt-kconfig:owner: Steven Rostedt <rostedt@goodmis.org>
>> repo/linux/rostedt-ktest:owner: Steven Rostedt <rostedt@goodmis.org>
>> repo/linux/rostedt-trace:owner: Steven Rostedt <rostedt@goodmis.org>
>>
>> so at least some testing should still happen. I did notice though
>> that "notify_build_success_branch:" is only set in one of the files,
>> so you might not get notified if a build was successful in the other
>> two.
>>
> 
> Thanks for the update!
> 
> Yeah, I started using topic branches (requested by Linus) and I didn't
> update the success notifications. That explains why I don't receive
> them anymore.
> 
> Now I have to ask. What's the benefit of pushing to linux-next over
> waiting for the zero-day bot?
> 

I push my changes into the same branches that are checked by 0-day
and pulled into linux-next. linux-next shows interference with other
branches. Once in a while I do get a notification telling me that
one or more of the patches interfere with other patches, so I know that
something happened, and I can prepare for that for the next commit window.

Testing-wise, I do run build and boot tests on linux-next (the same tests
as those running on release candidates), so I do know what is wrong there
and (which did happen a couple of times) if a patch in one of my trees
is responsible.

Yes, that means that in many cases I do know ahead of time which problems
are going to pop up in the mainline kernel. But I don't have the time
tracking those down when seen in linux-next - there are just too many
and, as already mentioned, that would be a full-time job on its own.
Also, it happens a lot that they have been reported but the report was
ignored or missed. On top of that I found that _if_ I am reporting them,
the receiving side is at least sometimes either not responsive to almost
abusive, so for the most part I gave up on it (and frankly I found that
people tend to be _much_ more responsive if one Linus Torvalds is listed
in Cc:).

Note that I do collect known fixes in my 'fixes' and 'testing' branches,
primarily to have something clean available to keep testing. Linus even
pulled my fixes branch once directly because the responsible maintainers
didn't send pull requests to him for weeks.

Guenter

Re: linus-next: improving functional testing for to-be-merged pull requests

[Luis Chamberlain]

On Thu, Oct 24, 2024 at 02:49:28AM -0400, Steven Rostedt wrote:
> Now I have to ask. What's the benefit of pushing to linux-next over
> waiting for the zero-day bot?

0-day only does build tests by default, there are many places which have
actual run time tests which *run* off of linux-next, those are both bots
and human. Granted you can get your own run time tests out of your own
branches but that's on each developer to set up and a developer's test
exposure of just one branch is small compared to linux-next. For example
I've seen obscure bugs creep up on linux-next for modules which only some
odd arch or setup was able to capture before which no test we had during
development was able to capture. So more exposure to system variability
and test variability.

The other benefit is you get to see *way ahead of time* possible merge
conflicts, and if you can coordinate with the respective maintainers
which your code conflicts with, you can prepare so that this is smooth
sailing upon pull request to Linus.

  Luis

Re: linus-next: improving functional testing for to-be-merged pull requests

[Steven Rostedt]

On Thu, 24 Oct 2024 07:39:00 -0700
Guenter Roeck <linux@roeck-us.net> wrote:

> > 
> > Now I have to ask. What's the benefit of pushing to linux-next over
> > waiting for the zero-day bot?
> >   
> 
> I push my changes into the same branches that are checked by 0-day
> and pulled into linux-next. linux-next shows interference with other
> branches. Once in a while I do get a notification telling me that
> one or more of the patches interfere with other patches, so I know that
> something happened, and I can prepare for that for the next commit window.

Remember, this is about pushing to linux-next before sending fixes
after -rc1. Not for things that are going to land in the next merge
window. My fixes seldom ever interfere with others work as it's usually
much more focused on code that is already in Linus's tree. Like adding
a missing mutex_unlock() from an error path. How is it helpful to push
something like that to linux-next?

> 
> Testing-wise, I do run build and boot tests on linux-next (the same tests
> as those running on release candidates), so I do know what is wrong there
> and (which did happen a couple of times) if a patch in one of my trees
> is responsible.
> 
> Yes, that means that in many cases I do know ahead of time which problems
> are going to pop up in the mainline kernel. But I don't have the time
> tracking those down when seen in linux-next - there are just too many
> and, as already mentioned, that would be a full-time job on its own.
> Also, it happens a lot that they have been reported but the report was
> ignored or missed. On top of that I found that _if_ I am reporting them,
> the receiving side is at least sometimes either not responsive to almost
> abusive, so for the most part I gave up on it (and frankly I found that
> people tend to be _much_ more responsive if one Linus Torvalds is listed
> in Cc:).
> 
> Note that I do collect known fixes in my 'fixes' and 'testing' branches,
> primarily to have something clean available to keep testing. Linus even
> pulled my fixes branch once directly because the responsible maintainers
> didn't send pull requests to him for weeks.

Or are you saying that it's helpful to "fix" linux-next before fixing
Linus's tree? That way others will have the fixes too?

-- Steve

Re: linus-next: improving functional testing for to-be-merged pull requests

[Steven Rostedt]

On Thu, 24 Oct 2024 10:53:19 -0700
Luis Chamberlain <mcgrof@kernel.org> wrote:

> On Thu, Oct 24, 2024 at 02:49:28AM -0400, Steven Rostedt wrote:
> > Now I have to ask. What's the benefit of pushing to linux-next over
> > waiting for the zero-day bot?  
> 
> 0-day only does build tests by default, there are many places which have
> actual run time tests which *run* off of linux-next, those are both bots
> and human. Granted you can get your own run time tests out of your own
> branches but that's on each developer to set up and a developer's test
> exposure of just one branch is small compared to linux-next. For example
> I've seen obscure bugs creep up on linux-next for modules which only some
> odd arch or setup was able to capture before which no test we had during
> development was able to capture. So more exposure to system variability
> and test variability.

I have a test suite that takes 8 to 13 hours to run (depending on how
many commits I'm testing) that I run before sending to Linus or linux-next.

> 
> The other benefit is you get to see *way ahead of time* possible merge
> conflicts, and if you can coordinate with the respective maintainers
> which your code conflicts with, you can prepare so that this is smooth
> sailing upon pull request to Linus.
> 

Remember, this is talking about fixes after -rc1 not for things heading
to the merge window. I find linux-next extremely useful for that work.
But for fixes, what benefit is it to push to linux-next before sending
to Linus a fix that adds a missing mutex_unlock() in the error path?

-- Steve

Re: linus-next: improving functional testing for to-be-merged pull requests

[Luis Chamberlain]

On Thu, Oct 24, 2024 at 09:17:28PM -0400, Steven Rostedt wrote:
> Remember, this is talking about fixes after -rc1 not for things heading
> to the merge window. I find linux-next extremely useful for that work.
> But for fixes, what benefit is it to push to linux-next before sending
> to Linus a fix that adds a missing mutex_unlock() in the error path?

Yeah, sorry I did not get that context, in that case think the value is
more immediate than a round trip through 0-day given linus tree has
more immediacy for testing and value for fixes.

  Luis

Re: linus-next: improving functional testing for to-be-merged pull requests

[Guenter Roeck]

On 10/24/24 18:11, Steven Rostedt wrote:
> On Thu, 24 Oct 2024 07:39:00 -0700
> Guenter Roeck <linux@roeck-us.net> wrote:
> 
>>>
>>> Now I have to ask. What's the benefit of pushing to linux-next over
>>> waiting for the zero-day bot?
>>>    
>>
>> I push my changes into the same branches that are checked by 0-day
>> and pulled into linux-next. linux-next shows interference with other
>> branches. Once in a while I do get a notification telling me that
>> one or more of the patches interfere with other patches, so I know that
>> something happened, and I can prepare for that for the next commit window.
> 
> Remember, this is about pushing to linux-next before sending fixes
> after -rc1. Not for things that are going to land in the next merge
> window. My fixes seldom ever interfere with others work as it's usually
> much more focused on code that is already in Linus's tree. Like adding
> a missing mutex_unlock() from an error path. How is it helpful to push
> something like that to linux-next?
> 

I still try to have my patches rest in -next for a few days before sending
a pull request to Linus. At the very least this gives others a chance to
pick up those patches if they encounter a problem fixed by them. Also,
sometimes bug fixes do introduce new problems, so, yes, I think it is
very useful to have as many eyes (or test systems) as possible look
at them before sending a pull request.

>>
>> Testing-wise, I do run build and boot tests on linux-next (the same tests
>> as those running on release candidates), so I do know what is wrong there
>> and (which did happen a couple of times) if a patch in one of my trees
>> is responsible.
>>
>> Yes, that means that in many cases I do know ahead of time which problems
>> are going to pop up in the mainline kernel. But I don't have the time
>> tracking those down when seen in linux-next - there are just too many
>> and, as already mentioned, that would be a full-time job on its own.
>> Also, it happens a lot that they have been reported but the report was
>> ignored or missed. On top of that I found that _if_ I am reporting them,
>> the receiving side is at least sometimes either not responsive to almost
>> abusive, so for the most part I gave up on it (and frankly I found that
>> people tend to be _much_ more responsive if one Linus Torvalds is listed
>> in Cc:).
>>
>> Note that I do collect known fixes in my 'fixes' and 'testing' branches,
>> primarily to have something clean available to keep testing. Linus even
>> pulled my fixes branch once directly because the responsible maintainers
>> didn't send pull requests to him for weeks.
> 
> Or are you saying that it's helpful to "fix" linux-next before fixing
> Linus's tree? That way others will have the fixes too?
> 

My fixes and testing branches apply on top of mainline. All patches in the fixes
branch have been sent to maintainers, and they _should_ be (and for the most
part are) available in linux-next. If they are not, the maintainers did not
respond to the patch e-mails or push them out to any branch that is used to
generate -next. The only exception is if I needed to revert some patch to work
around a problem, but even then I make sure that the responsible maintainer
knows about the problem (if they read their email).

Guenter

Re: linus-next: improving functional testing for to-be-merged pull requests

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 1131 bytes --]

On Thu, Oct 24, 2024 at 09:11:49PM -0400, Steven Rostedt wrote:

> much more focused on code that is already in Linus's tree. Like adding
> a missing mutex_unlock() from an error path. How is it helpful to push
> something like that to linux-next?

How is it helpful to not push things to -next?  Pushing your unsent
fixes to a branch that Stephen can pick up costs you approximately
nothing so there's no meaningful downside but perhaps one of these days
some test system will find some issue and it's setting a good example
for those who don't (or can't) have the same detailed testing you have.

> > Note that I do collect known fixes in my 'fixes' and 'testing' branches,
> > primarily to have something clean available to keep testing. Linus even
> > pulled my fixes branch once directly because the responsible maintainers
> > didn't send pull requests to him for weeks.

> Or are you saying that it's helpful to "fix" linux-next before fixing
> Linus's tree? That way others will have the fixes too?

That's also true, it gets the fixes into the hands of people doing -next
testing faster which is hopefully useful to them.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: linus-next: improving functional testing for to-be-merged pull requests

[Paul E. McKenney]

On Thu, Oct 24, 2024 at 09:11:49PM -0400, Steven Rostedt wrote:
> On Thu, 24 Oct 2024 07:39:00 -0700
> Guenter Roeck <linux@roeck-us.net> wrote:

[ . . . ]

> > Note that I do collect known fixes in my 'fixes' and 'testing' branches,
> > primarily to have something clean available to keep testing. Linus even
> > pulled my fixes branch once directly because the responsible maintainers
> > didn't send pull requests to him for weeks.
> 
> Or are you saying that it's helpful to "fix" linux-next before fixing
> Linus's tree? That way others will have the fixes too?

It is a risk-management issue, and so the answer will vary depending on
the situation.

For one extreme, if the fix is a regression, but the bug is an odd corner
case that is unlikely to be triggered, then it is very helpful to run
it through 0day and -next before sending the pull request to Linus.
This is because the delay won't inconvenient many people, if any, and
0day and -next can help catch any bugs introduced by the fix.

At the other extreme, if the fix is for a regression that (say) bricks
50% of the systems out there, clearly speed is of the essence.  And let's
face it, in this case, Linus might well just apply your initial patch.
Which would expose it to both 0day and -next.

But!!!  In this case, I sure would hope that there would then be some
serious follow up to fix the bug in the validation, which clearly failed
to catch a very ugly bug.

							Thanx, Paul

Re: linus-next: improving functional testing for to-be-merged pull requests

[Jiri Kosina]

On Thu, 24 Oct 2024, Steven Rostedt wrote:

> So basically, all I need to do to satisfy your request is to add fixes
> branch that I push to that is pushed after it passes my tests (and not
> the urgent branch that is still being tested and may have bugs) and
> then have that be added to linux-next?

That is one option.

Alternatively do what I have been doing since ever: I have a couple of 
'for-N+1/topic' branches, and then I have one 'for-N/fixes' branch. All of 
them get automerged into 'for-next' branch which Stephen is pulling. Works 
like a charm.

-- 
Jiri Kosina
SUSE Labs

Re: linus-next: improving functional testing for to-be-merged pull requests

[Sasha Levin]

On Mon, Oct 21, 2024 at 11:48:34PM -0700, Christoph Hellwig wrote:
>On Mon, Oct 21, 2024 at 09:54:53PM -0700, Kees Cook wrote:
>> For example, for a given PR, the bot can report:
>>
>> - Were the patches CCed to a mailing list?
>> - A histogram of how long the patches were in next (to show bake times)
>> - Are any patches associated with test failures? (0day and many other
>> CIs are already running tests against -next; parse those reports)
>>
>> We could have a real pre-submit checker! :)
>
>That would be very useful.  Items 1 and 2 should be trivial, 3 would
>require a bit of work but would still be very useful.

If you've been following so far, there is a bot that is capable of doing
most of the above
(https://git.kernel.org/pub/scm/linux/kernel/git/sashal/next-analysis.git/).

Here's a histogram that describes v6.12-rc4..v6.12-rc5 as far as how
long commits spent in -next:

Days in linux-next:
----------------------------------------
  0 | +++++++++++++++++++++++++++++++++++++++++++++++++ (89)
<1 | +++++++++++ (21)
  1 | +++++++++++ (21)
  2 | +++++++++++++++++++++++++ (45)
  3 | ++++++++++++++ (25)
  4 | +++++ (10)
  5 |
  6 | + (2)
  7 |
  8 | + (3)
  9 | ++ (4)
10 |
11 | +++ (6)
12 |
13 |
14+| ++++++++ (15)

This is where I think the value of linus-next comes during the -rc
cycles: the (89 + 21) commits that haven't gone through the -next
workflow before being pulled. I'm not looking to delay the process and
add latency, I'm looking to plug a hole where code would flow directly
to Linus's tree bypassing -next.

With linus-next, we can at least squeeze in build tests as well as some
rudimentary testing if we get a few hours before Linus pulls (and we
usually do).

-- 
Thanks,
Sasha

Re: linus-next: improving functional testing for to-be-merged pull requests

[Thorsten Leemhuis]

On 28.10.24 23:46, Sasha Levin wrote:
> On Mon, Oct 21, 2024 at 11:48:34PM -0700, Christoph Hellwig wrote:
>> On Mon, Oct 21, 2024 at 09:54:53PM -0700, Kees Cook wrote:
>>> For example, for a given PR, the bot can report:
>>>
>>> - Were the patches CCed to a mailing list?
>>> - A histogram of how long the patches were in next (to show bake times)
>>> - Are any patches associated with test failures? (0day and many other
>>> CIs are already running tests against -next; parse those reports)
>>>
>>> We could have a real pre-submit checker! :)
>>
>> That would be very useful.  Items 1 and 2 should be trivial, 3 would
>> require a bit of work but would still be very useful.
> 
> If you've been following so far, there is a bot that is capable of doing
> most of the above
> (https://git.kernel.org/pub/scm/linux/kernel/git/sashal/next-
> analysis.git/).
> 
> Here's a histogram that describes v6.12-rc4..v6.12-rc5 as far as how
> long commits spent in -next:

I took a quick look at that tree and histo.sh that lead to a few
questions here the code had no obvious answers to (or I missed them due
to the "quick" aspect):

* How does histo.sh handle changes where the commit-id changed between
the first time in -next and their merge into Linus' tree (while the
patch itself did not change)? For example due to a rebase or workflows
where the commit-id changes regularly, such as those used by the
bluetooth tree (for -fixes, as it queues them in their -next branch
first) or the -mm tree (for most of it iirc -- this made things hard in
a script of mine that looks up the arrival in -next)?

* Do those lore scripts detect if a committer adjusted the subject of a
patch that has been on lore?

* How do the scripts handle patches that changed a lot while they were
in -next? I know of one subsystem that regularly drops whole patch-sets
from their trees included in -next to replace them with newer versions
of said patch-sets -- and then the timer maybe should restart.

> This is where I think the value of linus-next comes during the -rc
> cycles: the (89 + 21) commits that haven't gone through the -next
> workflow before being pulled. 
>
> I'm not looking to delay the process and
> add latency, I'm looking to plug a hole where code would flow directly
> to Linus's tree bypassing -next.

Overall after all the discussions in this thread I still fail to see why
we need a new tree for that. Why not make pending-fixes a bit more
prominent while motivating maintainers to have proper -fixes branches
included there?

Ciao, Thorsten

Re: linus-next: improving functional testing for to-be-merged pull requests

[Geert Uytterhoeven]

Hi Sasha,

On Mon, Oct 28, 2024 at 11:46 PM Sasha Levin <sashal@kernel.org> wrote:
> On Mon, Oct 21, 2024 at 11:48:34PM -0700, Christoph Hellwig wrote:
> >On Mon, Oct 21, 2024 at 09:54:53PM -0700, Kees Cook wrote:
> >> For example, for a given PR, the bot can report:
> >>
> >> - Were the patches CCed to a mailing list?
> >> - A histogram of how long the patches were in next (to show bake times)
> >> - Are any patches associated with test failures? (0day and many other
> >> CIs are already running tests against -next; parse those reports)
> >>
> >> We could have a real pre-submit checker! :)
> >
> >That would be very useful.  Items 1 and 2 should be trivial, 3 would
> >require a bit of work but would still be very useful.
>
> If you've been following so far, there is a bot that is capable of doing
> most of the above
> (https://git.kernel.org/pub/scm/linux/kernel/git/sashal/next-analysis.git/).
>
> Here's a histogram that describes v6.12-rc4..v6.12-rc5 as far as how
> long commits spent in -next:
>
> Days in linux-next:
> ----------------------------------------
>   0 | +++++++++++++++++++++++++++++++++++++++++++++++++ (89)
> <1 | +++++++++++ (21)
>   1 | +++++++++++ (21)
>   2 | +++++++++++++++++++++++++ (45)
>   3 | ++++++++++++++ (25)
>   4 | +++++ (10)
>   5 |
>   6 | + (2)
>   7 |
>   8 | + (3)
>   9 | ++ (4)
> 10 |
> 11 | +++ (6)
> 12 |
> 13 |
> 14+| ++++++++ (15)
>
> This is where I think the value of linus-next comes during the -rc
> cycles: the (89 + 21) commits that haven't gone through the -next
> workflow before being pulled. I'm not looking to delay the process and
> add latency, I'm looking to plug a hole where code would flow directly
> to Linus's tree bypassing -next.
>
> With linus-next, we can at least squeeze in build tests as well as some
> rudimentary testing if we get a few hours before Linus pulls (and we
> usually do).

[Oops, I misread "linus-next"-with-an-S in the last two paragraphs,
 and only noticed _after_ I wrote my comments below.  Anyway, I am
 still sending them as they (partly) explain the histogram's shape.]

Linux-next releases are not created automatically.
The last linux-next release happens on Friday (AU time), while Linus
creates the -rc on Sunday (US time), so most PRs sent during the second
half of the week would miss linux-next until Monday...

"Release early" still matters...

Gr{oetje,eeting}s,

                        Geert

-- 
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: linus-next: improving functional testing for to-be-merged pull requests

[Sasha Levin]

On Tue, Oct 29, 2024 at 09:10:25AM +0100, Thorsten Leemhuis wrote:
>On 28.10.24 23:46, Sasha Levin wrote:
>> On Mon, Oct 21, 2024 at 11:48:34PM -0700, Christoph Hellwig wrote:
>>> On Mon, Oct 21, 2024 at 09:54:53PM -0700, Kees Cook wrote:
>>>> For example, for a given PR, the bot can report:
>>>>
>>>> - Were the patches CCed to a mailing list?
>>>> - A histogram of how long the patches were in next (to show bake times)
>>>> - Are any patches associated with test failures? (0day and many other
>>>> CIs are already running tests against -next; parse those reports)
>>>>
>>>> We could have a real pre-submit checker! :)
>>>
>>> That would be very useful.  Items 1 and 2 should be trivial, 3 would
>>> require a bit of work but would still be very useful.
>>
>> If you've been following so far, there is a bot that is capable of doing
>> most of the above
>> (https://git.kernel.org/pub/scm/linux/kernel/git/sashal/next-
>> analysis.git/).
>>
>> Here's a histogram that describes v6.12-rc4..v6.12-rc5 as far as how
>> long commits spent in -next:
>
>I took a quick look at that tree and histo.sh that lead to a few
>questions here the code had no obvious answers to (or I missed them due
>to the "quick" aspect):
>
>* How does histo.sh handle changes where the commit-id changed between
>the first time in -next and their merge into Linus' tree (while the
>patch itself did not change)? For example due to a rebase or workflows
>where the commit-id changes regularly, such as those used by the
>bluetooth tree (for -fixes, as it queues them in their -next branch
>first) or the -mm tree (for most of it iirc -- this made things hard in
>a script of mine that looks up the arrival in -next)?

The "database" the scripts use stores 3 things:

  - commit ID
  - git patch-id
  - subject line

We try to match by either of the three. It means that maintainers can
rebase, change the subject, or even change the patch slightly, but as
long as one of the above stays the same we treat the commit the same.

>* Do those lore scripts detect if a committer adjusted the subject of a
>patch that has been on lore?

Yes, they also look up by patch-id, so if only the subject was adjusted
then we will still find the commit.

>* How do the scripts handle patches that changed a lot while they were
>in -next? I know of one subsystem that regularly drops whole patch-sets
>from their trees included in -next to replace them with newer versions
>of said patch-sets -- and then the timer maybe should restart.

The timer should just restart, right? If we uploaded patches that look
different from older ones, then their timer starts from 0 again.

>> This is where I think the value of linus-next comes during the -rc
>> cycles: the (89 + 21) commits that haven't gone through the -next
>> workflow before being pulled.
>>
>> I'm not looking to delay the process and
>> add latency, I'm looking to plug a hole where code would flow directly
>> to Linus's tree bypassing -next.
>
>Overall after all the discussions in this thread I still fail to see why
>we need a new tree for that. Why not make pending-fixes a bit more
>prominent while motivating maintainers to have proper -fixes branches
>included there?

Because that will add latency: my understanding that we don't want to
necessarily add another day or two between when fixes are ready and the
time it would take to get them through linux-next.

-- 
Thanks,
Sasha

Re: linus-next: improving functional testing for to-be-merged pull requests

[Thorsten Leemhuis]

On 29.10.24 12:30, Sasha Levin wrote:
> On Tue, Oct 29, 2024 at 09:10:25AM +0100, Thorsten Leemhuis wrote:
>> * How does histo.sh handle changes where the commit-id changed between
>> the first time in -next and their merge into Linus' tree  [...
> The "database" the scripts use stores 3 things:
> [...]

Ahh, many thx. And sorry, I should have taken a closer look at query.db,
that is pretty obvious now that I look at it again.

>>> This is where I think the value of linus-next comes during the -rc
>>> cycles: the (89 + 21) commits that haven't gone through the -next
>>> workflow before being pulled.
>>>
>>> I'm not looking to delay the process and
>>> add latency, I'm looking to plug a hole where code would flow directly
>>> to Linus's tree bypassing -next.
>>
>> Overall after all the discussions in this thread I still fail to see why
>> we need a new tree for that. Why not make pending-fixes a bit more
>> prominent while motivating maintainers to have proper -fixes branches
>> included there?
> 
> Because that will add latency: my understanding that we don't want to
> necessarily add another day or two between when fixes are ready and the
> time it would take to get them through linux-next.

Hmmm. After all those mails in this thread improving (and maybe even
separating & somewhat automating[1]) pending-fixes to me still sounds
like time better spend, as then more things could tested before they
even read a PR; but yes, I understand, the timing/order of merges can
mess things up, so testing on PR time has benefits, too.

Maybe I'm just biased, as I could need a better working pending-fixes
for regression tracking[2], as that allows me to ensure regression fixes
are on the right track (which usually is the current merge window and
not the next).

Thx again for the answer! Ciao, Thorsten

[1] yes, I known, there will be conflicts, so some human will need to be
involved

[2] subsystems without -fixes trees or workflows like the one Jiri
suggested recently elsewhere in this thread[3] break this :-/

[3]
https://lore.kernel.org/all/nycvar.YFH.7.76.2410252303270.20286@cbobk.fhfr.pm/

Re: linus-next: improving functional testing for to-be-merged pull requests

[Sasha Levin]

On Tue, Oct 29, 2024 at 01:46:23PM +0100, Thorsten Leemhuis wrote:
>Hmmm. After all those mails in this thread improving (and maybe even
>separating & somewhat automating[1]) pending-fixes to me still sounds
>like time better spend, as then more things could tested before they
>even read a PR; but yes, I understand, the timing/order of merges can
>mess things up, so testing on PR time has benefits, too.

Automating how? Having it be generated more often?

>Maybe I'm just biased, as I could need a better working pending-fixes
>for regression tracking[2], as that allows me to ensure regression fixes
>are on the right track (which usually is the current merge window and
>not the next).

I'd love to throw the linus-next away once I find it useless :)

-- 
Thanks,
Sasha

Re: linus-next: improving functional testing for to-be-merged pull requests

[Thorsten Leemhuis]

On 29.10.24 16:07, Sasha Levin wrote:
> On Tue, Oct 29, 2024 at 01:46:23PM +0100, Thorsten Leemhuis wrote:
>> Hmmm. After all those mails in this thread improving (and maybe even
>> separating & somewhat automating[1]) pending-fixes to me still sounds
>> like time better spend, as then more things could tested before they
>> even read a PR; but yes, I understand, the timing/order of merges can
>> mess things up, so testing on PR time has benefits, too.
> Automating how? Having it be generated more often?

Have the list of -fixes trees which a "no rebases" policy somewhere and
a script that regularly merges them into a tree. But as indicated, it's
not that easy in practice and can't be fully automated, as there will be
merge conflicts occasionally. But Linus wants to see them, so they will
happen at pull requests time, too -- doing it constantly has the benefit
that you can notice and resolve them ahead of time.

How much work this is: no idea, maybe Stephen could help answering that
from experiences for pending-fixes. But I expect conflicts should not
happen as often as they do when it comes to merging -for-next branches.

But that obviously only helps outside of merge windows.

Ciao, Thorsten

Re: linus-next: improving functional testing for to-be-merged pull requests

[Sasha Levin]

On Wed, Oct 30, 2024 at 07:46:40AM +0100, Thorsten Leemhuis wrote:
>On 29.10.24 16:07, Sasha Levin wrote:
>> On Tue, Oct 29, 2024 at 01:46:23PM +0100, Thorsten Leemhuis wrote:
>>> Hmmm. After all those mails in this thread improving (and maybe even
>>> separating & somewhat automating[1]) pending-fixes to me still sounds
>>> like time better spend, as then more things could tested before they
>>> even read a PR; but yes, I understand, the timing/order of merges can
>>> mess things up, so testing on PR time has benefits, too.
>> Automating how? Having it be generated more often?
>
>Have the list of -fixes trees which a "no rebases" policy somewhere and
>a script that regularly merges them into a tree. But as indicated, it's
>not that easy in practice and can't be fully automated, as there will be
>merge conflicts occasionally. But Linus wants to see them, so they will
>happen at pull requests time, too -- doing it constantly has the benefit
>that you can notice and resolve them ahead of time.
>
>How much work this is: no idea, maybe Stephen could help answering that
>from experiences for pending-fixes. But I expect conflicts should not
>happen as often as they do when it comes to merging -for-next branches.
>
>But that obviously only helps outside of merge windows.

Do you want to give it ago?

I've pushed something to
https://git.kernel.org/pub/scm/linux/kernel/git/sashal/linus-next.git/log/?h=auto-pending-fixes
, and scripted a bit around it to try and keep it updated.

We can try a model where it tries to avoid rebases as much as it can,
and if it needs to rebase it will tag the old HEAD before re-creating
the branch?

-- 
Thanks,
Sasha

Re: linus-next: improving functional testing for to-be-merged pull requests

[Paul E. McKenney]

On Mon, Oct 28, 2024 at 06:46:19PM -0400, Sasha Levin wrote:
> On Mon, Oct 21, 2024 at 11:48:34PM -0700, Christoph Hellwig wrote:
> > On Mon, Oct 21, 2024 at 09:54:53PM -0700, Kees Cook wrote:
> > > For example, for a given PR, the bot can report:
> > > 
> > > - Were the patches CCed to a mailing list?
> > > - A histogram of how long the patches were in next (to show bake times)
> > > - Are any patches associated with test failures? (0day and many other
> > > CIs are already running tests against -next; parse those reports)
> > > 
> > > We could have a real pre-submit checker! :)
> > 
> > That would be very useful.  Items 1 and 2 should be trivial, 3 would
> > require a bit of work but would still be very useful.
> 
> If you've been following so far, there is a bot that is capable of doing
> most of the above
> (https://git.kernel.org/pub/scm/linux/kernel/git/sashal/next-analysis.git/).

Nice!!!

What does this make of these commits of mine?

744e87210b1ae rcu: Finer-grained grace-period-end checks in rcu_dump_cpu_stacks()
cbe644aa6fe17 rcu: Stop stall warning from dumping stacks if grace period ends
26ff1fb02991e rcu: Delete unused rcu_gp_might_be_stalled() function

> Here's a histogram that describes v6.12-rc4..v6.12-rc5 as far as how
> long commits spent in -next:
> 
> Days in linux-next:
> ----------------------------------------
>  0 | +++++++++++++++++++++++++++++++++++++++++++++++++ (89)
> <1 | +++++++++++ (21)
>  1 | +++++++++++ (21)
>  2 | +++++++++++++++++++++++++ (45)
>  3 | ++++++++++++++ (25)
>  4 | +++++ (10)
>  5 |
>  6 | + (2)
>  7 |
>  8 | + (3)
>  9 | ++ (4)
> 10 |
> 11 | +++ (6)
> 12 |
> 13 |
> 14+| ++++++++ (15)
> 
> This is where I think the value of linus-next comes during the -rc
> cycles: the (89 + 21) commits that haven't gone through the -next
> workflow before being pulled. I'm not looking to delay the process and
> add latency, I'm looking to plug a hole where code would flow directly
> to Linus's tree bypassing -next.
> 
> With linus-next, we can at least squeeze in build tests as well as some
> rudimentary testing if we get a few hours before Linus pulls (and we
> usually do).

OK, if linus-next is only active just before the merge window opens,
then I have fewer testing-bandwidth concerns.

							Thanx, Paul

Re: linus-next: improving functional testing for to-be-merged pull requests

[Sasha Levin]

On Wed, Oct 30, 2024 at 10:08:57AM -0700, Paul E. McKenney wrote:
>On Mon, Oct 28, 2024 at 06:46:19PM -0400, Sasha Levin wrote:
>> On Mon, Oct 21, 2024 at 11:48:34PM -0700, Christoph Hellwig wrote:
>> > On Mon, Oct 21, 2024 at 09:54:53PM -0700, Kees Cook wrote:
>> > > For example, for a given PR, the bot can report:
>> > >
>> > > - Were the patches CCed to a mailing list?
>> > > - A histogram of how long the patches were in next (to show bake times)
>> > > - Are any patches associated with test failures? (0day and many other
>> > > CIs are already running tests against -next; parse those reports)
>> > >
>> > > We could have a real pre-submit checker! :)
>> >
>> > That would be very useful.  Items 1 and 2 should be trivial, 3 would
>> > require a bit of work but would still be very useful.
>>
>> If you've been following so far, there is a bot that is capable of doing
>> most of the above
>> (https://git.kernel.org/pub/scm/linux/kernel/git/sashal/next-analysis.git/).
>
>Nice!!!
>
>What does this make of these commits of mine?
>
>744e87210b1ae rcu: Finer-grained grace-period-end checks in rcu_dump_cpu_stacks()
>cbe644aa6fe17 rcu: Stop stall warning from dumping stacks if grace period ends
>26ff1fb02991e rcu: Delete unused rcu_gp_might_be_stalled() function

Days in linux-next:
----------------------------------------
  1 |
  2 |
  3 |
  4 |
  5 |
  6 | +++ (3)

-- 
Thanks,
Sasha

Re: linus-next: improving functional testing for to-be-merged pull requests

[Paul E. McKenney]

On Wed, Oct 30, 2024 at 01:15:39PM -0400, Sasha Levin wrote:
> On Wed, Oct 30, 2024 at 10:08:57AM -0700, Paul E. McKenney wrote:
> > On Mon, Oct 28, 2024 at 06:46:19PM -0400, Sasha Levin wrote:
> > > On Mon, Oct 21, 2024 at 11:48:34PM -0700, Christoph Hellwig wrote:
> > > > On Mon, Oct 21, 2024 at 09:54:53PM -0700, Kees Cook wrote:
> > > > > For example, for a given PR, the bot can report:
> > > > >
> > > > > - Were the patches CCed to a mailing list?
> > > > > - A histogram of how long the patches were in next (to show bake times)
> > > > > - Are any patches associated with test failures? (0day and many other
> > > > > CIs are already running tests against -next; parse those reports)
> > > > >
> > > > > We could have a real pre-submit checker! :)
> > > >
> > > > That would be very useful.  Items 1 and 2 should be trivial, 3 would
> > > > require a bit of work but would still be very useful.
> > > 
> > > If you've been following so far, there is a bot that is capable of doing
> > > most of the above
> > > (https://git.kernel.org/pub/scm/linux/kernel/git/sashal/next-analysis.git/).
> > 
> > Nice!!!
> > 
> > What does this make of these commits of mine?
> > 
> > 744e87210b1ae rcu: Finer-grained grace-period-end checks in rcu_dump_cpu_stacks()
> > cbe644aa6fe17 rcu: Stop stall warning from dumping stacks if grace period ends
> > 26ff1fb02991e rcu: Delete unused rcu_gp_might_be_stalled() function
> 
> Days in linux-next:
> ----------------------------------------
>  1 |
>  2 |
>  3 |
>  4 |
>  5 |
>  6 | +++ (3)

Very good!

Any indications of errors for any of them?

							Thanx, Paul

Re: linus-next: improving functional testing for to-be-merged pull requests

[Thorsten Leemhuis]

On 30.10.24 15:10, Sasha Levin wrote:
> On Wed, Oct 30, 2024 at 07:46:40AM +0100, Thorsten Leemhuis wrote:
>> On 29.10.24 16:07, Sasha Levin wrote:
>>> On Tue, Oct 29, 2024 at 01:46:23PM +0100, Thorsten Leemhuis wrote:
>>>> Hmmm. After all those mails in this thread improving (and maybe even
>>>> separating & somewhat automating[1]) pending-fixes to me still sounds
>>>> like time better spend, as then more things could tested before they
>>>> even read a PR; but yes, I understand, the timing/order of merges can
>>>> mess things up, so testing on PR time has benefits, too.
>>> Automating how? Having it be generated more often?
>>
>> Have the list of -fixes trees which a "no rebases" policy somewhere and
>> a script that regularly merges them into a tree. But as indicated, it's
>> not that easy in practice and can't be fully automated, as there will be
>> merge conflicts occasionally. But Linus wants to see them, so they will
>> happen at pull requests time, too -- doing it constantly has the benefit
>> that you can notice and resolve them ahead of time.
>>
>> How much work this is: no idea, maybe Stephen could help answering that
>> from experiences for pending-fixes. But I expect conflicts should not
>> happen as often as they do when it comes to merging -for-next branches.
>>
>> But that obviously only helps outside of merge windows.
> 
> Do you want to give it ago?
> 
> I've pushed something to
> https://git.kernel.org/pub/scm/linux/kernel/git/sashal/linus-next.git/
> log/?h=auto-pending-fixes
> , and scripted a bit around it to try and keep it updated.

Cool!

> We can try a model where it tries to avoid rebases as much as it can,
> and if it needs to rebase it will tag the old HEAD before re-creating
> the branch?

So here is the thing: with regression tracking I'm already having a huge
never ending and hard task that creates a lot of work every day -- work
I'm not getting payed for now since months with no concrete funding
solution yet on the horizon. So I don't want to put even more work on my
shoulders with helping in other areas currently, sorry.

That being said: I'm willing to help with creating improvements for our
docs (something along the lines of "you should have a -fixes branches
that is never rebased that is included in the -pending-fixes branch of
-next and -auto-pending-fixes") and even trying to talk maintainers into
that do not yet have such a branch into maintaining one.

Ciao, Thorsten

Re: linus-next: improving functional testing for to-be-merged pull requests

[Shuah Khan]

On 10/23/24 03:19, Steven Rostedt wrote:
> On Wed, 23 Oct 2024 10:36:20 +0200
> Geert Uytterhoeven <geert@linux-m68k.org> wrote:
> 
>>> To put it this way. The bugs I'm fixing was for code in linux-next
>>> where the bugs were never found. They only appeared when they went into
>>> Linus's tree. So why put the fixes in linux-next, if it didn't catch
>>> the bugs I fixed in the first place?
>>
>> Hmmm...
>>
>> Your arguments sound very similar to those being used in recent
>> discussions about not posting patches for public review...
>>
>> Please follow the process! ;-)
> 
> What process?
> 
> Note, I probably post everything to mailing lists more than anyone
> else (besides stable). All my commits come from mailing lists. Even
> things I change myself. I always send out the change to a list. Then I
> use patchwork to pull it into my tree.
> 
> After the changes are tested, I send out the patches *again* with my
> [for-next] tags in the subject. If it's a fix for Linus, it goes out as
> a "[for-linus]" tag. These emails automatically update my patchwork
> status.
> 
> No change goes into Linus's tree from me that hasn't been sent out
> publicly.
> 
> But pushing to linux-next for a day or two, what does that give me?
> 

I don't send pull requests without soaking it in next for 2 to 3 days.
It helps me find problems related to things something breaking on an
architecture and/or config.

I find next very helpful in finding merge conflicts and build/test
breakages.

thanks,
-- Shuah

Re: linus-next: improving functional testing for to-be-merged pull requests

[Steven Rostedt]

On Thu, 31 Oct 2024 13:08:12 -0600
Shuah Khan <skhan@linuxfoundation.org> wrote:

> > But pushing to linux-next for a day or two, what does that give me?
> >   
> 
> I don't send pull requests without soaking it in next for 2 to 3 days.
> It helps me find problems related to things something breaking on an
> architecture and/or config.
> 
> I find next very helpful in finding merge conflicts and build/test
> breakages.

I have a 13 hour test suite that I run before pushing anything. And since
it's going into the rc release, any merge conflict that happens in next is
likely something that isn't even applicable for Linus's tree. Sure, it will
need to be addressed in the next merge window, but that would have been
found after getting in Linus's tree anyway, because really a fix to Linus's
tree should have no merge conflicts.

-- Steve

Re: linus-next: improving functional testing for to-be-merged pull requests

[Shuah Khan]

On 10/22/24 03:10, Thorsten Leemhuis wrote:
> On 21.10.24 23:41, Mark Brown wrote:
>> On Mon, Oct 21, 2024 at 12:07:13PM -0400, Sasha Levin wrote:
>>
>>> In an attempt to address the concerns, we're trying out a new "linus-next"
>>> tree is being created and maintained with the following characteristics:
> 
> BTW, in case anyone cares: I fully agree with what Kees wrote earlier
> today elsewhere in this thread, e.g. things like "improve -next instead"
> and "pre-merge bot":
> http://lore.kernel.org/all/792F4759-EA33-48B8-9AD0-FA14FA69E86E%40kernel.org\

I am catching up on this thread. I agree with Kees and others on improve
next instead. Adding one more next-like-thing will add to the confusion.

> 
> Regarding that bot: a few of the CI folks and a developer or two told me
> they want regzbot to react to PRs for Linus as well, so it can send
> mails like "hey Linus, just so you know, this PR contains changes that
> cause the following regressions not yet fixed". I think I like the idea,
> but well, quite a few other improvements around regzbot and its use have
> a much higher priority currently.
> 
>>> 	4. Continuous tree (not daily tags like in linux-next),
>>> 	facilitating easier bisection
>>
>> Is this a pressing problem?  I routinely bisect -next, you have to base
>> things on Linus' tree (or pending-fixes) but otherwise it's not
>> especially problematic.
> 
> I wonder if part of this is a "don't know how to do that" aka "lack of
> documentation" problem. I've recently seen some good guide or mailing
> list post how to bisect -next somewhere, but I think it wasn't in our
> Documentation/ directory. I need to search where that was (Mark, I might
> misremember, but wasn't it you who posted it somewhere?) and could work
> towards upstreaming that or some other guide. And don't worry, due to
> the different target audience it would be much shorter text than other
> documents I contributed. ;-)
> 

Documentation could help. Tailoring the workflow to linux-next could help.
All my branches are in linux-next. I don't send my PRs without keeping
the content in linux-next for 3 days to week after rc1 - it is usually
longer for merge-window content.

This helps me find conflicts if any between all the tree selftest go
through and mine. It helps work out conflicts in linux-next prior to
sending pull request.

My vote is for improving next instead of re-inventing next-like-thing.

thanks,
-- Shuah

Re: Re: linus-next: improving functional testing for to-be-merged pull requests

[Joel Granados]

On Mon, Oct 28, 2024 at 06:46:19PM -0400, Sasha Levin wrote:
> On Mon, Oct 21, 2024 at 11:48:34PM -0700, Christoph Hellwig wrote:
> >On Mon, Oct 21, 2024 at 09:54:53PM -0700, Kees Cook wrote:
> >> For example, for a given PR, the bot can report:
> >>
> >> - Were the patches CCed to a mailing list?
> >> - A histogram of how long the patches were in next (to show bake times)
> >> - Are any patches associated with test failures? (0day and many other
> >> CIs are already running tests against -next; parse those reports)
> >>
> >> We could have a real pre-submit checker! :)
> >
> >That would be very useful.  Items 1 and 2 should be trivial, 3 would
> >require a bit of work but would still be very useful.
> 
> If you've been following so far, there is a bot that is capable of doing
> most of the above
> (https://git.kernel.org/pub/scm/linux/kernel/git/sashal/next-analysis.git/).
> 
> Here's a histogram that describes v6.12-rc4..v6.12-rc5 as far as how
> long commits spent in -next:
> 
> Days in linux-next:
> ----------------------------------------
>   0 | +++++++++++++++++++++++++++++++++++++++++++++++++ (89)
> <1 | +++++++++++ (21)
>   1 | +++++++++++ (21)
>   2 | +++++++++++++++++++++++++ (45)
>   3 | ++++++++++++++ (25)
>   4 | +++++ (10)
>   5 |
>   6 | + (2)
>   7 |
>   8 | + (3)
>   9 | ++ (4)
> 10 |
> 11 | +++ (6)
> 12 |
> 13 |
> 14+| ++++++++ (15)
This looks super nice. Sometimes I need to answer how long a
commit/series has been in next to either take it out of the PR or to at
least have a comment to Linus.
I see that I can use the script like `histo.sh /PATH/TO/DB COMMIT-ID`,
which is exactly what I would expect.

Is the idea to run the scripts from
https://git.kernel.org/pub/scm/linux/kernel/git/sashal/next-analysis.git
or to populate ones own DB in linux-next and then run histo?

thx

Best

-- 

Joel Granados

Re: Re: linus-next: improving functional testing for to-be-merged pull requests

[Sasha Levin]

On Mon, Nov 04, 2024 at 09:49:15AM +0100, Joel Granados wrote:
>On Mon, Oct 28, 2024 at 06:46:19PM -0400, Sasha Levin wrote:
>> On Mon, Oct 21, 2024 at 11:48:34PM -0700, Christoph Hellwig wrote:
>> >On Mon, Oct 21, 2024 at 09:54:53PM -0700, Kees Cook wrote:
>> >> For example, for a given PR, the bot can report:
>> >>
>> >> - Were the patches CCed to a mailing list?
>> >> - A histogram of how long the patches were in next (to show bake times)
>> >> - Are any patches associated with test failures? (0day and many other
>> >> CIs are already running tests against -next; parse those reports)
>> >>
>> >> We could have a real pre-submit checker! :)
>> >
>> >That would be very useful.  Items 1 and 2 should be trivial, 3 would
>> >require a bit of work but would still be very useful.
>>
>> If you've been following so far, there is a bot that is capable of doing
>> most of the above
>> (https://git.kernel.org/pub/scm/linux/kernel/git/sashal/next-analysis.git/).
>>
>> Here's a histogram that describes v6.12-rc4..v6.12-rc5 as far as how
>> long commits spent in -next:
>>
>> Days in linux-next:
>> ----------------------------------------
>>   0 | +++++++++++++++++++++++++++++++++++++++++++++++++ (89)
>> <1 | +++++++++++ (21)
>>   1 | +++++++++++ (21)
>>   2 | +++++++++++++++++++++++++ (45)
>>   3 | ++++++++++++++ (25)
>>   4 | +++++ (10)
>>   5 |
>>   6 | + (2)
>>   7 |
>>   8 | + (3)
>>   9 | ++ (4)
>> 10 |
>> 11 | +++ (6)
>> 12 |
>> 13 |
>> 14+| ++++++++ (15)
>This looks super nice. Sometimes I need to answer how long a
>commit/series has been in next to either take it out of the PR or to at
>least have a comment to Linus.
>I see that I can use the script like `histo.sh /PATH/TO/DB COMMIT-ID`,
>which is exactly what I would expect.

Thanks!

>Is the idea to run the scripts from
>https://git.kernel.org/pub/scm/linux/kernel/git/sashal/next-analysis.git
>or to populate ones own DB in linux-next and then run histo?

You can use the DB in the git repo, but prep.sh is provided in case you
want to create your own or if I stop updating the repo for some reason.

-- 
Thanks,
Sasha

Re: Re: linus-next: improving functional testing for to-be-merged pull requests

[Joel Granados]

On Mon, Oct 28, 2024 at 06:46:19PM -0400, Sasha Levin wrote:
> On Mon, Oct 21, 2024 at 11:48:34PM -0700, Christoph Hellwig wrote:
> >On Mon, Oct 21, 2024 at 09:54:53PM -0700, Kees Cook wrote:
> >> For example, for a given PR, the bot can report:
> >>
> >> - Were the patches CCed to a mailing list?
> >> - A histogram of how long the patches were in next (to show bake times)
> >> - Are any patches associated with test failures? (0day and many other
> >> CIs are already running tests against -next; parse those reports)
> >>
> >> We could have a real pre-submit checker! :)
> >
> >That would be very useful.  Items 1 and 2 should be trivial, 3 would
> >require a bit of work but would still be very useful.
> 
> If you've been following so far, there is a bot that is capable of doing
> most of the above
> (https://git.kernel.org/pub/scm/linux/kernel/git/sashal/next-analysis.git/).
> 
> Here's a histogram that describes v6.12-rc4..v6.12-rc5 as far as how
> long commits spent in -next:
> 
> Days in linux-next:
> ----------------------------------------
>   0 | +++++++++++++++++++++++++++++++++++++++++++++++++ (89)
> <1 | +++++++++++ (21)
>   1 | +++++++++++ (21)
>   2 | +++++++++++++++++++++++++ (45)
>   3 | ++++++++++++++ (25)
>   4 | +++++ (10)
>   5 |
>   6 | + (2)
>   7 |
>   8 | + (3)
>   9 | ++ (4)
> 10 |
> 11 | +++ (6)
> 12 |
> 13 |
> 14+| ++++++++ (15)

I kept working on this for this release and ended up with something that answers
the question how much time has my PR been in linux-next. Not sure if it is
useful to anyone, but decided to share it just in case.

# The command
``` bash
./pr-hist.sh -d ./db --linux-next ~/src/linux-next --pr /tmp/pull_request.txt
```
* here [1] are the script sources
* pull_request.txt is a file that contains a PR created with the `git
  request-pull` command.

# The output

``` bash
                         Histogram of Counts by Date
120 +-------------------------------------------------------------------+
    |                                                     * *           |
100 |                                                     * *           |
    |                                                  ** * *           |
 80 |                                                * ** * *           |
 60 |                                        * *     * ** * *           |
    |                                    * * * *     * ** * *           |
 40 |                        * * *     * * * * *     * ** * *           |
    |           * * *    * * * * *     * * * * *     * ** * *           |
 20 |           * * *    * * * * *     * * * * *     * ** * *           |
    |           * * *    * * * * *     * * * * *     * ** * *           |
  0 +-------------------------------------------------------------------+
  1017          1024         1031          1107         1114          1121
                                    Date
```

# Command help
``` bash
  Usage: pr-hist.sh ARGS [OPTIONS]

  Lists the databases where the <commit_ids> are found

  Args: [-c <list> | -p <pr>] -d <db_dir>
    -c, --commit-ids <list>       Comma separated list of commits to query
    -p, --pr <pr>                 Pull request file (generated by request-pull)
    -d, --db <db_dir>             Database directory

  Options:
    -n, --linux-next <next_dir>   The linux-next directory
    -h, --help                    Display this help
```

Best

[1] https://git.kernel.org/pub/scm/linux/kernel/git/joel.granados/next-analysis.git/tree/pr-hist.sh?h=jag/master

-- 

Joel Granados