[Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Theodore Y. Ts'o]

Hi all,

Many of you attended Dmitry Vyukov's talk at the Kernel Summit track
today, "Reflections on Kernel Development Process, Quality, and
Testing".  (For those of you who haven't, the slides are available
here[1].)

[1] https://linuxplumbersconf.org/event/4/contributions/554/attachments/353/584/Reflections__Kernel_Summit_2019.pdf

Greg K-H has suggested, and I strongly agree, that it would be
worthwhile to add this to the agenda of the Maintainer's Summit.  In
particular, what next steps should we take and what should be the
criteria and the process for trying to further standardize our tools
and processes in order to make make our development processes more
mature and to improve developer productivity and happiness.

If you didn't attend the talk, I encourage you to take a look at the
slide, so we don't have to spend time trying to bring people up to
speed on the discussion to date.  My plan is to schedule this as our
first topic tomorrow afternoon.

Thanks,

						- Ted
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

[Ksummit-discuss] Video of Dmitry's Kernel Summit talk (Was: Reflections on kernel development processes)

[Theodore Y. Ts'o]

On Wed, Sep 11, 2019 at 11:08:04AM -0400, Theodore Y. Ts'o wrote:
> Hi all,
> 
> Many of you attended Dmitry Vyukov's talk at the Kernel Summit track
> today, "Reflections on Kernel Development Process, Quality, and
> Testing".  (For those of you who haven't, the slides are available
> here[1].)
> 
> [1] https://linuxplumbersconf.org/event/4/contributions/554/attachments/353/584/Reflections__Kernel_Summit_2019.pdf
> 
> Greg K-H has suggested, and I strongly agree, that it would be
> worthwhile to add this to the agenda of the Maintainer's Summit.  In
> particular, what next steps should we take and what should be the
> criteria and the process for trying to further standardize our tools
> and processes in order to make make our development processes more
> mature and to improve developer productivity and happiness.

The raw video is (temporarily) available at:

	https://youtu.be/a2Nv-KJyqPk?t=5239

This will disappear when we can get something which is properly
cropped and edited --- this is basically the "hot wash" video which
was made available this morning, and after I failed my saving throw
against Linux video editing tools' user interfaces, I just uploaded it
straight to YouTube.

						- Ted
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Konstantin Ryabitsev]

On Wed, Sep 11, 2019 at 11:08:04AM -0400, Theodore Y. Ts'o wrote:
> Hi all,
> 
> Many of you attended Dmitry Vyukov's talk at the Kernel Summit track
> today, "Reflections on Kernel Development Process, Quality, and
> Testing".  (For those of you who haven't, the slides are available
> here[1].)
> 
> [1] https://linuxplumbersconf.org/event/4/contributions/554/attachments/353/584/Reflections__Kernel_Summit_2019.pdf
> 
> Greg K-H has suggested, and I strongly agree, that it would be
> worthwhile to add this to the agenda of the Maintainer's Summit.  In
> particular, what next steps should we take and what should be the
> criteria and the process for trying to further standardize our tools
> and processes in order to make make our development processes more
> mature and to improve developer productivity and happiness.
> 
> If you didn't attend the talk, I encourage you to take a look at the
> slide, so we don't have to spend time trying to bring people up to
> speed on the discussion to date.  My plan is to schedule this as our
> first topic tomorrow afternoon.

To follow-up, this is a very rough outline of a proposal that I am going
to submit to the Foundation in hopes to fund maintainer tool
development. It follows along some of the lines highlighted in Dmitry's
talk.

--------

# Stage 1 (Normal brain): "local patchwork"

- Implement a mutt-like tool ("putt"?) that uses locally cloned
  public-inbox archives to track patches/series submitted to mailing
  lists
    - Pre-filters by keywords and paths in patches
    - Tracks and automatically inserts taglines
      (Reviewed-by, Acked-by, Tested-by)
    - Can ignore a patch/series until it sees certain taglines
      (Tested-by: zeroday bot, Reviewed-by: Trusty Intern)
    - Automatically tracks latest series and offers an interdiff view
      between series revisions ("show me what changed between v1 and v2")
    - Allows responding to patches and conversations a-la mutt
    - Allows applying patches/series to local repos


# Stage 2 (Enlightened brain): "now with CI and workflows"

- Add configurable workflow functionality allowing maintainers to run
  local or remote tasks on patches and series, before maintainer sees
  the patches, e.g.:
    - Create a branch and attempt to apply series
    - If succeeds, run a batch of CI tests
    - If succeeds, mark as "CI passed" and show the maintainer
    - If fails, reject automatically using a "sorry, tests failed"
      template, including relevant error messages

- All of the above runs outside of the UI tool ("putt-cid"?) and defines CI
  routines that can run in cloudy environments or locally using
  containers.
- Putt communicates with putt-cid locally or remotely to identify
  patches/series that the maintainer should review


# Stage 3 (Galaxy brain): "email as a secondary channel"

- Support additional distributed communication mechanisms in conjunction
  with existing mailing lists.
  - SSB is a peer-to-peer replication framework that has built-in
    cryptographic integrity and attestation ("immutable git-like
    chains per participating developer")
    - offers native support for structured data like bug reports, CI
      results, code review comments, etc.
    - can easily support email-to-SSB and web-to-SSB bridges, so
      developers can choose to participate using familiar tools
    - has known limitations in v1 of the protocol, but v2 is being
      actively developed to address them.
    - or we can take it as a base and develop an SSB-like protocol that
      better suits distributed development needs.

  - Radicle is another interesting alternative that creates a mechanism
    for automating some maintainer tasks by defining "state machines,"
    e.g.:
    - automatically merge a revision if all tests pass and at least 2
      Reviewed-by's are seen.
    - May have been sipping the blockchain cool-aid a bit too much
      ("Immutable append-only records").
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Rob Herring]

On Thu, Sep 12, 2019 at 1:06 PM Konstantin Ryabitsev
<konstantin@linuxfoundation.org> wrote:
>
> On Wed, Sep 11, 2019 at 11:08:04AM -0400, Theodore Y. Ts'o wrote:
> > Hi all,
> >
> > Many of you attended Dmitry Vyukov's talk at the Kernel Summit track
> > today, "Reflections on Kernel Development Process, Quality, and
> > Testing".  (For those of you who haven't, the slides are available
> > here[1].)
> >
> > [1] https://linuxplumbersconf.org/event/4/contributions/554/attachments/353/584/Reflections__Kernel_Summit_2019.pdf
> >
> > Greg K-H has suggested, and I strongly agree, that it would be
> > worthwhile to add this to the agenda of the Maintainer's Summit.  In
> > particular, what next steps should we take and what should be the
> > criteria and the process for trying to further standardize our tools
> > and processes in order to make make our development processes more
> > mature and to improve developer productivity and happiness.
> >
> > If you didn't attend the talk, I encourage you to take a look at the
> > slide, so we don't have to spend time trying to bring people up to
> > speed on the discussion to date.  My plan is to schedule this as our
> > first topic tomorrow afternoon.
>
> To follow-up, this is a very rough outline of a proposal that I am going
> to submit to the Foundation in hopes to fund maintainer tool
> development. It follows along some of the lines highlighted in Dmitry's
> talk.

As long as the submitter side is a free-for-all, that's going to make
the maintainer tools more complex to implement and less reliable.
Seems like submitter tools should be included here too. Specifically,
the steps between having a git branch ready for review and
git-send-email are a lot of manual steps. Though maybe a lot of that
is just all of us agreeing on more standardized requirements as is
being discussed in 'Maintainer Entry Profiles'.

> --------
>
> # Stage 1 (Normal brain): "local patchwork"
>
> - Implement a mutt-like tool ("putt"?) that uses locally cloned
>   public-inbox archives to track patches/series submitted to mailing
>   lists
>     - Pre-filters by keywords and paths in patches
>     - Tracks and automatically inserts taglines
>       (Reviewed-by, Acked-by, Tested-by)

A frequent problem I see is submitters failing to add tags on
subsequent versions. Either the tool can check older versions and
apply those tags too or provide an auto reply to the submitter. Either
way, if we have an interdiff, it should be easy to distinguish cases
of the submitter forgetting vs. patch changed and we need to re-review
it.

>     - Can ignore a patch/series until it sees certain taglines
>       (Tested-by: zeroday bot, Reviewed-by: Trusty Intern)

There's also ignore after seeing a tag. IOW, if I've acked something I
want to ignore it.

>     - Automatically tracks latest series and offers an interdiff view
>       between series revisions ("show me what changed between v1 and v2")
>     - Allows responding to patches and conversations a-la mutt
>     - Allows applying patches/series to local repos
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Laurent Pinchart]

Hi Konstantin,

On Thu, Sep 12, 2019 at 08:06:02AM -0400, Konstantin Ryabitsev wrote:
> On Wed, Sep 11, 2019 at 11:08:04AM -0400, Theodore Y. Ts'o wrote:
> > Hi all,
> > 
> > Many of you attended Dmitry Vyukov's talk at the Kernel Summit track
> > today, "Reflections on Kernel Development Process, Quality, and
> > Testing".  (For those of you who haven't, the slides are available
> > here[1].)
> > 
> > [1] https://linuxplumbersconf.org/event/4/contributions/554/attachments/353/584/Reflections__Kernel_Summit_2019.pdf
> > 
> > Greg K-H has suggested, and I strongly agree, that it would be
> > worthwhile to add this to the agenda of the Maintainer's Summit.  In
> > particular, what next steps should we take and what should be the
> > criteria and the process for trying to further standardize our tools
> > and processes in order to make make our development processes more
> > mature and to improve developer productivity and happiness.
> > 
> > If you didn't attend the talk, I encourage you to take a look at the
> > slide, so we don't have to spend time trying to bring people up to
> > speed on the discussion to date.  My plan is to schedule this as our
> > first topic tomorrow afternoon.
> 
> To follow-up, this is a very rough outline of a proposal that I am going
> to submit to the Foundation in hopes to fund maintainer tool
> development. It follows along some of the lines highlighted in Dmitry's
> talk.
> 
> --------
> 
> # Stage 1 (Normal brain): "local patchwork"
> 
> - Implement a mutt-like tool ("putt"?) that uses locally cloned
>   public-inbox archives to track patches/series submitted to mailing
>   lists
>     - Pre-filters by keywords and paths in patches
>     - Tracks and automatically inserts taglines
>       (Reviewed-by, Acked-by, Tested-by)
>     - Can ignore a patch/series until it sees certain taglines
>       (Tested-by: zeroday bot, Reviewed-by: Trusty Intern)
>     - Automatically tracks latest series and offers an interdiff view
>       between series revisions ("show me what changed between v1 and v2")
>     - Allows responding to patches and conversations a-la mutt
>     - Allows applying patches/series to local repos

Do you plan for this tool to support shallow clones ? Some mailing lists
have really high traffic an have been around for years, one may not want
to clone a full public-inbox archives when interested in patches
submitted for the last N months only.

> # Stage 2 (Enlightened brain): "now with CI and workflows"
> 
> - Add configurable workflow functionality allowing maintainers to run
>   local or remote tasks on patches and series, before maintainer sees
>   the patches, e.g.:
>     - Create a branch and attempt to apply series
>     - If succeeds, run a batch of CI tests
>     - If succeeds, mark as "CI passed" and show the maintainer
>     - If fails, reject automatically using a "sorry, tests failed"
>       template, including relevant error messages
> 
> - All of the above runs outside of the UI tool ("putt-cid"?) and defines CI
>   routines that can run in cloudy environments or locally using
>   containers.
> - Putt communicates with putt-cid locally or remotely to identify
>   patches/series that the maintainer should review
> 
> 
> # Stage 3 (Galaxy brain): "email as a secondary channel"
> 
> - Support additional distributed communication mechanisms in conjunction
>   with existing mailing lists.
>   - SSB is a peer-to-peer replication framework that has built-in
>     cryptographic integrity and attestation ("immutable git-like
>     chains per participating developer")
>     - offers native support for structured data like bug reports, CI
>       results, code review comments, etc.
>     - can easily support email-to-SSB and web-to-SSB bridges, so
>       developers can choose to participate using familiar tools
>     - has known limitations in v1 of the protocol, but v2 is being
>       actively developed to address them.
>     - or we can take it as a base and develop an SSB-like protocol that
>       better suits distributed development needs.
> 
>   - Radicle is another interesting alternative that creates a mechanism
>     for automating some maintainer tasks by defining "state machines,"
>     e.g.:
>     - automatically merge a revision if all tests pass and at least 2
>       Reviewed-by's are seen.
>     - May have been sipping the blockchain cool-aid a bit too much
>       ("Immutable append-only records").

-- 
Regards,

Laurent Pinchart
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Daniel Borkmann]

On 9/22/19 2:02 PM, Dmitry Vyukov wrote:
[...]
> Also adding people from the "Kernel development collaboration platform
> wish list" discussion on the workflows list [1].
> (Rafael et al, thanks for collecting the requirements, that's very useful!)
> 
> I second the idea expressed by several people that addressing the
> contributor side is a very important part of this effort.
> 
> While I understand the intention to provide something useful as fast
> as possible, I also a bit afraid that the Stage 1 ("putt") diverges
> us into investing into particular UI, tying capabilities with this UI
> and not addressing the fundamental problems.
> People are expressing a preference for different types of UIs
> (CL, terminal GUI, web, scripting), I think eventually we will have
> several. So I would suggest to untie features/capabilities from
> any particular UI as much as possible, and start with addressing more
> fundamental aspects. Building richer features on top of the current
> human-oriented emails is also going to be much harder, and that's the
> work that we eventually will (hopefully) throw away.
> 
>  From UI perspective I think we should start with a CL interface because
> (1) it's the simplest to build (we don't invest too much into it,
> don't shift focus and will shake down more important things faster),
> (2) there are some important actions that are best done with CL
> anyway (e.g. mailing a patch). Later it may serve as an
> entry point for starting the richer terminal GUI or other types of GUIs.

+1, agree.

> There are 3 groups of people we should be looking at:
> - contributors (important special case: sending first patch)
> - maintainers
> - reviewers
> 
> I would set the first milestone as having the CL utility (let's call
> it "kit"*) which can do:
> 
> $ kit init
> # Does some necessary one-time initialization, executed from the
> # kernel git checkout.
> 
> $ kit mail
> # Sends the top commit in the current branch for review.
> 
> So that would be the workflow for sending your first kernel patch.
> 
> Later "kit mail" can also run checkpatch, check SOB tag, add some kind
> of change ID or anything else we will consider necessary. It may be
> necessary to be able to force-override some of the checks, but by default
> you are now getting patches that have SOB, checkpatch-clean, etc.
> 
> If there is an easy way to make it work with the current email-based
> process (i.e. send email on your behalf and you receive incoming emails),
> then we could do that first and give it to new developers to relief from
> setting up email client. Otherwise, we should continue developing it
> based on something like SSB (or whatever protocol we will choose).
> 
> Obviously, the intention is that if you do "kit mail" second time
> with a changed patch, it sends "V2". Or if you have multiple local
> commits it will properly mail the series (or V2 of the series).
> 
> Most (all) of the "kit" functionality should be separated from the UI
> and be available for scripting/automation/other UIs. Whether it's
> done as "libgit" or as "shell out" is discussable.
[...]
On that note, such a tool would also need to co-exist with the current
email based process for some (long?) time in order to allow a smooth
transition period. Last week I spent a few of nights hacking a small tool
which is regularly pulling the lore git trees I'm interested in and checking
out all [new] mails into maildir format so they can be read naturally by
UIs like mutt et al [0]. As an experiment, in case of bpf vger mailing list,
it extracts all current ~8k mails in under a second:

$ ./l2md
general.maildir = /home/foo/.l2md/maildir/common
general.period = 30
repos.bpf.maildir = /home/foo/.l2md/maildir/bpf
repos.bpf.initial_import = 0
repos.bpf.url = https://lore.kernel.org/bpf/0
repos.bpf.oid_maildir = [unknown]
Initial repository check.
Cloning: https://lore.kernel.org/bpf/0
Remote: Counting objects: 23859, done.
Remote: Compressing objects: 100% (14476/14476), done.
Remote: Total 23859 (delta 1561), reused 22125 (delta 1430)
Initial repository check completed.
Bootstrap done.
Resyncing maildirs.
Processed 7953 new mails for https://lore.kernel.org/bpf/0 in 0.586466s.
Sync done. Sleeping 30s.
Resyncing repositories.
Fetching: https://lore.kernel.org/bpf/0
Merging: refs/heads/master commit d608393d011aa0c0fc5983059052362cebeafc91
Resyncing maildirs.
Processed 0 new mails for https://lore.kernel.org/bpf/0 in 0.53611s.
[...]

I've started using it daily now and it appears to work nicely so far.
Given that, I'm wondering whether for a "kit" tool and beyond, we could use
something like lore git trees as a basis. The 'receive' side is already
there today, but I'm wondering if it's feasible to have a sendmail compatible
interface that would allow to push new "mails" into lore git trees as well,
where a bridge from lore git to vger list then distributes the message to
all email-based subscribers. Obviously this needs similar spam-filtering and
sanity checks as with vger lists, but eventually it wouldn't be any different
than anyone being able to send to vger via email. The nice thing would be
that the trees are mirrored on a wide basis already (and wouldn't need any
additional transport) and CI infrastructure can just pull them, work with
git sha ids, and reply with the results by pushing (implicitly by mentioned
sendmail compat tool, or "kit" etc). "kit" tool for contributors/reviewers
(I'd probably put both into the same bucket on a high level) and maintainers
could be built around the trees as foundation which could already solve the
issues around mail pointed out by Konstantin some time ago [1] while allowing
max compatibility to current workflows as a transition period. Just a thought.

Cheers,
Daniel

   [0] https://git.kernel.org/pub/scm/linux/kernel/git/dborkman/l2md.git/
   [1] https://people.kernel.org/monsieuricon/patches-carved-into-developer-sigchains
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Dmitry Vyukov via Ksummit-discuss]

On Mon, Sep 23, 2019 at 2:52 PM Daniel Borkmann <daniel@iogearbox.net> wrote:
>
> On 9/22/19 2:02 PM, Dmitry Vyukov wrote:
> [...]
> > Also adding people from the "Kernel development collaboration platform
> > wish list" discussion on the workflows list [1].
> > (Rafael et al, thanks for collecting the requirements, that's very useful!)
> >
> > I second the idea expressed by several people that addressing the
> > contributor side is a very important part of this effort.
> >
> > While I understand the intention to provide something useful as fast
> > as possible, I also a bit afraid that the Stage 1 ("putt") diverges
> > us into investing into particular UI, tying capabilities with this UI
> > and not addressing the fundamental problems.
> > People are expressing a preference for different types of UIs
> > (CL, terminal GUI, web, scripting), I think eventually we will have
> > several. So I would suggest to untie features/capabilities from
> > any particular UI as much as possible, and start with addressing more
> > fundamental aspects. Building richer features on top of the current
> > human-oriented emails is also going to be much harder, and that's the
> > work that we eventually will (hopefully) throw away.
> >
> >  From UI perspective I think we should start with a CL interface because
> > (1) it's the simplest to build (we don't invest too much into it,
> > don't shift focus and will shake down more important things faster),
> > (2) there are some important actions that are best done with CL
> > anyway (e.g. mailing a patch). Later it may serve as an
> > entry point for starting the richer terminal GUI or other types of GUIs.
>
> +1, agree.
>
> > There are 3 groups of people we should be looking at:
> > - contributors (important special case: sending first patch)
> > - maintainers
> > - reviewers
> >
> > I would set the first milestone as having the CL utility (let's call
> > it "kit"*) which can do:
> >
> > $ kit init
> > # Does some necessary one-time initialization, executed from the
> > # kernel git checkout.
> >
> > $ kit mail
> > # Sends the top commit in the current branch for review.
> >
> > So that would be the workflow for sending your first kernel patch.
> >
> > Later "kit mail" can also run checkpatch, check SOB tag, add some kind
> > of change ID or anything else we will consider necessary. It may be
> > necessary to be able to force-override some of the checks, but by default
> > you are now getting patches that have SOB, checkpatch-clean, etc.
> >
> > If there is an easy way to make it work with the current email-based
> > process (i.e. send email on your behalf and you receive incoming emails),
> > then we could do that first and give it to new developers to relief from
> > setting up email client. Otherwise, we should continue developing it
> > based on something like SSB (or whatever protocol we will choose).
> >
> > Obviously, the intention is that if you do "kit mail" second time
> > with a changed patch, it sends "V2". Or if you have multiple local
> > commits it will properly mail the series (or V2 of the series).
> >
> > Most (all) of the "kit" functionality should be separated from the UI
> > and be available for scripting/automation/other UIs. Whether it's
> > done as "libgit" or as "shell out" is discussable.
> [...]
> On that note, such a tool would also need to co-exist with the current
> email based process for some (long?) time in order to allow a smooth
> transition period. Last week I spent a few of nights hacking a small tool
> which is regularly pulling the lore git trees I'm interested in and checking
> out all [new] mails into maildir format so they can be read naturally by
> UIs like mutt et al [0]. As an experiment, in case of bpf vger mailing list,
> it extracts all current ~8k mails in under a second:
>
> $ ./l2md
> general.maildir = /home/foo/.l2md/maildir/common
> general.period = 30
> repos.bpf.maildir = /home/foo/.l2md/maildir/bpf
> repos.bpf.initial_import = 0
> repos.bpf.url = https://lore.kernel.org/bpf/0
> repos.bpf.oid_maildir = [unknown]
> Initial repository check.
> Cloning: https://lore.kernel.org/bpf/0
> Remote: Counting objects: 23859, done.
> Remote: Compressing objects: 100% (14476/14476), done.
> Remote: Total 23859 (delta 1561), reused 22125 (delta 1430)
> Initial repository check completed.
> Bootstrap done.
> Resyncing maildirs.
> Processed 7953 new mails for https://lore.kernel.org/bpf/0 in 0.586466s.
> Sync done. Sleeping 30s.
> Resyncing repositories.
> Fetching: https://lore.kernel.org/bpf/0
> Merging: refs/heads/master commit d608393d011aa0c0fc5983059052362cebeafc91
> Resyncing maildirs.
> Processed 0 new mails for https://lore.kernel.org/bpf/0 in 0.53611s.
> [...]
>
> I've started using it daily now and it appears to work nicely so far.
> Given that, I'm wondering whether for a "kit" tool and beyond, we could use
> something like lore git trees as a basis. The 'receive' side is already
> there today, but I'm wondering if it's feasible to have a sendmail compatible
> interface that would allow to push new "mails" into lore git trees as well,
> where a bridge from lore git to vger list then distributes the message to
> all email-based subscribers. Obviously this needs similar spam-filtering and
> sanity checks as with vger lists, but eventually it wouldn't be any different
> than anyone being able to send to vger via email. The nice thing would be
> that the trees are mirrored on a wide basis already (and wouldn't need any
> additional transport) and CI infrastructure can just pull them, work with
> git sha ids, and reply with the results by pushing (implicitly by mentioned
> sendmail compat tool, or "kit" etc). "kit" tool for contributors/reviewers
> (I'd probably put both into the same bucket on a high level) and maintainers
> could be built around the trees as foundation which could already solve the
> issues around mail pointed out by Konstantin some time ago [1] while allowing
> max compatibility to current workflows as a transition period. Just a thought.

Hi Daniel,

Do I understand correctly that you propose to use git as a transport
layer to store some structured data about changes? This was mentioned
during the maintainers summit [1], see "Git as the transport layer"
section.

Using git is nice in the sense that we could reuse lots of existing
infrastructure/code/tools rather than invent our own. And I think in
the end the exact transport layer does not matter much (for users) as
long as we can build the same interface on top of it.

However, Konstantin mentioned potential scalability problem with git
in such scenario. It wasn't supposed to store "journal" type info, but
rather small diffs to a code base. If we are considering such option
for real, I guess we just need to evaluate it in such scenario and
check how well/bad it works. We could also try to optimize git for
such a scenario, maybe there is a single bottleneck that we could
optimize...

There will probably also be some implications on possibility of local peering.

If everybody will be able to push into the single git, we will need to
figure out user auth story and single user screwing the whole thing
intentionally or by accident.

[1] https://lwn.net/Articles/799134/
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Daniel Borkmann]

On 9/23/19 4:08 PM, Dmitry Vyukov wrote:
> On Mon, Sep 23, 2019 at 2:52 PM Daniel Borkmann <daniel@iogearbox.net> wrote:
>> On 9/22/19 2:02 PM, Dmitry Vyukov wrote:
>> [...]
>>> Also adding people from the "Kernel development collaboration platform
>>> wish list" discussion on the workflows list [1].
>>> (Rafael et al, thanks for collecting the requirements, that's very useful!)
>>>
>>> I second the idea expressed by several people that addressing the
>>> contributor side is a very important part of this effort.
>>>
>>> While I understand the intention to provide something useful as fast
>>> as possible, I also a bit afraid that the Stage 1 ("putt") diverges
>>> us into investing into particular UI, tying capabilities with this UI
>>> and not addressing the fundamental problems.
>>> People are expressing a preference for different types of UIs
>>> (CL, terminal GUI, web, scripting), I think eventually we will have
>>> several. So I would suggest to untie features/capabilities from
>>> any particular UI as much as possible, and start with addressing more
>>> fundamental aspects. Building richer features on top of the current
>>> human-oriented emails is also going to be much harder, and that's the
>>> work that we eventually will (hopefully) throw away.
>>>
>>>   From UI perspective I think we should start with a CL interface because
>>> (1) it's the simplest to build (we don't invest too much into it,
>>> don't shift focus and will shake down more important things faster),
>>> (2) there are some important actions that are best done with CL
>>> anyway (e.g. mailing a patch). Later it may serve as an
>>> entry point for starting the richer terminal GUI or other types of GUIs.
>>
>> +1, agree.
>>
>>> There are 3 groups of people we should be looking at:
>>> - contributors (important special case: sending first patch)
>>> - maintainers
>>> - reviewers
>>>
>>> I would set the first milestone as having the CL utility (let's call
>>> it "kit"*) which can do:
>>>
>>> $ kit init
>>> # Does some necessary one-time initialization, executed from the
>>> # kernel git checkout.
>>>
>>> $ kit mail
>>> # Sends the top commit in the current branch for review.
>>>
>>> So that would be the workflow for sending your first kernel patch.
>>>
>>> Later "kit mail" can also run checkpatch, check SOB tag, add some kind
>>> of change ID or anything else we will consider necessary. It may be
>>> necessary to be able to force-override some of the checks, but by default
>>> you are now getting patches that have SOB, checkpatch-clean, etc.
>>>
>>> If there is an easy way to make it work with the current email-based
>>> process (i.e. send email on your behalf and you receive incoming emails),
>>> then we could do that first and give it to new developers to relief from
>>> setting up email client. Otherwise, we should continue developing it
>>> based on something like SSB (or whatever protocol we will choose).
>>>
>>> Obviously, the intention is that if you do "kit mail" second time
>>> with a changed patch, it sends "V2". Or if you have multiple local
>>> commits it will properly mail the series (or V2 of the series).
>>>
>>> Most (all) of the "kit" functionality should be separated from the UI
>>> and be available for scripting/automation/other UIs. Whether it's
>>> done as "libgit" or as "shell out" is discussable.
>> [...]
>> On that note, such a tool would also need to co-exist with the current
>> email based process for some (long?) time in order to allow a smooth
>> transition period. Last week I spent a few of nights hacking a small tool
>> which is regularly pulling the lore git trees I'm interested in and checking
>> out all [new] mails into maildir format so they can be read naturally by
>> UIs like mutt et al [0]. As an experiment, in case of bpf vger mailing list,
>> it extracts all current ~8k mails in under a second:
>>
>> $ ./l2md
>> general.maildir = /home/foo/.l2md/maildir/common
>> general.period = 30
>> repos.bpf.maildir = /home/foo/.l2md/maildir/bpf
>> repos.bpf.initial_import = 0
>> repos.bpf.url = https://lore.kernel.org/bpf/0
>> repos.bpf.oid_maildir = [unknown]
>> Initial repository check.
>> Cloning: https://lore.kernel.org/bpf/0
>> Remote: Counting objects: 23859, done.
>> Remote: Compressing objects: 100% (14476/14476), done.
>> Remote: Total 23859 (delta 1561), reused 22125 (delta 1430)
>> Initial repository check completed.
>> Bootstrap done.
>> Resyncing maildirs.
>> Processed 7953 new mails for https://lore.kernel.org/bpf/0 in 0.586466s.
>> Sync done. Sleeping 30s.
>> Resyncing repositories.
>> Fetching: https://lore.kernel.org/bpf/0
>> Merging: refs/heads/master commit d608393d011aa0c0fc5983059052362cebeafc91
>> Resyncing maildirs.
>> Processed 0 new mails for https://lore.kernel.org/bpf/0 in 0.53611s.
>> [...]
>>
>> I've started using it daily now and it appears to work nicely so far.
>> Given that, I'm wondering whether for a "kit" tool and beyond, we could use
>> something like lore git trees as a basis. The 'receive' side is already
>> there today, but I'm wondering if it's feasible to have a sendmail compatible
>> interface that would allow to push new "mails" into lore git trees as well,
>> where a bridge from lore git to vger list then distributes the message to
>> all email-based subscribers. Obviously this needs similar spam-filtering and
>> sanity checks as with vger lists, but eventually it wouldn't be any different
>> than anyone being able to send to vger via email. The nice thing would be
>> that the trees are mirrored on a wide basis already (and wouldn't need any
>> additional transport) and CI infrastructure can just pull them, work with
>> git sha ids, and reply with the results by pushing (implicitly by mentioned
>> sendmail compat tool, or "kit" etc). "kit" tool for contributors/reviewers
>> (I'd probably put both into the same bucket on a high level) and maintainers
>> could be built around the trees as foundation which could already solve the
>> issues around mail pointed out by Konstantin some time ago [1] while allowing
>> max compatibility to current workflows as a transition period. Just a thought.
> 
> Hi Daniel,
> 
> Do I understand correctly that you propose to use git as a transport
> layer to store some structured data about changes? This was mentioned
> during the maintainers summit [1], see "Git as the transport layer"
> section.
> 
> Using git is nice in the sense that we could reuse lots of existing
> infrastructure/code/tools rather than invent our own. And I think in
> the end the exact transport layer does not matter much (for users) as
> long as we can build the same interface on top of it.
> 
> However, Konstantin mentioned potential scalability problem with git
> in such scenario. It wasn't supposed to store "journal" type info, but
> rather small diffs to a code base. If we are considering such option
> for real, I guess we just need to evaluate it in such scenario and
> check how well/bad it works. We could also try to optimize git for
> such a scenario, maybe there is a single bottleneck that we could
> optimize...
>
> There will probably also be some implications on possibility of local peering.
> 
> If everybody will be able to push into the single git, we will need to
> figure out user auth story and single user screwing the whole thing
> intentionally or by accident.

Yep all true. The way lore git works is that there's a single file called 'm'
in the repo which always contains the top most mail from the list [0], and [1]
contains basically the diff to the previous mail from the list. Extracting the
actual git object blobs is sufficiently fast (imho, see above and [2]), and
basically the receive side integration into a mail client is already there with
a mechanism like l2md tool has. A push mechanism would need more server side
logic that ensures to not screw up the repo, never allow force push, and always
have the pushed 'm' file(s) conflict-free at the top of the tree (along with the
usual mail/spam filtering there is in place on vger). The way 'm' is handled
today may not be nicest or most natural way for git indeed, and despite that
it may seem more of an implementation detail at this point I thought I mention
it here along with l2md since it can be used today for reading and it feels the
'push' side may not be too far off to realize. It wouldn't solve all the other
discussed items from the wish list you/others brought up in your original thread
(which would need to work on top of the transport), but could be an alternative
way with regards to the email problem.

Thanks,
Daniel

  [0] https://git.kernel.org/pub/scm/public-inbox/vger.kernel.org/bpf/0.git/tree/m
  [1] https://git.kernel.org/pub/scm/public-inbox/vger.kernel.org/bpf/0.git/commit/?id=0c0e0a248293aa382c3451680e696bbeaf38c47e
  [2] Converting the entire netdev history (if anyone ever needs it in their mail client)
      with l2md from 03/2002 till 09/2019 takes on my laptop:
      [...]
      Processed 137396 new mails for https://lore.kernel.org/netdev/1 in 11.198241s.
      Processed 498001 new mails for https://lore.kernel.org/netdev/0 in 41.414329s.
      [...]

> [1] https://lwn.net/Articles/799134/
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Konstantin Ryabitsev]

On Mon, Sep 23, 2019 at 02:52:01PM +0200, Daniel Borkmann wrote:
> > Most (all) of the "kit" functionality should be separated from the UI
> > and be available for scripting/automation/other UIs. Whether it's
> > done as "libgit" or as "shell out" is discussable.
> [...]
> On that note, such a tool would also need to co-exist with the current
> email based process for some (long?) time in order to allow a smooth
> transition period. Last week I spent a few of nights hacking a small tool
> which is regularly pulling the lore git trees I'm interested in and checking
> out all [new] mails into maildir format so they can be read naturally by
> UIs like mutt et al [0]. As an experiment, in case of bpf vger mailing list,
> it extracts all current ~8k mails in under a second:

Thanks for working on this -- I've started on a similar tool in the
past, but got distracted and never completed it. In my implementation,
it was piping messages to procmail, which allowed writing complex rules
for folders/pre-processing, etc. May I suggest that your tool also
offers a stdout that can be piped to procmail?

-K
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Daniel Borkmann]

On 9/30/19 11:24 PM, Konstantin Ryabitsev wrote:
> On Mon, Sep 23, 2019 at 02:52:01PM +0200, Daniel Borkmann wrote:
>>> Most (all) of the "kit" functionality should be separated from the UI
>>> and be available for scripting/automation/other UIs. Whether it's
>>> done as "libgit" or as "shell out" is discussable.
>> [...]
>> On that note, such a tool would also need to co-exist with the current
>> email based process for some (long?) time in order to allow a smooth
>> transition period. Last week I spent a few of nights hacking a small tool
>> which is regularly pulling the lore git trees I'm interested in and checking
>> out all [new] mails into maildir format so they can be read naturally by
>> UIs like mutt et al [0]. As an experiment, in case of bpf vger mailing list,
>> it extracts all current ~8k mails in under a second:
> 
> Thanks for working on this -- I've started on a similar tool in the
> past, but got distracted and never completed it. In my implementation,
> it was piping messages to procmail, which allowed writing complex rules
> for folders/pre-processing, etc. May I suggest that your tool also
> offers a stdout that can be piped to procmail?

I have a rough version working now. ;-) Just pushed to [0]. Let me know if that
does the trick on your side, I've added example configs for procmail to the repo
as well for getting started. Did a quick run for l2md -> procmail -> mutt and
seems fine as far as I can tell. (Patches always welcome of course.)

Thanks,
Daniel

   [0] https://git.kernel.org/pub/scm/linux/kernel/git/dborkman/l2md.git/
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Reflections on kernel development processes

[Konstantin Ryabitsev]

On Tue, Oct 01, 2019 at 11:33:06PM +0200, Daniel Borkmann wrote:
> > Thanks for working on this -- I've started on a similar tool in the
> > past, but got distracted and never completed it. In my implementation,
> > it was piping messages to procmail, which allowed writing complex rules
> > for folders/pre-processing, etc. May I suggest that your tool also
> > offers a stdout that can be piped to procmail?
> 
> I have a rough version working now. ;-) Just pushed to [0]. Let me know if that
> does the trick on your side, I've added example configs for procmail to the repo
> as well for getting started. Did a quick run for l2md -> procmail -> mutt and
> seems fine as far as I can tell. (Patches always welcome of course.)

Nice, thanks! Until we have a better tool for holistic patch/series
management backed by public-inbox, this should allow folks to pre-filter
their messages.

Best,
-K
_______________________________________________
Ksummit-discuss mailing list
Ksummit-discuss@lists.linuxfoundation.org
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss