[Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Jiri Kosina]

I think it'd be useful to have a session where maintainers could come up 
with feature / improvement requests for kernel.org infrastructure and have 
a subsequent discussion about whether they would be generally useful.

Let me start with my personal wish:

I personally would very welcome a facility that'd send out an e-mail if a 
new commit is pushed to a git.kernel.org repo branch (sort of what tip-bot 
and akpm's scripts are doing these days) to automatically notify the patch 
author that the patch has been merged and pushed out.

Suggested attendance: Konstantin, maintainers :)

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 573 bytes --]

On Tue, Jul 07, 2015 at 10:42:49PM +0200, Jiri Kosina wrote:

> I personally would very welcome a facility that'd send out an e-mail if a 
> new commit is pushed to a git.kernel.org repo branch (sort of what tip-bot 
> and akpm's scripts are doing these days) to automatically notify the patch 
> author that the patch has been merged and pushed out.

Those are useful, I know Greg has one and I just wrote one too.  I'm not
sure it needs to be on the server side though, they can also be done
from the client which lets you try to fine tune things more readily.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Dmitry Torokhov]

On Tue, Jul 07, 2015 at 11:40:25PM +0100, Mark Brown wrote:
> On Tue, Jul 07, 2015 at 10:42:49PM +0200, Jiri Kosina wrote:
> 
> > I personally would very welcome a facility that'd send out an e-mail if a 
> > new commit is pushed to a git.kernel.org repo branch (sort of what tip-bot 
> > and akpm's scripts are doing these days) to automatically notify the patch 
> > author that the patch has been merged and pushed out.
> 
> Those are useful, I know Greg has one and I just wrote one too.  I'm not
> sure it needs to be on the server side though, they can also be done
> from the client which lets you try to fine tune things more readily.

If either of you could share the scripts that would be great.

-- 
Dmitry

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Greg KH]

On Tue, Jul 07, 2015 at 03:52:23PM -0700, Dmitry Torokhov wrote:
> On Tue, Jul 07, 2015 at 11:40:25PM +0100, Mark Brown wrote:
> > On Tue, Jul 07, 2015 at 10:42:49PM +0200, Jiri Kosina wrote:
> > 
> > > I personally would very welcome a facility that'd send out an e-mail if a 
> > > new commit is pushed to a git.kernel.org repo branch (sort of what tip-bot 
> > > and akpm's scripts are doing these days) to automatically notify the patch 
> > > author that the patch has been merged and pushed out.
> > 
> > Those are useful, I know Greg has one and I just wrote one too.  I'm not
> > sure it needs to be on the server side though, they can also be done
> > from the client which lets you try to fine tune things more readily.
> 
> If either of you could share the scripts that would be great.

Mine is based on Andrew's scripts, and can be found buried in my
"gregkh-linux" repo which is a shadow of my 'linux/' subdir on my
development machines, specifically this file:
	https://github.com/gregkh/gregkh-linux/blob/master/work/do.sh

Yeah, my scripts are named 'do.sh', naming is hard...

greg k-h

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Jiri Kosina]

On Tue, 7 Jul 2015, Greg KH wrote:

> > > Those are useful, I know Greg has one and I just wrote one too.  I'm not
> > > sure it needs to be on the server side though, they can also be done
> > > from the client which lets you try to fine tune things more readily.
> > 
> > If either of you could share the scripts that would be great.
> 
> Mine is based on Andrew's scripts, and can be found buried in my
> "gregkh-linux" repo which is a shadow of my 'linux/' subdir on my
> development machines, specifically this file:
> 	https://github.com/gregkh/gregkh-linux/blob/master/work/do.sh

Well, I think you've nicely demonstrated the reason why I think there 
should be a central machinery; perhaps as a stanard open-source project, 
i.e reviewed, contributed to, and shared by everybody.

The "everybody has his home-brew hacked-up script" model is just 
unnecessary duplication of work a brings potential for unnecessary bugs.

For example, please correct me if I am wrong, but I believe that if I send 
you a patch that would have authorship

	Jiri Kosina <jkosina`rm -rf .`@suse.com>

and you wouldn't notice prior to applying it using the above do.sh, I 
think your precious tree is gone.

This would be caught immediately if it's properly maintained "project".

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Mark Brown]

[-- Attachment #1: Type: text/plain, Size: 805 bytes --]

On Wed, Jul 08, 2015 at 10:02:36AM +0200, Jiri Kosina wrote:

> The "everybody has his home-brew hacked-up script" model is just 
> unnecessary duplication of work a brings potential for unnecessary bugs.

I think the only barrier here is someone writing some tooling that is
sufficiently useful and generic enough to work for people.  I know I
wrote my scripts mainly because none of the scripts I could find tie in
with my workflow (mainly around figuring out which branches in my local
tree correspond to branches on the server and syncing them up).  If
there'd been something I could just pick up I'd have happily done so.

> This would be caught immediately if it's properly maintained "project".

Or alternatively would allow people to quickly attack a large number of
developers :)

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Theodore Ts'o]

On Wed, Jul 08, 2015 at 10:35:11AM +0100, Mark Brown wrote:
> 
> I think the only barrier here is someone writing some tooling that is
> sufficiently useful and generic enough to work for people.  I know I
> wrote my scripts mainly because none of the scripts I could find tie in
> with my workflow (mainly around figuring out which branches in my local
> tree correspond to branches on the server and syncing them up).  If
> there'd been something I could just pick up I'd have happily done so.

Yeah, your concerns mirror mine:

1) It will require a lot of configuration --- just because a commit
shows up on a branch does not mean it is guaranteed that it will hit
mainline.  In fact, a maintainer might push a commit onto a throwaway
branch on kernel.org just so that the zero-day testing systems can
give the commit a spin.  So that means it's not just enough to throw a
bunch of git hook scripts on master.kernel.org, because maintainers
will need to have to configure, if not customize, them.

This leads to my second concern which is:

2) Having shell scripts run on master.kernel.org can be a significant
security concern; this is *especially* true if customization or
configuration is required.

> > This would be caught immediately if it's properly maintained "project".
> 
> Or alternatively would allow people to quickly attack a large number of
> developers :)

Indeed.

						- Ted

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Jason Cooper]

On Wed, Jul 08, 2015 at 10:01:55AM -0400, Theodore Ts'o wrote:
> On Wed, Jul 08, 2015 at 10:35:11AM +0100, Mark Brown wrote:
> > 
> > I think the only barrier here is someone writing some tooling that is
> > sufficiently useful and generic enough to work for people.  I know I
> > wrote my scripts mainly because none of the scripts I could find tie in
> > with my workflow (mainly around figuring out which branches in my local
> > tree correspond to branches on the server and syncing them up).  If
> > there'd been something I could just pick up I'd have happily done so.
> 
> Yeah, your concerns mirror mine:
> 
> 1) It will require a lot of configuration --- just because a commit
> shows up on a branch does not mean it is guaranteed that it will hit
> mainline.  In fact, a maintainer might push a commit onto a throwaway
> branch on kernel.org just so that the zero-day testing systems can
> give the commit a spin.  So that means it's not just enough to throw a
> bunch of git hook scripts on master.kernel.org, because maintainers
> will need to have to configure, if not customize, them.

Ack.  Is there room for a convention here?  e.g. dev opts in, which
means that tree follows a certain branch naming convention.

> This leads to my second concern which is:
> 
> 2) Having shell scripts run on master.kernel.org can be a significant
> security concern; this is *especially* true if customization or
> configuration is required.

Is it worse for an attacker to execute code as a designated,
unprivileged user on the server, with access to copies of repos?  Or, to
execute code as a dev on a dev box?  The latter potentially giving
access to ssh and pgp keys.

I'd argue that it's Konstantin's and David's jobs to keep those servers
locked down and properly configured.  Devs and dev boxes are held to no
such standard, and yet contain much more critical resources.

We can easily detect and recover from an attacker changing a repo on the
server.  The PGP signed tag will fail.  How do we detect and recover
from, say, Greg or Linus' machine being compromised?

This threat model assumes an intentionally malicious attacker whose goal
is to get bad shit into the kernel and pass git tag verification.

thx,

Jason.

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Jiri Kosina]

On Wed, 8 Jul 2015, Theodore Ts'o wrote:

> 1) It will require a lot of configuration --- just because a commit 
> shows up on a branch does not mean it is guaranteed that it will hit 
> mainline.  
>
> In fact, a maintainer might push a commit onto a throwaway branch on 
> kernel.org just so that the zero-day testing systems can give the commit 
> a spin.  So that means it's not just enough to throw a bunch of git hook 
> scripts on master.kernel.org, because maintainers will need to have to 
> configure, if not customize, them.
> 
> This leads to my second concern which is:

Yeah, some kind of pattern matching would be very useful (like "send mails 
only for commits that newly appar in 'for-next' branch", or some such).

> 2) Having shell scripts run on master.kernel.org can be a significant
> security concern; this is *especially* true if customization or
> configuration is required.

I don't think there is any principal reason this needs to run on master. 
It can be on any kind of scratch one-purpose system that would just be a 
slave git mirror.

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Greg KH]

On Wed, Jul 08, 2015 at 10:01:55AM -0400, Theodore Ts'o wrote:
> On Wed, Jul 08, 2015 at 10:35:11AM +0100, Mark Brown wrote:
> > 
> > I think the only barrier here is someone writing some tooling that is
> > sufficiently useful and generic enough to work for people.  I know I
> > wrote my scripts mainly because none of the scripts I could find tie in
> > with my workflow (mainly around figuring out which branches in my local
> > tree correspond to branches on the server and syncing them up).  If
> > there'd been something I could just pick up I'd have happily done so.
> 
> Yeah, your concerns mirror mine:
> 
> 1) It will require a lot of configuration --- just because a commit
> shows up on a branch does not mean it is guaranteed that it will hit
> mainline.  In fact, a maintainer might push a commit onto a throwaway
> branch on kernel.org just so that the zero-day testing systems can
> give the commit a spin.  So that means it's not just enough to throw a
> bunch of git hook scripts on master.kernel.org, because maintainers
> will need to have to configure, if not customize, them.
> 
> This leads to my second concern which is:
> 
> 2) Having shell scripts run on master.kernel.org can be a significant
> security concern; this is *especially* true if customization or
> configuration is required.

Yeah, I really don't want to see shell access return to any kernel.org
systems, and that includes git hooks.  Using local git hooks to send
emails out seems to work well given that people here are already doing
it for their workflows.

thanks,

greg k-h

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Fengguang Wu]

On Wed, Jul 08, 2015 at 10:01:55AM -0400, Theodore Ts'o wrote:
> On Wed, Jul 08, 2015 at 10:35:11AM +0100, Mark Brown wrote:
> > 
> > I think the only barrier here is someone writing some tooling that is
> > sufficiently useful and generic enough to work for people.  I know I
> > wrote my scripts mainly because none of the scripts I could find tie in
> > with my workflow (mainly around figuring out which branches in my local
> > tree correspond to branches on the server and syncing them up).  If
> > there'd been something I could just pick up I'd have happily done so.
> 
> Yeah, your concerns mirror mine:
> 
> 1) It will require a lot of configuration --- just because a commit
> shows up on a branch does not mean it is guaranteed that it will hit
> mainline.  In fact, a maintainer might push a commit onto a throwaway
> branch on kernel.org just so that the zero-day testing systems can
> give the commit a spin.  So that means it's not just enough to throw a
> bunch of git hook scripts on master.kernel.org, because maintainers
> will need to have to configure, if not customize, them.

Agreed. 0day also accumulated a complex set rules for deciding which
errors/trees should be auto reported, which should be manually checked,
and which can be silently ignored.

> This leads to my second concern which is:
> 
> 2) Having shell scripts run on master.kernel.org can be a significant
> security concern; this is *especially* true if customization or
> configuration is required.

0day has the same security concern. We end up running a dedicated git
mirror server for pulling 500 git trees and feeding them to the test
farm behind firewalls.

So the chances are that 0day infrastructure has the ready made setups
to run such service. We already have the commit-by-commit walker to
run checkpatch.pl (though I rarely send its report out due to it'd be
noisy and there is no clear criterion to answer "who would be glad to
receive such reports?").

If we are going to do it, the notification scripts may be hosted in

git://git.kernel.org/pub/scm/linux/kernel/git/wfg/lkp-tests.git

since there will be a copy of it in each build server.

> > > This would be caught immediately if it's properly maintained "project".
> > 
> > Or alternatively would allow people to quickly attack a large number of
> > developers :)
> 
> Indeed.
> 
> 						- Ted
> _______________________________________________
> Ksummit-discuss mailing list
> Ksummit-discuss@lists.linuxfoundation.org
> https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Steven Rostedt]

On Sun, 12 Jul 2015 20:55:52 +0800
Fengguang Wu <fengguang.wu@intel.com> wrote:


> So the chances are that 0day infrastructure has the ready made setups
> to run such service. We already have the commit-by-commit walker to
> run checkpatch.pl (though I rarely send its report out due to it'd be
> noisy and there is no clear criterion to answer "who would be glad to
> receive such reports?").
> 

Is there a web site available that would let us see what checkpatch
errors the 0day bot found for our repo? I don't want an email, but
it would be nice to just go to some web page to look if anything new
appeared.

-- Steve

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Fengguang Wu]

On Mon, Jul 13, 2015 at 12:22:57PM -0400, Steven Rostedt wrote:
> On Sun, 12 Jul 2015 20:55:52 +0800
> Fengguang Wu <fengguang.wu@intel.com> wrote:
> 
> 
> > So the chances are that 0day infrastructure has the ready made setups
> > to run such service. We already have the commit-by-commit walker to
> > run checkpatch.pl (though I rarely send its report out due to it'd be
> > noisy and there is no clear criterion to answer "who would be glad to
> > receive such reports?").
> > 
> 
> Is there a web site available that would let us see what checkpatch
> errors the 0day bot found for our repo? I don't want an email, but

I'd expect checkpatch trivial to run locally for one's own patches.

> it would be nice to just go to some web page to look if anything new
> appeared.

Ah OK, you are interested in "new" checkpatch warnings. Actually I
run checkpatch against the patch file rather than source file. In
patch mode, there is no record of new/old error ids.

Thanks,
Fengguang

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Greg KH]

On Wed, Jul 08, 2015 at 10:02:36AM +0200, Jiri Kosina wrote:
> On Tue, 7 Jul 2015, Greg KH wrote:
> 
> > > > Those are useful, I know Greg has one and I just wrote one too.  I'm not
> > > > sure it needs to be on the server side though, they can also be done
> > > > from the client which lets you try to fine tune things more readily.
> > > 
> > > If either of you could share the scripts that would be great.
> > 
> > Mine is based on Andrew's scripts, and can be found buried in my
> > "gregkh-linux" repo which is a shadow of my 'linux/' subdir on my
> > development machines, specifically this file:
> > 	https://github.com/gregkh/gregkh-linux/blob/master/work/do.sh
> 
> Well, I think you've nicely demonstrated the reason why I think there 
> should be a central machinery; perhaps as a stanard open-source project, 
> i.e reviewed, contributed to, and shared by everybody.
> 
> The "everybody has his home-brew hacked-up script" model is just 
> unnecessary duplication of work a brings potential for unnecessary bugs.
> 
> For example, please correct me if I am wrong, but I believe that if I send 
> you a patch that would have authorship
> 
> 	Jiri Kosina <jkosina`rm -rf .`@suse.com>
> 
> and you wouldn't notice prior to applying it using the above do.sh, I 
> think your precious tree is gone.

Note, I don't have "precious" trees, nor "precious" development systems,
that's what git is for :)

But you are right in that this isn't good, and Andrew's scripts also
have the same bug as mine are based on his.  But this isn't a matter of
them not being open, both of ours have been published publicly for over
a decade.

Every 6 months or so I start to rewrite the script to use a "proper"
email address parser library, and give up.  I guess now I have to do it
for real, thanks for giving me a reason to do so.

> This would be caught immediately if it's properly maintained "project".

Hm, ok, but different maintainers have different needs.  James's git
hooks are also really nice, and work for some workflows (note, I use
them for some stable workflows).  Wolfram has published his scripts that
he uses for applying and testing patches in the past as well, and while
they were great for him, they didn't fit mine at the time.

So given 4 different maintainers, all of whom have posted their tools,
all of them are different given different methods of working.  Creating
one set of tools might be a bit hard based on this tiny sample size
already...

But, I'm willing to try, and clean up my mess, I'll work on making my
scripts "work robustly" and publish them and take feedback.  Ideally I
can tie them into the stable patch review process as well, as I know
others have been asking for those scripts to be unified into something
that someone other than me can get working.

thanks,

greg k-h

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Jiri Kosina]

On Wed, 8 Jul 2015, Greg KH wrote:

> > For example, please correct me if I am wrong, but I believe that if I send 
> > you a patch that would have authorship
> > 
> > 	Jiri Kosina <jkosina`rm -rf .`@suse.com>
> > 
> > and you wouldn't notice prior to applying it using the above do.sh, I 
> > think your precious tree is gone.
> 
> Note, I don't have "precious" trees, nor "precious" development systems,
> that's what git is for :)

Oh, sure, that was just an example because I was too lazy to construct a 
command that would send me your ssh keys :)

> > This would be caught immediately if it's properly maintained "project".
> 
> Hm, ok, but different maintainers have different needs.  James's git
> hooks are also really nice, and work for some workflows (note, I use
> them for some stable workflows).  Wolfram has published his scripts that
> he uses for applying and testing patches in the past as well, and while
> they were great for him, they didn't fit mine at the time.

Yes, my main point really was that I personally would find it super-nice 
to just mark particular branches in my kernel.org repositories as "send 
notifications for these", and author (and anyone else in "-by:" lines) 
would automatically receive nice notification once commit appears there. 
Without any need of wrapping simple git-am + git push with more and more 
bloated scripting.

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Steven Rostedt]

On Wed, 8 Jul 2015 17:08:50 +0200 (CEST)
Jiri Kosina <jkosina@suse.com> wrote:

> Yes, my main point really was that I personally would find it super-nice 
> to just mark particular branches in my kernel.org repositories as "send 
> notifications for these", and author (and anyone else in "-by:" lines) 
> would automatically receive nice notification once commit appears there. 
> Without any need of wrapping simple git-am + git push with more and more 
> bloated scripting.

What about the opposite. Have people subscribe to git repos and get
emails when a patch they authored is pushed to it.

I subscribe to Linus's tree, but I get all commits. I just have a
filter that only sends me anything that has my SoB, and the rest goes
to /dev/null. But it does make my fetchmail waste a lot of time pulling
in emails that mostly get discarded. It would be nice to only see those
that I'm interested in.

I wouldn't expect a korg to parse the tags, but perhaps it could just
send people emails that they authored, and want a reply to.

Note, if you are worried about security, this could all be done on an
isolated machine that doesn't have any inside access to korg itself. As
it only needs to do a git pull and parse the content. Any mischievous
scripts would only get access to a box that has nothing but a dummy
user account and a bunch of git repos that it pulls in but never
pushes, and also a way to send emails.

-- Steve

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[josh]

On Tue, Jul 07, 2015 at 11:40:25PM +0100, Mark Brown wrote:
> On Tue, Jul 07, 2015 at 10:42:49PM +0200, Jiri Kosina wrote:
> 
> > I personally would very welcome a facility that'd send out an e-mail if a 
> > new commit is pushed to a git.kernel.org repo branch (sort of what tip-bot 
> > and akpm's scripts are doing these days) to automatically notify the patch 
> > author that the patch has been merged and pushed out.
> 
> Those are useful, I know Greg has one and I just wrote one too.  I'm not
> sure it needs to be on the server side though, they can also be done
> from the client which lets you try to fine tune things more readily.

I agree that it could be done on a system other than the git server
(though it's more efficient if hooked into git hooks), but it'd be nice
to have an automated service running on kernel.org, rather than
something every interested kernel developer has to run for themselves.
I'd certainly like to have mails telling me when a patch I submitted
gets merged, like those I get from Andrew Morton's automated scripts.

- Josh Triplett

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Jiri Kosina]

On Tue, 7 Jul 2015, Mark Brown wrote:

> > I personally would very welcome a facility that'd send out an e-mail if a 
> > new commit is pushed to a git.kernel.org repo branch (sort of what tip-bot 
> > and akpm's scripts are doing these days) to automatically notify the patch 
> > author that the patch has been merged and pushed out.
> 
> Those are useful, I know Greg has one and I just wrote one too.  I'm not
> sure it needs to be on the server side though, they can also be done
> from the client which lets you try to fine tune things more readily.

I explicitly proposed these to be (to some extent configurable) 
server-side hooks, exactly to avoid every committer maintaining his own 
set of home-brew scripts.

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Jason Cooper]

On Tue, Jul 07, 2015 at 11:40:25PM +0100, Mark Brown wrote:
> On Tue, Jul 07, 2015 at 10:42:49PM +0200, Jiri Kosina wrote:
> 
> > I personally would very welcome a facility that'd send out an e-mail if a 
> > new commit is pushed to a git.kernel.org repo branch (sort of what tip-bot 
> > and akpm's scripts are doing these days) to automatically notify the patch 
> > author that the patch has been merged and pushed out.
> 
> Those are useful, I know Greg has one and I just wrote one too.  I'm not
> sure it needs to be on the server side though, they can also be done
> from the client which lets you try to fine tune things more readily.

The tip tree has one as well.

There are many benefits to server-side, and I think it's worth
discussing the scope and success/fail criteria.  For example:

 - What will the script do?
 - What will it *not* do?
 - Should we try it for a year and report back?
    - How do we quantify success/failure?
 - Who is going to write it?  Maintain it?

It's also worth noting that this ties in neatly with the recruitment
effort.  I clearly recall the first automated email I received from
Greg, followed up quickly by an automated email from Andrew.  The "Holy
sh*t!  My patch got merged!" feeling hooked me.

I'll chime in with more thoughts on the recruitment thread, I don't want
to hijack this one.  Suffice to say, I think automated, targeted,
concise notifications would be a good thing.

thx,

Jason.

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Andy Lutomirski]

On Tue, Jul 7, 2015 at 1:42 PM, Jiri Kosina <jkosina@suse.com> wrote:
> I think it'd be useful to have a session where maintainers could come up
> with feature / improvement requests for kernel.org infrastructure and have
> a subsequent discussion about whether they would be generally useful.
>
> Let me start with my personal wish:
>
> I personally would very welcome a facility that'd send out an e-mail if a
> new commit is pushed to a git.kernel.org repo branch (sort of what tip-bot
> and akpm's scripts are doing these days) to automatically notify the patch
> author that the patch has been merged and pushed out.
>
> Suggested attendance: Konstantin, maintainers :)

I wouldn't mind something like patchwork for lkml.  Since I use gmail
(sigh), it's a minor PITA to pull out patchbombs in git am-able
format.

--Andy

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Guenter Roeck]

On 07/07/2015 04:31 PM, Andy Lutomirski wrote:
> On Tue, Jul 7, 2015 at 1:42 PM, Jiri Kosina <jkosina@suse.com> wrote:
>> I think it'd be useful to have a session where maintainers could come up
>> with feature / improvement requests for kernel.org infrastructure and have
>> a subsequent discussion about whether they would be generally useful.
>>
>> Let me start with my personal wish:
>>
>> I personally would very welcome a facility that'd send out an e-mail if a
>> new commit is pushed to a git.kernel.org repo branch (sort of what tip-bot
>> and akpm's scripts are doing these days) to automatically notify the patch
>> author that the patch has been merged and pushed out.
>>
>> Suggested attendance: Konstantin, maintainers :)
>
> I wouldn't mind something like patchwork for lkml.  Since I use gmail
> (sigh), it's a minor PITA to pull out patchbombs in git am-able
> format.
>
What is wrong with https://patchwork.kernel.org/project/LKML/list/ ?

Guenter

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Andy Lutomirski]

On Tue, Jul 7, 2015 at 4:37 PM, Guenter Roeck <linux@roeck-us.net> wrote:
> On 07/07/2015 04:31 PM, Andy Lutomirski wrote:
>>
>> On Tue, Jul 7, 2015 at 1:42 PM, Jiri Kosina <jkosina@suse.com> wrote:
>>>
>>> I think it'd be useful to have a session where maintainers could come up
>>> with feature / improvement requests for kernel.org infrastructure and
>>> have
>>> a subsequent discussion about whether they would be generally useful.
>>>
>>> Let me start with my personal wish:
>>>
>>> I personally would very welcome a facility that'd send out an e-mail if a
>>> new commit is pushed to a git.kernel.org repo branch (sort of what
>>> tip-bot
>>> and akpm's scripts are doing these days) to automatically notify the
>>> patch
>>> author that the patch has been merged and pushed out.
>>>
>>> Suggested attendance: Konstantin, maintainers :)
>>
>>
>> I wouldn't mind something like patchwork for lkml.  Since I use gmail
>> (sigh), it's a minor PITA to pull out patchbombs in git am-able
>> format.
>>
> What is wrong with https://patchwork.kernel.org/project/LKML/list/ ?

It's not spelled "linux-kernel".

/me runs away

--Andy

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Geert Uytterhoeven]

On Wed, Jul 8, 2015 at 1:31 AM, Andy Lutomirski <luto@amacapital.net> wrote:
> On Tue, Jul 7, 2015 at 1:42 PM, Jiri Kosina <jkosina@suse.com> wrote:
>> I think it'd be useful to have a session where maintainers could come up
>> with feature / improvement requests for kernel.org infrastructure and have
>> a subsequent discussion about whether they would be generally useful.
>>
>> Let me start with my personal wish:
>>
>> I personally would very welcome a facility that'd send out an e-mail if a
>> new commit is pushed to a git.kernel.org repo branch (sort of what tip-bot
>> and akpm's scripts are doing these days) to automatically notify the patch
>> author that the patch has been merged and pushed out.
>>
>> Suggested attendance: Konstantin, maintainers :)
>
> I wouldn't mind something like patchwork for lkml.  Since I use gmail
> (sigh), it's a minor PITA to pull out patchbombs in git am-able
> format.

For backup purposes, I run getmail from cron to download all my gmail
email.

To pull out a patchbomb:
  1. Select "Show original" in gmail for the cover letter,
  2. Copy Message-Id,
  3. Run "gmail-extract-patch-series message-id /tmp/cool-patch-series-"
     from the directory that contains the downloaded emails (I use one
     directory per day),
  4. git am -s /tmp/cool-patch-series-*.patch

$ cat $(type -p gmail-extract-patch-series)
#!/bin/bash

set -e

function usage()
{
        cat <<END

Usage: $(basename $0) <message-id> <prefix> [ <files> ... ]

END
        exit -1
}


if [ $# -lt 2 ]; then
        usage
fi

mid=$1
prefix=$2
shift
shift

if [ $# -lt 1 ]; then
        files=*
else
        files=$*
fi

for src in $(grep -l $mid $files); do
        if grep -m 1 "^Subject: Re: " $src > /dev/null; then
                echo Ignoring reply in $src
        else
                i=$(grep ^Subject: $src | sed -e 's@/.*$@@g' -e 's@^.* @@g')
                case $i in
                0|00|000) # Sending more than 999 patches is insane
                       suffix=txt
                        ;;
                *)
                        suffix=patch
                        ;;
                esac
                dst=$prefix$i.$suffix
                echo Extracting \"$(grep ^Subject $src | sed -e
's@Subject: @@g')\" to $dst
                cp -i $src $dst
        fi
done
$

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[James Bottomley]

On Tue, 2015-07-07 at 22:42 +0200, Jiri Kosina wrote:
> I think it'd be useful to have a session where maintainers could come up 
> with feature / improvement requests for kernel.org infrastructure and have 
> a subsequent discussion about whether they would be generally useful.
> 
> Let me start with my personal wish:
> 
> I personally would very welcome a facility that'd send out an e-mail if a 
> new commit is pushed to a git.kernel.org repo branch (sort of what tip-bot 
> and akpm's scripts are doing these days) to automatically notify the patch 
> author that the patch has been merged and pushed out.
> 
> Suggested attendance: Konstantin, maintainers :)

I can give you my scripts: they send email when patches are added and
dropped from the SCSI git trees.  The only problem is you need a
non-kernel.org server to run them on.  It's a simple perl script
(attached).

James

---

#!/usr/bin/perl
#
# An example hook script to mail out commit update information.
# It can also blocks tags that aren't annotated.
# Called by git-receive-pack with arguments: refname sha1-old sha1-new
#
# To enable this hook, make this file executable by "chmod +x update".
#
# Config
# ------
# hooks.mailinglist
#   This is the list that all pushes will go to; leave it blank to not send
#   emails frequently.  The log email will list every log entry in full between
#   the old ref value and the new ref value.
# hooks.announcelist
#   This is the list that all pushes of annotated tags will go to.  Leave it
#   blank to just use the mailinglist field.  The announce emails list the
#   short log summary of the changes since the last annotated tag
# hooks.allowunannotated
#   This boolean sets whether unannotated tags will be allowed into the
#   repository.  By default they won't be.
#
# Notes
# -----
# All emails have their subjects prefixed with "[SCM]" to aid filtering.
# All emails include the headers "X-Git-Refname", "X-Git-Oldrev",
# "X-Git-Newrev", and "X-Git-Reftype" to enable fine tuned filtering and info.

# --- Constants
chomp($cwd=`pwd`);
require $cwd.'/hooks/update-variables.pl' || die;

# branches which comprise the base of the tree (i.e. ones not to
# report commits in) for a push of master.  The default is 'linus'.
# 'merge-base' only gets used when the tree has to be based on a
# non-standard tree because of conflicts
%BRANCHES = (
    'merge-base' => 1,
    'linus' => 1,
);
%IGNORED_BRANCHES = (
    'for-next' => 1,
    'for-linus' => 1,
    'master' => 1,
);
@IGNOREDCC = (
    'stable@kernel.org',
    'stable@vger.kernel.org',
);

%EMAILTO = (
    'James.Bottomley@HansenPartnership.com' => 1,
);

# --- Command line
$refname = $ARGV[0];
if ($refname =~ m|^refs/tags/|) {
    $refname =~ s|^refs/tags/||;
    $tag = 1;
} else {
    $refname =~ s|^refs/heads/||;
}
$oldrev=$ARGV[1];
$newrev=$ARGV[2];

# --- Safety check
if ($ENV{'GIT_DIR'} eq '') {
	print STDERR "Don't run this script from the command line.";
	print STDERR " (if you want, you could supply GIT_DIR then run";
	print STDERR "  $0 <ref> <oldrev> <newrev>)";
	exit 1;
}

if ($refname eq '' || $oldrev eq '' || $newrev eq '') {
	print STDERR "Usage: $0 <ref> <oldrev> <newrev>";
	exit 1;
}

if ($tag) {
    chomp(@t = `git cat-file tag $newrev`);
    foreach (@t) {
	if (/^tagger (.*>)/) {
	    $recipients = $1;
	    break;
	}
    }

    if (!$recipients) {
	print STDERR "Can't find a tagger in $refname\n";
	exit 1;
    }

    if (%EMAILTO) {
	$EMAILTO{$recipients} = 1;

	$recipients = join(', ', keys(%EMAILTO));
    }

    chomp(@_ = `git merge-base $newrev master`);

    $oldrev = $_[0];

    chomp(@commitlist = `git rev-list $newrev ^$oldrev`);

    @commitlog = ();
    foreach $commit (@commitlist) {
	chomp(@_ = `git show --oneline $commit`);
	push @commitlog, $_[0];
    }
    @email = (
# Generate header
	"From: James Bottomley <jejb\@kernel.org>",
	"To: $recipients",
	"Subject: Tag $revname added to tree ${TREE}",
	"X-Git-Oldrev: $oldrev",
	"X-Git-Newrev: $newrev",
	"X-Git-Tree: $TREEHDR",
	"",
	"Your tag $refname",
	"",
	"Containing:",
	);
    push @email, @commitlog;
    push @email, (
	"",
	"has been added to the $TREETYPE $TREE tree",
	"",
	"You can find it here:",
	"",
	"http://git.kernel.org/?p=linux/kernel/git/jejb/${TREE}.git;a=tag;h=$newrev",
	"",
	$WHENPUSH,
	"",
	"James Bottomley",
	"",
	"P.S. If you find this email unwanted, set up a procmail rule junking on",
	"the header:",
	"",
	"X-Git-Tree: $TREEHDR",
	"",
    );
    open(EMAIL, "| /usr/sbin/sendmail -t") || die;
    print EMAIL join("\n", @email);
    close(EMAIL);
    #print STDERR join("\n", @email);
    print STDERR "Email sent to: $recipients\n";
    exit 0;
}

if ($IGNORED_BRANCHES{$refname}) {
    print STDERR "Branch $refname ignored, not generating email\n";
    exit 0;
}
if ($refname =~ m/-base$/ || $BRANCHES{$refname}) {
    print STDERR "Updating base branch\n";
    exit 0;
}

$branch = '';
chomp(@branches = `git branch`);

if ($refname eq 'master') {
    foreach $b (keys(%BRANCHES)) {
	next if (!grep(/^..$b/,@branches));
	$branch = $b;
	last;
    }
} elsif (grep(/^..${refname}-base/,@branches)) {
    $branch = $refname.'-base';
}

if (!$branch) {
    print STDERR 'Upstream head has no needed bases: '.join(" ", @BRANCHES)." or ${refname}-base\n";
    exit 1;
}

if ($oldrev eq '0000000000000000000000000000000000000000' ) {
    print STDERR "Creating new branch $refname\n";
    exit 0;
}

@commitlist = ();
chomp(@revlist = `git rev-list $newrev ^$oldrev ^$branch`);
chomp(@cherrylist = `git cherry $refname $newrev`);

# don't annoy Linus guard: check that all of the cherrylist is actually mine
foreach $commit (@revlist) {
    # - prefix means the commit from $newrev is already in $branch
    next if (grep(/^- $commit/, @cherrylist));
    push @commitlist, $commit;
    $_ = `git log --pretty='format:%cn' $commit^..$commit`;
    next if (/^James Bottomley/);
    next if (/^Christoph Hellwig/);
    print STDERR "ERROR: commit $commit is not yours\n";
    print STDERR "ERROR: $_\n";
    exit 1;
}

$WHENPUSH = $WHENPUSHED{$refname};
if (!$WHENPUSH) {
    $WHENPUSH = $WHENPUSHED{'default'};
}
$NEEDACKS = $NEEDACKS{$refname};

#find the deleted commits and add them to the commit list
chomp(@deletelist = `git cherry $newrev $refname $branch`);
foreach $_ (@deletelist) {
    next if (!m/^\+ /);
    # found a delete; add it to the email list keeping the + prefix
    push @commitlist, $_;
}

foreach $commit (@commitlist) {
    if ($commit =~ m/^\+ /) {
	$commit = substr($commit,2);
	$delete = 1;
	$EMAILPREFIX="Patch dropped from ${TREE}: ";
    } else {
	$delete = 0;
	$EMAILPREFIX="Patch added to ${TREE}: ";
    }
    
    @email = ();
    $author = '';
    if (%EMAILTO) {
	%recipients = %EMAILTO;
    }
    %ackrequired = ();
    $subject = '';

    outer: foreach $_ (`git log $commit^..$commit`) {
	if (/^    / && $subject eq '') {
	    chomp($subject = substr($_, 4));
	} elsif (/^Author: / && $author eq '') {
	    chomp($author = substr($_, 8));
	    $recipients{$author} = 1;
	} elsif (/^    signed-off-by: /i) {
	    chomp($a = substr($_, 19));
	    $recipients{$a} = 1;
	    delete $ackrequired{$a};
	} elsif (/^    reviewed-by: /i) {
	    chomp($a = substr($_, 16));
	    $recipients{$a} = 1;
	    delete $ackrequired{$a};
	} elsif(/^    acked-by: /i) {
	    chomp($a = substr($_, 14));
	    $recipients{$a} = 1;
	    delete $ackrequired{$a};
	} elsif(/^    tested-by: /i) {
	    chomp($a = substr($_, 15));
	    $recipients{$a} = 1;
	    delete $ackrequired{$a};
	} elsif(/^    reported-by: /i) {
	    chomp($a = substr($_, 17));
	    $recipients{$a} = 1;
	    delete $ackrequired{$a};
	} elsif (/^    cc: /i) {
	    chomp($a = substr($_, 8));
	    foreach(@IGNOREDCC) {
		if ($a =~ m/$_/) {
		    next outer;
		}
	    }
	    $recipients{$a} = 1;
	    $ackrequired{$a} = 1;
	}
    }
    # I don't want to receive email
    #delete $recipients{'James Bottomley <James.Bottomley@SteelEye.com>'};
    # --- Email (all stdout will be the email)
    $recipients = join(', ', keys(%recipients));
    #$recipients = 'James.Bottomley@HansenPartnership.com';
    
    @email = (
# Generate header
	      "From: James Bottomley <jejb\@kernel.org>",
	      "To: $recipients",
	      "Subject: ${EMAILPREFIX} $subject",
	      "X-Git-Oldrev: $oldrev",
	      "X-Git-Newrev: $newrev",
	      "X-Git-Tree: $TREEHDR",
	      "",
# body goes here
	      "Your commit:",
	      );
    chomp(@commitlog = `git log $commit^..$commit`);
    # get rid of the commit/author/date header
    shift @commitlog;
    shift @commitlog;
    shift @commitlog;
    push @email, @commitlog;
    if ($delete) {
	push @email, (
	    "",
	    "has been dropped from the $TREETYPE $TREE tree",
	    "On branch \"$refname\"",
	    "",
	);
    } else {
	push @email, (
	      "",
	      "has been added to the $TREETYPE $TREE tree",
	      "On branch \"$refname\"",
	      "You can find it here:",
	      "",
	      "http://git.kernel.org/?p=linux/kernel/git/jejb/${TREE}.git;a=commit;h=$commit",
	      "",
	      $WHENPUSH,
	);
    }
    if ($NEEDACKS && %ackrequired) {
	push @email, (
	       "",
	       "This patch is pending because it requires ACKs from:",
	       "",
	       join("\n", keys(%ackrequired)),
	       "",
	       "If those are received it may be moved into one of the upstream SCSI trees",
		      );
    }
    push @email, (
	      "",
	      "James Bottomley",
	      "",
	      "P.S. If you find this email unwanted, set up a procmail rule junking on",
	      "the header:",
	      "",
	      "X-Git-Tree: $TREEHDR",
	      "",
	      );
    open(EMAIL, "| /usr/sbin/sendmail -t") || die;
    print EMAIL join("\n", @email);
    close(EMAIL);
    #print STDERR join("\n", @email);
    print STDERR "Email sent to: $recipients\n";
}

exit 0

Re: [Ksummit-discuss] [CORE TOPIC] services needed from kernel.org infrastructure

[Konstantin Ryabitsev]

[-- Attachment #1: Type: text/plain, Size: 2128 bytes --]

On Tue, Jul 07, 2015 at 10:42:49PM +0200, Jiri Kosina wrote:
> I think it'd be useful to have a session where maintainers could come up 
> with feature / improvement requests for kernel.org infrastructure and have 
> a subsequent discussion about whether they would be generally useful.

I suggest people just email users@linux.kernel.org. :) I've always
solicited such conversations, but in my previous experience these tend
to fall into two categories:

1. "Here's a minor suggestion that could be kinda cool, whenever you
   have some time."
2. "Here's a radical idea that would require a park of 1,000 various
   hardware systems and a dedicated staff of 10 people."

There doesn't appear to be a lot that falls between these two
categories. This is not to say that #2 is not possible, it's just that
we have to operate within LF budget constraints and people never tend to
follow such grand requests with "and here's my $EMPLOYER who's happy to
fund this initiative." If you do happen to be in such a position, then
I'll be happy to get you in touch with the Collab Projects management
team.

> Let me start with my personal wish:
> 
> I personally would very welcome a facility that'd send out an e-mail if a 
> new commit is pushed to a git.kernel.org repo branch (sort of what tip-bot 
> and akpm's scripts are doing these days) to automatically notify the patch 
> author that the patch has been merged and pushed out.

We have the following script running on torvalds/linux.git:
http://ur1.ca/n2iof

It has two downsides:

1. It sends an email per each commit, which tends to thrash the lists
   (e.g. see https://marc.info/?l=git-commits-head where Linus's tree goes)
2. It doesn't fork into background while doing that, so you have to wait
   for it to finish sending the emails.

If that's fine with you all, I can enable this for other repos as well.
All I'd need would be the destination list address.

> Suggested attendance: Konstantin, maintainers :)

I'll be at the summit.

Best,
-- 
Konstantin Ryabitsev
Linux Foundation Collab Projects
Montréal, Québec

[-- Attachment #2: Type: application/pgp-signature, Size: 473 bytes --]