[Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Takashi Iwai]

I think this has been already raised a few times, but I'm still
dreaming one thing in our git management: having some metadata
collection / link for each commit.

I don't mean for a thing like post-commit sign-off, but rather for
tracking the information that has been revealed after commit, e.g. a
regression the commit causes, the later fix commit, or a bugzilla or
ML link for the further discussion or debugging session.  The
regression markers would be especially helpful to avoid a pitfall
through performing bisection.

IMO, it would be convenient if such information can be embedded in the
published git tree, something like (infamous) git-notes.  But, I'd
like to hear any other options as well.


thanks,

Takashi

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1503 bytes --]

On Tue, 13 May 2014 15:25:57 +0200 Takashi Iwai <tiwai@suse.de> wrote:

> I think this has been already raised a few times, but I'm still
> dreaming one thing in our git management: having some metadata
> collection / link for each commit.
> 
> I don't mean for a thing like post-commit sign-off, but rather for
> tracking the information that has been revealed after commit, e.g. a
> regression the commit causes, the later fix commit, or a bugzilla or
> ML link for the further discussion or debugging session.  The
> regression markers would be especially helpful to avoid a pitfall
> through performing bisection.
> 
> IMO, it would be convenient if such information can be embedded in the
> published git tree, something like (infamous) git-notes.  But, I'd
> like to hear any other options as well.
> 

+1

A particular practical issue is that when doing a git-bisect there might be a
range of commits that only compile/run if some later commit is applied.
If I'm bisecting in that range, I have to repeatedly apply that commit by
hand.

If would really like it if a new commit could be marked as "a really
important successor to the first parent" so that git could keep the extra
metadata to easily find the parent->child link, and so that "git-bisect"
could be told --always-include-important-children

This linkage would be enough to add anything else as "important successors".

Yes, I know I should send a patch rather than a suggestion.  Sorry.

NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Jiri Kosina]

On Wed, 14 May 2014, NeilBrown wrote:

> A particular practical issue is that when doing a git-bisect there might 
> be a range of commits that only compile/run if some later commit is 
> applied. If I'm bisecting in that range, I have to repeatedly apply that 
> commit by hand.

You can just have it stored in a separate branch and perform the bisection 
using a script that will always first merge the branch, and reset after 
good/bad verification is done.

If the fix is already applied (because you are testing a state of the tree 
which has the commit already), git handles the merge of the 
already-applied patch with grace even if they don't have the same SHA-1.

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 1152 bytes --]

On Wed, 14 May 2014 01:29:29 +0200 (CEST) Jiri Kosina <jkosina@suse.cz> wrote:

> On Wed, 14 May 2014, NeilBrown wrote:
> 
> > A particular practical issue is that when doing a git-bisect there might 
> > be a range of commits that only compile/run if some later commit is 
> > applied. If I'm bisecting in that range, I have to repeatedly apply that 
> > commit by hand.
> 
> You can just have it stored in a separate branch and perform the bisection 
> using a script that will always first merge the branch, and reset after 
> good/bad verification is done.
> 
> If the fix is already applied (because you are testing a state of the tree 
> which has the commit already), git handles the merge of the 
> already-applied patch with grace even if they don't have the same SHA-1.
> 

True.  But every extra step I have to add to a script is an extra barrier to
performing what is already a somewhat tedious task.

If fixes were always Strongly-Attached to the commit they fix, then I
wouldn't even need to find these "fix-up" patches.  One person commits the
fix in the "right" way, and everyone benefits.

Thanks,
NeilBrown

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Li Zefan]

On 2014/5/13 21:25, Takashi Iwai wrote:
> I think this has been already raised a few times, but I'm still
> dreaming one thing in our git management: having some metadata
> collection / link for each commit.
> 
> I don't mean for a thing like post-commit sign-off, but rather for
> tracking the information that has been revealed after commit, e.g. a
> regression the commit causes, the later fix commit, or a bugzilla or
> ML link for the further discussion or debugging session.  The
> regression markers would be especially helpful to avoid a pitfall
> through performing bisection.
> 

Yeah, this is very useful.

For example, instead of adding a tag in the changelog to point to
the guilt commit that introduces the bug:

	Fixes: commit_id ("Subject")

It's much more convenient and useful if we can add metadata to the
guilt commit that points to later fixes, and then we simply ask
git to show us those fixes. This helps a lot to people who need to
backport things to older kernels.

> IMO, it would be convenient if such information can be embedded in the
> published git tree, something like (infamous) git-notes.  But, I'd
> like to hear any other options as well.
> 

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Jason Cooper]

Takashi,

On Tue, May 13, 2014 at 03:25:57PM +0200, Takashi Iwai wrote:
> I think this has been already raised a few times, but I'm still
> dreaming one thing in our git management: having some metadata
> collection / link for each commit.
> 
> I don't mean for a thing like post-commit sign-off, but rather for
> tracking the information that has been revealed after commit, e.g. a
> regression the commit causes, the later fix commit, 

For the stuff flying by me, I've been adding the:

Fixes: <12-char hash>: ('Offending patch subject')

On patches fixing a regression.  It helps the stable team (when Cc
stable is also added) and in your scenario, you could grep the commits
for the result of your bisect.

> or a bugzilla or

Sorry, I don't use it.

> ML link for the further discussion or debugging session.

We've also been autogenerating a tag:

Link: https://lkml.kernel.org/r/<Message-Id>

For all patches that points to the email the patch came from.  Not
exactly what you were looking for, but nothing prevents someone from
replying to that thread a year later with a regression report.

I've also been contemplating adding

Coverletter: https://lkml.kernel.org/r/<First Reference Msg-Id>

for large series where the patch submitter has done a thorough writeup
in the coverletter.

thx,

Jason.

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Christian Couder]

Hi,

From: Jason Cooper <jason@lakedaemon.net>

> Takashi,
> 
> On Tue, May 13, 2014 at 03:25:57PM +0200, Takashi Iwai wrote:
>> I think this has been already raised a few times, but I'm still
>> dreaming one thing in our git management: having some metadata
>> collection / link for each commit.
>> 
>> I don't mean for a thing like post-commit sign-off, but rather for
>> tracking the information that has been revealed after commit, e.g. a
>> regression the commit causes, the later fix commit, 
> 
> For the stuff flying by me, I've been adding the:
> 
> Fixes: <12-char hash>: ('Offending patch subject')
> 
> On patches fixing a regression.  It helps the stable team (when Cc
> stable is also added) and in your scenario, you could grep the commits
> for the result of your bisect.
> 
>> or a bugzilla or
> 
> Sorry, I don't use it.
> 
>> ML link for the further discussion or debugging session.
> 
> We've also been autogenerating a tag:
> 
> Link: https://lkml.kernel.org/r/<Message-Id>
> 
> For all patches that points to the email the patch came from.  Not
> exactly what you were looking for, but nothing prevents someone from
> replying to that thread a year later with a regression report.
> 
> I've also been contemplating adding
> 
> Coverletter: https://lkml.kernel.org/r/<First Reference Msg-Id>
> 
> for large series where the patch submitter has done a thorough writeup
> in the coverletter.

I have been working on a new git command "git interpret-trailers" to
help people add the above kind of trailer lines:

http://thread.gmane.org/gmane.comp.version-control.git/245874/

This was started by the following thread on the git mailing list:

http://thread.gmane.org/gmane.comp.version-control.git/236770/

which itself was started by decisions at Linux Kernel Summit 2013.

About bisecting with untestable commits, I suggest preparing fixed up
branches where the bug that prevents testing is fixed and then using
"git replace" to tell Git to use the fixed up branch instead of the
original one. This way you can easily share these fixed up branches.

Here is a presentation I give about "git replace": 

http://www.slideshare.net/ChristianCouder/new-views-onyourhistorywithgitr

where using fixed up branches to bisect is described starting at slide
19.

(By the way, I gave that presentation at LinuxTag last week in Berlin
and I proposed it for LinuxCon North America next August.)

Best,
Christian.

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Li Zefan]

On 2014/5/16 11:07, Jason Cooper wrote:
> Takashi,
> 
> On Tue, May 13, 2014 at 03:25:57PM +0200, Takashi Iwai wrote:
>> I think this has been already raised a few times, but I'm still
>> dreaming one thing in our git management: having some metadata
>> collection / link for each commit.
>>
>> I don't mean for a thing like post-commit sign-off, but rather for
>> tracking the information that has been revealed after commit, e.g. a
>> regression the commit causes, the later fix commit, 
> 
> For the stuff flying by me, I've been adding the:
> 
> Fixes: <12-char hash>: ('Offending patch subject')
> 
> On patches fixing a regression.  It helps the stable team (when Cc
> stable is also added) and in your scenario, you could grep the commits
> for the result of your bisect.
> 

Yeah, but the problem with this is you can't add this "Fixes" tag
to git-log after the patch has been applied to the git tree.

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Takashi Iwai]

At Thu, 15 May 2014 23:07:08 -0400,
Jason Cooper wrote:
> 
> Takashi,
> 
> On Tue, May 13, 2014 at 03:25:57PM +0200, Takashi Iwai wrote:
> > I think this has been already raised a few times, but I'm still
> > dreaming one thing in our git management: having some metadata
> > collection / link for each commit.
> > 
> > I don't mean for a thing like post-commit sign-off, but rather for
> > tracking the information that has been revealed after commit, e.g. a
> > regression the commit causes, the later fix commit, 
> 
> For the stuff flying by me, I've been adding the:
> 
> Fixes: <12-char hash>: ('Offending patch subject')
> 
> On patches fixing a regression.  It helps the stable team (when Cc
> stable is also added) and in your scenario, you could grep the commits
> for the result of your bisect.

Yeah, that was suggested in the last year's KS, and helps some cases.

However, the problem is that people(*1) often notice this too late
after the tree has been already published.  Also, some information
(e.g. bug reports) can come only after commits.

(*1) statistics taken from one person :)

And, for bisection, we need some reverse mapping for efficiency.
It'd take time to look all commit logs from each revlist, especially
if the bisection is done for the early history.

I tried a hackish way once ago: keeping simple text files named with
$SHAID in a separate branch, and refers to it at git log or bisect
time.  There must be much elegant way, I suppose, though.

> > or a bugzilla or
> 
> Sorry, I don't use it.
> 
> > ML link for the further discussion or debugging session.
> 
> We've also been autogenerating a tag:
> 
> Link: https://lkml.kernel.org/r/<Message-Id>
> 
> For all patches that points to the email the patch came from.  Not
> exactly what you were looking for, but nothing prevents someone from
> replying to that thread a year later with a regression report.
> 
> I've also been contemplating adding
> 
> Coverletter: https://lkml.kernel.org/r/<First Reference Msg-Id>
> 
> for large series where the patch submitter has done a thorough writeup
> in the coverletter.

Yes, this kind of information is helpful for checking patches at later
point, indeed.


thanks,

Takashi

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Christian Couder]

From: Takashi Iwai <tiwai@suse.de>
>
> At Thu, 15 May 2014 23:07:08 -0400,
> Jason Cooper wrote:
>> 
>> Takashi,
>> 
>> On Tue, May 13, 2014 at 03:25:57PM +0200, Takashi Iwai wrote:
>> > I think this has been already raised a few times, but I'm still
>> > dreaming one thing in our git management: having some metadata
>> > collection / link for each commit.
>> > 
>> > I don't mean for a thing like post-commit sign-off, but rather for
>> > tracking the information that has been revealed after commit, e.g. a
>> > regression the commit causes, the later fix commit, 
>> 
>> For the stuff flying by me, I've been adding the:
>> 
>> Fixes: <12-char hash>: ('Offending patch subject')
>> 
>> On patches fixing a regression.  It helps the stable team (when Cc
>> stable is also added) and in your scenario, you could grep the commits
>> for the result of your bisect.
> 
> Yeah, that was suggested in the last year's KS, and helps some cases.
> 
> However, the problem is that people(*1) often notice this too late
> after the tree has been already published.  Also, some information
> (e.g. bug reports) can come only after commits.

It is possible to use "git notes" or even "git replace" to add
information to existing commits.

> (*1) statistics taken from one person :)
> 
> And, for bisection, we need some reverse mapping for efficiency.
> It'd take time to look all commit logs from each revlist, especially
> if the bisection is done for the early history.
> 
> I tried a hackish way once ago: keeping simple text files named with
> $SHAID in a separate branch, and refers to it at git log or bisect
> time.  There must be much elegant way, I suppose, though.

Yeah, using "git replace" is more elegant. And there will be hopefully
soon the --edit option that will make it very easy to use git replace.

Best,
Christian.

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Jason Cooper]

On Sun, May 18, 2014 at 09:23:15PM +0200, Christian Couder wrote:
> From: Takashi Iwai <tiwai@suse.de>
...
> > I tried a hackish way once ago: keeping simple text files named with
> > $SHAID in a separate branch, and refers to it at git log or bisect
> > time.  There must be much elegant way, I suppose, though.
> 
> Yeah, using "git replace" is more elegant. And there will be hopefully
> soon the --edit option that will make it very easy to use git replace.

Yikes.  Sorry, but I don't like that at all.  I hope that feature is
disabled by default (git replace).

I guess my first question is: Does the PGP signature verification of a
signed tag fail if there are replaced commits in the history?  And if
so, is that a cryptographic failure, or a logical failure?

thx,

Jason.

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Christian Couder]

From: Jason Cooper <jason@lakedaemon.net>

> On Sun, May 18, 2014 at 09:23:15PM +0200, Christian Couder wrote:
>> From: Takashi Iwai <tiwai@suse.de>
> ...
>> > I tried a hackish way once ago: keeping simple text files named with
>> > $SHAID in a separate branch, and refers to it at git log or bisect
>> > time.  There must be much elegant way, I suppose, though.
>> 
>> Yeah, using "git replace" is more elegant. And there will be hopefully
>> soon the --edit option that will make it very easy to use git replace.
> 
> Yikes.  Sorry, but I don't like that at all.  I hope that feature is
> disabled by default (git replace).

By default replace refs are not fetched and not pushed. So you only
have the replace refs you created yourself on your own repo. They are
only fetched or pushed if you use or configure a refspec like
'refs/replace/*:refs/replace/*'.

You can also use the GIT_NO_REPLACE_OBJECTS environment variable or
"alias git='git --no-replace-objects'", if/when you want to make sure that
replace refs are not used.

A hook could also check that for example each replace ref you add to
your repo replaces a commit with another commit that points to the
same tree.

You could also use 2 different repos, one where you do most of your
work and don't have any replace ref, and one where you do your
bisecting and where you have replace refs. You can create, fetch and
push replace refs in your bisecting repo.

> I guess my first question is: Does the PGP signature verification of a
> signed tag fail if there are replaced commits in the history?

First any kind of Git object can be replaced, not just commits, even
tag objects. Also I think the PGP signature verification only checks
the content of the tag for signed tags or the content of the commit
for signed commits.

So it might be a good idea to disable replace refs (for example using
the GIT_NO_REPLACE_OBJECTS environment variable or
--no-replace-objects) when you want to check signatures, and maybe Git
could warn or error out if the commit or tags that are being verified
are replaced.

But if you have replace refs and some blobs or trees are replaced, you
have to trust these replace refs. (And maybe you are very right to
trust them because for example you created them yourself and never
fetched any replace ref.)

Anyway maybe we should cc Junio and other people on this because
signing is not a part of Git that I know well.

> And if
> so, is that a cryptographic failure, or a logical failure?

I don't think there is any failure.

Some things are inherently risky. For example if you bisect kernels,
you might compile and run kernels that have known security bugs, so
you have to be careful.

Best,
Christian.

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Takashi Iwai]

At Sun, 18 May 2014 21:23:15 +0200 (CEST),
Christian Couder wrote:
> 
> From: Takashi Iwai <tiwai@suse.de>
> >
> > At Thu, 15 May 2014 23:07:08 -0400,
> > Jason Cooper wrote:
> >> 
> >> Takashi,
> >> 
> >> On Tue, May 13, 2014 at 03:25:57PM +0200, Takashi Iwai wrote:
> >> > I think this has been already raised a few times, but I'm still
> >> > dreaming one thing in our git management: having some metadata
> >> > collection / link for each commit.
> >> > 
> >> > I don't mean for a thing like post-commit sign-off, but rather for
> >> > tracking the information that has been revealed after commit, e.g. a
> >> > regression the commit causes, the later fix commit, 
> >> 
> >> For the stuff flying by me, I've been adding the:
> >> 
> >> Fixes: <12-char hash>: ('Offending patch subject')
> >> 
> >> On patches fixing a regression.  It helps the stable team (when Cc
> >> stable is also added) and in your scenario, you could grep the commits
> >> for the result of your bisect.
> > 
> > Yeah, that was suggested in the last year's KS, and helps some cases.
> > 
> > However, the problem is that people(*1) often notice this too late
> > after the tree has been already published.  Also, some information
> > (e.g. bug reports) can come only after commits.
> 
> It is possible to use "git notes" or even "git replace" to add
> information to existing commits.

Yes, I know of git-notes, as already mentioned in the first post.  But
the problem is that publishing and importing git-notes changes isn't
well established for kernel git tree management.  And, IIRC, Linus
didn't like its usage somehow.  Not sure whether it's because of
git-notes technical design or its concept...

> > (*1) statistics taken from one person :)
> > 
> > And, for bisection, we need some reverse mapping for efficiency.
> > It'd take time to look all commit logs from each revlist, especially
> > if the bisection is done for the early history.
> > 
> > I tried a hackish way once ago: keeping simple text files named with
> > $SHAID in a separate branch, and refers to it at git log or bisect
> > time.  There must be much elegant way, I suppose, though.
> 
> Yeah, using "git replace" is more elegant. And there will be hopefully
> soon the --edit option that will make it very easy to use git replace.

We don't want to change the history at all.  A preferred option is
just addendum on top of the existing commits, and the way to easily
share the change (at best with the normal git pull).  If git-replace
provides such a good integration, I'd love to see it in our use case.


thanks,

Takashi

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Christian Couder]

From: Takashi Iwai <tiwai@suse.de>

> At Sun, 18 May 2014 21:23:15 +0200 (CEST),
> Christian Couder wrote:
>> 
>> From: Takashi Iwai <tiwai@suse.de>
>> >
>> > At Thu, 15 May 2014 23:07:08 -0400,
>> > Jason Cooper wrote:
>> >> 
>> >> Takashi,
>> >> 
>> >> On Tue, May 13, 2014 at 03:25:57PM +0200, Takashi Iwai wrote:
>> >> > I think this has been already raised a few times, but I'm still
>> >> > dreaming one thing in our git management: having some metadata
>> >> > collection / link for each commit.
>> >> > 
>> >> > I don't mean for a thing like post-commit sign-off, but rather for
>> >> > tracking the information that has been revealed after commit, e.g. a
>> >> > regression the commit causes, the later fix commit, 
>> >> 
>> >> For the stuff flying by me, I've been adding the:
>> >> 
>> >> Fixes: <12-char hash>: ('Offending patch subject')
>> >> 
>> >> On patches fixing a regression.  It helps the stable team (when Cc
>> >> stable is also added) and in your scenario, you could grep the commits
>> >> for the result of your bisect.
>> > 
>> > Yeah, that was suggested in the last year's KS, and helps some cases.
>> > 
>> > However, the problem is that people(*1) often notice this too late
>> > after the tree has been already published.  Also, some information
>> > (e.g. bug reports) can come only after commits.
>> 
>> It is possible to use "git notes" or even "git replace" to add
>> information to existing commits.
> 
> Yes, I know of git-notes, as already mentioned in the first post.  But
> the problem is that publishing and importing git-notes changes isn't
> well established for kernel git tree management.  And, IIRC, Linus
> didn't like its usage somehow.  Not sure whether it's because of
> git-notes technical design or its concept...

Maybe Linus just doesn't want to be bothered with maintaining
notes/replace refs in his git repos. But these refs could be
maintained by other people in other repos.

>> > (*1) statistics taken from one person :)
>> > 
>> > And, for bisection, we need some reverse mapping for efficiency.
>> > It'd take time to look all commit logs from each revlist, especially
>> > if the bisection is done for the early history.
>> > 
>> > I tried a hackish way once ago: keeping simple text files named with
>> > $SHAID in a separate branch, and refers to it at git log or bisect
>> > time.  There must be much elegant way, I suppose, though.
>> 
>> Yeah, using "git replace" is more elegant. And there will be hopefully
>> soon the --edit option that will make it very easy to use git replace.
> 
> We don't want to change the history at all.

What I understood is that you want to be able to bisect on a history
where many bugs that prevent testing can easily be fixed.

Using git replace, it can go like this:

- a developer learns that some part of the history can be fixed to
  bisect more easily by applying a patch,

- the developer recreates a part of the history where the patch has
  been applied (using git cherry-pick or git rebase -i for example)

- the developer uses git replace to create a replace ref so that the
  new part of the history is used instead of the old one

- the developer pushes this new replace ref to a common repo where
  replace refs are shared (this pushes the new part of the history
  too)

- other developers fetch the replace refs from the common repo before
  they bisect

- when these other developers bisect, the new parts of the history are
  automatically used, so bisecting is faster, easier and maybe safer
  too

I think that it is better than when everyone changes the history
manually by applying patches in the middle of a bisection.

> A preferred option is
> just addendum on top of the existing commits, and the way to easily
> share the change (at best with the normal git pull).  If git-replace
> provides such a good integration, I'd love to see it in our use case.

git notes is probably better if what you want is only addendum on top
of the existing commits. git notes creates notes refs that you can
easily share.

Best,
Christian.

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Takashi Iwai]

At Tue, 20 May 2014 08:37:18 +0200 (CEST),
Christian Couder wrote:
> 
> From: Takashi Iwai <tiwai@suse.de>
> 
> > At Sun, 18 May 2014 21:23:15 +0200 (CEST),
> > Christian Couder wrote:
> >> 
> >> From: Takashi Iwai <tiwai@suse.de>
> >> >
> >> > At Thu, 15 May 2014 23:07:08 -0400,
> >> > Jason Cooper wrote:
> >> >> 
> >> >> Takashi,
> >> >> 
> >> >> On Tue, May 13, 2014 at 03:25:57PM +0200, Takashi Iwai wrote:
> >> >> > I think this has been already raised a few times, but I'm still
> >> >> > dreaming one thing in our git management: having some metadata
> >> >> > collection / link for each commit.
> >> >> > 
> >> >> > I don't mean for a thing like post-commit sign-off, but rather for
> >> >> > tracking the information that has been revealed after commit, e.g. a
> >> >> > regression the commit causes, the later fix commit, 
> >> >> 
> >> >> For the stuff flying by me, I've been adding the:
> >> >> 
> >> >> Fixes: <12-char hash>: ('Offending patch subject')
> >> >> 
> >> >> On patches fixing a regression.  It helps the stable team (when Cc
> >> >> stable is also added) and in your scenario, you could grep the commits
> >> >> for the result of your bisect.
> >> > 
> >> > Yeah, that was suggested in the last year's KS, and helps some cases.
> >> > 
> >> > However, the problem is that people(*1) often notice this too late
> >> > after the tree has been already published.  Also, some information
> >> > (e.g. bug reports) can come only after commits.
> >> 
> >> It is possible to use "git notes" or even "git replace" to add
> >> information to existing commits.
> > 
> > Yes, I know of git-notes, as already mentioned in the first post.  But
> > the problem is that publishing and importing git-notes changes isn't
> > well established for kernel git tree management.  And, IIRC, Linus
> > didn't like its usage somehow.  Not sure whether it's because of
> > git-notes technical design or its concept...
> 
> Maybe Linus just doesn't want to be bothered with maintaining
> notes/replace refs in his git repos. But these refs could be
> maintained by other people in other repos.

Maybe.  But just a guess at this point.


> >> > (*1) statistics taken from one person :)
> >> > 
> >> > And, for bisection, we need some reverse mapping for efficiency.
> >> > It'd take time to look all commit logs from each revlist, especially
> >> > if the bisection is done for the early history.
> >> > 
> >> > I tried a hackish way once ago: keeping simple text files named with
> >> > $SHAID in a separate branch, and refers to it at git log or bisect
> >> > time.  There must be much elegant way, I suppose, though.
> >> 
> >> Yeah, using "git replace" is more elegant. And there will be hopefully
> >> soon the --edit option that will make it very easy to use git replace.
> > 
> > We don't want to change the history at all.
> 
> What I understood is that you want to be able to bisect on a history
> where many bugs that prevent testing can easily be fixed.

Well, the bisection is just an example where the extra metadata would
help.  It's not the purpose itself.

> 
> Using git replace, it can go like this:
> 
> - a developer learns that some part of the history can be fixed to
>   bisect more easily by applying a patch,
> 
> - the developer recreates a part of the history where the patch has
>   been applied (using git cherry-pick or git rebase -i for example)
> 
> - the developer uses git replace to create a replace ref so that the
>   new part of the history is used instead of the old one
> 
> - the developer pushes this new replace ref to a common repo where
>   replace refs are shared (this pushes the new part of the history
>   too)
> 
> - other developers fetch the replace refs from the common repo before
>   they bisect
> 
> - when these other developers bisect, the new parts of the history are
>   automatically used, so bisecting is faster, easier and maybe safer
>   too
> 
> I think that it is better than when everyone changes the history
> manually by applying patches in the middle of a bisection.

This is nice, but I'm not sure whether it'll work / suit well with a
tree like kernel, which is the results of tons of merges from various
maintainers.  I guess little maintainers would publish the corrected
repo.  I hope my guess is wrong.

> > A preferred option is
> > just addendum on top of the existing commits, and the way to easily
> > share the change (at best with the normal git pull).  If git-replace
> > provides such a good integration, I'd love to see it in our use case.
> 
> git notes is probably better if what you want is only addendum on top
> of the existing commits. git notes creates notes refs that you can
> easily share.

IMO, it's not that easy to share git notes.  At least, it's not in the
same level as sharing tags.  If it were as easy as tags, I see no
reason to avoid using git-notes.  That said, having a separate refs
that requires the extra non-standard step looks like a big obstacle.


thanks,

Takashi

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Christian Couder]

From: Takashi Iwai <tiwai@suse.de>
>
> At Tue, 20 May 2014 08:37:18 +0200 (CEST),
> Christian Couder wrote:
>> 
>> From: Takashi Iwai <tiwai@suse.de>
>> 
>> > We don't want to change the history at all.
>> 
>> What I understood is that you want to be able to bisect on a history
>> where many bugs that prevent testing can easily be fixed.
> 
> Well, the bisection is just an example where the extra metadata would
> help.  It's not the purpose itself.
> 
>> 
>> Using git replace, it can go like this:
>> 
>> - a developer learns that some part of the history can be fixed to
>>   bisect more easily by applying a patch,
>> 
>> - the developer recreates a part of the history where the patch has
>>   been applied (using git cherry-pick or git rebase -i for example)
>> 
>> - the developer uses git replace to create a replace ref so that the
>>   new part of the history is used instead of the old one
>> 
>> - the developer pushes this new replace ref to a common repo where
>>   replace refs are shared (this pushes the new part of the history
>>   too)
>> 
>> - other developers fetch the replace refs from the common repo before
>>   they bisect
>> 
>> - when these other developers bisect, the new parts of the history are
>>   automatically used, so bisecting is faster, easier and maybe safer
>>   too
>> 
>> I think that it is better than when everyone changes the history
>> manually by applying patches in the middle of a bisection.
> 
> This is nice, but I'm not sure whether it'll work / suit well with a
> tree like kernel, which is the results of tons of merges from various
> maintainers.  I guess little maintainers would publish the corrected
> repo.  I hope my guess is wrong.

Yeah, I don't know if it's easy enough to redo many merges with
current Git. On the other hand, I don't see why it couldn't be made
easy if it isn't already.

About maintainers publishing the corrected/improved repo, I think if a
few developers publish one that provide some benefits, and come up
with a way/workflow to do so quite easily, then other will follow.

>> > A preferred option is
>> > just addendum on top of the existing commits, and the way to easily
>> > share the change (at best with the normal git pull).  If git-replace
>> > provides such a good integration, I'd love to see it in our use case.
>> 
>> git notes is probably better if what you want is only addendum on top
>> of the existing commits. git notes creates notes refs that you can
>> easily share.
> 
> IMO, it's not that easy to share git notes.  At least, it's not in the
> same level as sharing tags.  If it were as easy as tags, I see no
> reason to avoid using git-notes.  That said, having a separate refs
> that requires the extra non-standard step looks like a big obstacle.

I don't see why it is not easy to share git notes.

Basically I think that you just need to add refspecs like

	  push = +refs/notes/*:refs/notes/*

or

	  fetch = +refs/notes/*:refs/notes/*

in your config file, in the sections of the remotes you want to push
or fetch notes.

But we should probably CC Johan Herland <johan@herland.net> or Junio
when discussing this as I don't know notes well.

Best,
Christian.

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Christian Couder]

I told Junio and Johan about this discussion, and Junio, as he did not
find a good way to subscribe to the list as a newcomer and still send
a response to an existing thread, said that I can forward the
following.

Johan agrees with Junio about this. I don't agree with their opinion
about "replace".

From: Junio C Hamano <gitster@pobox.com>:

-- >8 --

Takashi Iwai says, in response to what Jason cooper wrote:

>> For the stuff flying by me, I've been adding the:
>> 
>> Fixes: <12-char hash>: ('Offending patch subject')
>> ...
>> I've also been contemplating adding
>> 
>> Coverletter: https://lkml.kernel.org/r/<First Reference Msg-Id>
>> 
>> for large series where the patch submitter has done a thorough
>> writeup
>> in the coverletter.
>
> Yes, this kind of information is helpful for checking patches at
> later point, indeed.

I tend to think that:

 * "replace" is too heavy-handed tool to use in general.  Nobody
   sane should publish history full of replacements for public
   consumption, especially if that is done merely to help
   bisection.

 * "notes" is very handy and may be an efficient mechanism to add
   information after the fact to existing commits, but merging two
   or more lines of notes histories together is cumbersome.

A good way forward to solve Iwai-san's original issue might be

 * Establish the "Fixes:" mentioned above as a standard practice.
   Polishing Christian's interpret-trailers tool might be a good way
   to encourage developers to do so.

 * Have an easy way for developers to scan incoming commits for
   these "Fixes:" footer, and record the reverse mapping locally, so
   that we can go from a commit whose brokenness is discovered later
   to the commit that fixes its breakage efficiently.  "notes" may
   be a good mechanism to implement this mapping, and we do not have
   to worry about sharing the notes trees among developers.

 * The information is visible with "log --show-notes" if it is
   stored in local notes.  When an earlier commit that was later
   found to be broken is shown, the note that points at the commit
   that fixes it will be shown.

 * Teach "bisect" to also take notice of this information, and
   temporarily cherry-pick while testing commits with fixes that
   were discovered later, in a way similar to what was suggested by
   Jiri Kosina in an earlier message.

-- >8 --

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Takashi Iwai]

At Thu, 22 May 2014 06:49:10 +0200 (CEST),
Christian Couder wrote:
> 
> 
> I told Junio and Johan about this discussion, and Junio, as he did not
> find a good way to subscribe to the list as a newcomer and still send
> a response to an existing thread, said that I can forward the
> following.
> 
> Johan agrees with Junio about this. I don't agree with their opinion
> about "replace".
> 
> From: Junio C Hamano <gitster@pobox.com>:
> 
> -- >8 --
> 
> Takashi Iwai says, in response to what Jason cooper wrote:
> 
> >> For the stuff flying by me, I've been adding the:
> >> 
> >> Fixes: <12-char hash>: ('Offending patch subject')
> >> ...
> >> I've also been contemplating adding
> >> 
> >> Coverletter: https://lkml.kernel.org/r/<First Reference Msg-Id>
> >> 
> >> for large series where the patch submitter has done a thorough
> >> writeup
> >> in the coverletter.
> >
> > Yes, this kind of information is helpful for checking patches at
> > later point, indeed.
> 
> I tend to think that:
> 
>  * "replace" is too heavy-handed tool to use in general.  Nobody
>    sane should publish history full of replacements for public
>    consumption, especially if that is done merely to help
>    bisection.
> 
>  * "notes" is very handy and may be an efficient mechanism to add
>    information after the fact to existing commits, but merging two
>    or more lines of notes histories together is cumbersome.

Do you mean that it's cumbersome from technical/performance viewpoint,
or about the appearance of actual text outputs?


> A good way forward to solve Iwai-san's original issue might be
> 
>  * Establish the "Fixes:" mentioned above as a standard practice.
>    Polishing Christian's interpret-trailers tool might be a good way
>    to encourage developers to do so.
> 
>  * Have an easy way for developers to scan incoming commits for
>    these "Fixes:" footer, and record the reverse mapping locally, so
>    that we can go from a commit whose brokenness is discovered later
>    to the commit that fixes its breakage efficiently.  "notes" may
>    be a good mechanism to implement this mapping, and we do not have
>    to worry about sharing the notes trees among developers.
> 
>  * The information is visible with "log --show-notes" if it is
>    stored in local notes.  When an earlier commit that was later
>    found to be broken is shown, the note that points at the commit
>    that fixes it will be shown.
> 
>  * Teach "bisect" to also take notice of this information, and
>    temporarily cherry-pick while testing commits with fixes that
>    were discovered later, in a way similar to what was suggested by
>    Jiri Kosina in an earlier message.

These sound like a good plan, indeed.

But, one missing, and maybe often happening thing is: people forget to
tag at the right time.  In the scenario above, if a maintainer forgets
to add Fixes: tag in the fix commit, it's all gone?


thanks,

Takashi

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Johan Herland]

On Thu, May 22, 2014 at 7:58 AM, Takashi Iwai <tiwai@suse.de> wrote:
>> From: Junio C Hamano <gitster@pobox.com>:
>> A good way forward to solve Iwai-san's original issue might be
>>
>>  * Establish the "Fixes:" mentioned above as a standard practice.
>>    Polishing Christian's interpret-trailers tool might be a good way
>>    to encourage developers to do so.
>>
>>  * Have an easy way for developers to scan incoming commits for
>>    these "Fixes:" footer, and record the reverse mapping locally, so
>>    that we can go from a commit whose brokenness is discovered later
>>    to the commit that fixes its breakage efficiently.  "notes" may
>>    be a good mechanism to implement this mapping, and we do not have
>>    to worry about sharing the notes trees among developers.
>>
>>  * The information is visible with "log --show-notes" if it is
>>    stored in local notes.  When an earlier commit that was later
>>    found to be broken is shown, the note that points at the commit
>>    that fixes it will be shown.
>>
>>  * Teach "bisect" to also take notice of this information, and
>>    temporarily cherry-pick while testing commits with fixes that
>>    were discovered later, in a way similar to what was suggested by
>>    Jiri Kosina in an earlier message.
>
> These sound like a good plan, indeed.
>
> But, one missing, and maybe often happening thing is: people forget to
> tag at the right time.  In the scenario above, if a maintainer forgets
> to add Fixes: tag in the fix commit, it's all gone?

Yes, unless you add it (using git-notes) as an annotation to the fix
commit, but then you're back to the (perceived) problem of sharing
those notes.

Alternatively, you would have to find a different mechanism for
developers to insert entries into their reverse mappings from
post-facto/missing "Fixes:" tags.


...Johan

-- 
Johan Herland, <johan@herland.net>
www.herland.net

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Christian Couder]

From: Johan Herland <johan@herland.net>

> On Thu, May 22, 2014 at 7:58 AM, Takashi Iwai <tiwai@suse.de> wrote:
>>> From: Junio C Hamano <gitster@pobox.com>:
>>> A good way forward to solve Iwai-san's original issue might be
>>>
>>>  * Establish the "Fixes:" mentioned above as a standard practice.
>>>    Polishing Christian's interpret-trailers tool might be a good way
>>>    to encourage developers to do so.
>>>
>>>  * Have an easy way for developers to scan incoming commits for
>>>    these "Fixes:" footer, and record the reverse mapping locally, so
>>>    that we can go from a commit whose brokenness is discovered later
>>>    to the commit that fixes its breakage efficiently.  "notes" may
>>>    be a good mechanism to implement this mapping, and we do not have
>>>    to worry about sharing the notes trees among developers.
>>>
>>>  * The information is visible with "log --show-notes" if it is
>>>    stored in local notes.  When an earlier commit that was later
>>>    found to be broken is shown, the note that points at the commit
>>>    that fixes it will be shown.
>>>
>>>  * Teach "bisect" to also take notice of this information, and
>>>    temporarily cherry-pick while testing commits with fixes that
>>>    were discovered later, in a way similar to what was suggested by
>>>    Jiri Kosina in an earlier message.
>>
>> These sound like a good plan, indeed.
>>
>> But, one missing, and maybe often happening thing is: people forget to
>> tag at the right time.  In the scenario above, if a maintainer forgets
>> to add Fixes: tag in the fix commit, it's all gone?
> 
> Yes, unless you add it (using git-notes) as an annotation to the fix
> commit, but then you're back to the (perceived) problem of sharing
> those notes.

By the way, I wonder if it might be possible to have signed "notes" or
signed "replace"?

Best,
Christian.

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Johan Herland]

On Thu, May 22, 2014 at 8:52 AM, Christian Couder
<chriscool@tuxfamily.org> wrote:
> From: Johan Herland <johan@herland.net>
>> On Thu, May 22, 2014 at 7:58 AM, Takashi Iwai <tiwai@suse.de> wrote:
>>>> From: Junio C Hamano <gitster@pobox.com>:
>>>> A good way forward to solve Iwai-san's original issue might be
>>>>
>>>>  * Establish the "Fixes:" mentioned above as a standard practice.
>>>>    Polishing Christian's interpret-trailers tool might be a good way
>>>>    to encourage developers to do so.
>>>>
>>>>  * Have an easy way for developers to scan incoming commits for
>>>>    these "Fixes:" footer, and record the reverse mapping locally, so
>>>>    that we can go from a commit whose brokenness is discovered later
>>>>    to the commit that fixes its breakage efficiently.  "notes" may
>>>>    be a good mechanism to implement this mapping, and we do not have
>>>>    to worry about sharing the notes trees among developers.
>>>>
>>>>  * The information is visible with "log --show-notes" if it is
>>>>    stored in local notes.  When an earlier commit that was later
>>>>    found to be broken is shown, the note that points at the commit
>>>>    that fixes it will be shown.
>>>>
>>>>  * Teach "bisect" to also take notice of this information, and
>>>>    temporarily cherry-pick while testing commits with fixes that
>>>>    were discovered later, in a way similar to what was suggested by
>>>>    Jiri Kosina in an earlier message.
>>>
>>> These sound like a good plan, indeed.
>>>
>>> But, one missing, and maybe often happening thing is: people forget to
>>> tag at the right time.  In the scenario above, if a maintainer forgets
>>> to add Fixes: tag in the fix commit, it's all gone?
>>
>> Yes, unless you add it (using git-notes) as an annotation to the fix
>> commit, but then you're back to the (perceived) problem of sharing
>> those notes.
>
> By the way, I wonder if it might be possible to have signed "notes" or
> signed "replace"?

Isn't that merely a matter of creating a signed tag pointing at
whatever the corresponding refs/notes/* or refs/replace/* ref is
pointing at?

Of course that creates a new refs/tags/* ref that is separate from
your notes/replace ref. If you also want the refs/notes/* or
refs/replace/* ref to point at the signed tag (and have it
automatically peel through to the tagged object) you might run into
some problems. For notes, I'm not sure if the notes code peels through
tags to find the notes commit/tree object. For replace refs, I guess
it would look like the original object was not replaced by a different
object of the same type, but was instead replaced by a signed tag
object. Not sure how to solve that...


...Johan

-- 
Johan Herland, <johan@herland.net>
www.herland.net

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Takashi Iwai]

At Thu, 22 May 2014 08:28:37 +0200,
Johan Herland wrote:
> 
> On Thu, May 22, 2014 at 7:58 AM, Takashi Iwai <tiwai@suse.de> wrote:
> >> From: Junio C Hamano <gitster@pobox.com>:
> >> A good way forward to solve Iwai-san's original issue might be
> >>
> >>  * Establish the "Fixes:" mentioned above as a standard practice.
> >>    Polishing Christian's interpret-trailers tool might be a good way
> >>    to encourage developers to do so.
> >>
> >>  * Have an easy way for developers to scan incoming commits for
> >>    these "Fixes:" footer, and record the reverse mapping locally, so
> >>    that we can go from a commit whose brokenness is discovered later
> >>    to the commit that fixes its breakage efficiently.  "notes" may
> >>    be a good mechanism to implement this mapping, and we do not have
> >>    to worry about sharing the notes trees among developers.
> >>
> >>  * The information is visible with "log --show-notes" if it is
> >>    stored in local notes.  When an earlier commit that was later
> >>    found to be broken is shown, the note that points at the commit
> >>    that fixes it will be shown.
> >>
> >>  * Teach "bisect" to also take notice of this information, and
> >>    temporarily cherry-pick while testing commits with fixes that
> >>    were discovered later, in a way similar to what was suggested by
> >>    Jiri Kosina in an earlier message.
> >
> > These sound like a good plan, indeed.
> >
> > But, one missing, and maybe often happening thing is: people forget to
> > tag at the right time.  In the scenario above, if a maintainer forgets
> > to add Fixes: tag in the fix commit, it's all gone?
> 
> Yes, unless you add it (using git-notes) as an annotation to the fix
> commit, but then you're back to the (perceived) problem of sharing
> those notes.
> 
> Alternatively, you would have to find a different mechanism for
> developers to insert entries into their reverse mappings from
> post-facto/missing "Fixes:" tags.

After reconsidering this, now I don't think we can rely only on Fixes
tag.  The addendum notes are useful for other purposes, e.g. finding
external links.  And, yet one useful thing by addendum is that you can
mark the regression no matter whether it's already fixed or not.  With
Fixes-tag reverse mapping, you can find only fixed regressions, but
not open regressions.

So, we seem back to square now :)

Like Christian mentioned, what about some way to push / pull git notes
in a trusted way?  If we apply the maintainer model also for managing
such information, too, the similar mechanism would be required.

Or, better to manage such information in a totally different manner
(e.g. a web API)?


Takashi

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Geert Uytterhoeven]

On Thu, May 22, 2014 at 9:45 AM, Takashi Iwai <tiwai@suse.de> wrote:
>> Yes, unless you add it (using git-notes) as an annotation to the fix
>> commit, but then you're back to the (perceived) problem of sharing
>> those notes.
>>
>> Alternatively, you would have to find a different mechanism for
>> developers to insert entries into their reverse mappings from
>> post-facto/missing "Fixes:" tags.
>
> After reconsidering this, now I don't think we can rely only on Fixes
> tag.  The addendum notes are useful for other purposes, e.g. finding
> external links.  And, yet one useful thing by addendum is that you can
> mark the regression no matter whether it's already fixed or not.  With
> Fixes-tag reverse mapping, you can find only fixed regressions, but
> not open regressions.

Do you want to import the whole bug tracking system, too? ;-)

Gr{oetje,eeting}s,

                        Geert

--
Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
                                -- Linus Torvalds

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Takashi Iwai]

At Thu, 22 May 2014 09:49:44 +0200,
Geert Uytterhoeven wrote:
> 
> On Thu, May 22, 2014 at 9:45 AM, Takashi Iwai <tiwai@suse.de> wrote:
> >> Yes, unless you add it (using git-notes) as an annotation to the fix
> >> commit, but then you're back to the (perceived) problem of sharing
> >> those notes.
> >>
> >> Alternatively, you would have to find a different mechanism for
> >> developers to insert entries into their reverse mappings from
> >> post-facto/missing "Fixes:" tags.
> >
> > After reconsidering this, now I don't think we can rely only on Fixes
> > tag.  The addendum notes are useful for other purposes, e.g. finding
> > external links.  And, yet one useful thing by addendum is that you can
> > mark the regression no matter whether it's already fixed or not.  With
> > Fixes-tag reverse mapping, you can find only fixed regressions, but
> > not open regressions.
> 
> Do you want to import the whole bug tracking system, too? ;-)

Oh no.  But just adding a regression tag to a commit would be no hard
work for maintainers, I suppose.


Takashi

Re: [Ksummit-discuss] [TOPIC] Metadata addendum to git commit

[Theodore Ts'o]

On Thu, May 22, 2014 at 06:49:10AM +0200, Christian Couder wrote:
> 
> I told Junio and Johan about this discussion, and Junio, as he did not
> find a good way to subscribe to the list as a newcomer and still send
> a response to an existing thread, said that I can forward the
> following.

If someone sends a message who is not on the list, it goes into a
moderation queue.  When I approve such requests, I'll also whitelist
that particular e-mail address, so the 4-8 hour latency should only
apply for the first e-mail address from a particular e-mail address.

So just reply to the message; it will eventually make it out to the
whole list.  My approval time will generally be faster during US
business hours.

Cheers,

						- Ted