[PATCH 0/3] b4: Add git notes for submission link trailers

[PATCH 0/3] b4: Add git notes for submission link trailers

[Dan Williams]

As detailed in patch3, after the fact / dynamic lore link generation
continues to be awkward for managing work-in-progress patches across
multiple versions. 'git notes' allows for workflow specific metadata
that does not pollute upstream. The other desirable feature of notes is
that they are displayed by default in cgit.

So the mainline history and upstream pull-requests remain clean, but
work-in-progress series development can readily display which commits
came from which messages.

Example:
$ b4 shazam -L 20250815010645.2980846-1-alison.schofield@intel.com
$ git show | grep -A1 Notes
Notes:
    Link: https://patch.msgid.link/20250815010645.2980846-1-alison.schofield@intel.com


Dan Williams (3):
  b4: Move linktrailer to a LoreMessage property
  b4, ty: Move git_get_rev_diff to __init__ for reuse in post processing
    shazam
  mbox: Add a --add-link-note option to shazam

 src/b4/__init__.py | 53 +++++++++++++++++++++++++-----------------
 src/b4/command.py  |  2 ++
 src/b4/mbox.py     | 58 ++++++++++++++++++++++++++++++++++++++++++++++
 src/b4/ty.py       |  7 +-----
 4 files changed, 93 insertions(+), 27 deletions(-)


base-commit: f760a0468f91296750af072f7b3aed916f217e77
-- 
2.51.0

[PATCH 1/3] b4: Move linktrailer to a LoreMessage property

[Dan Williams]

In preparation for appending Link: trailers as 'git notes'. Arrange for a
submission link trailer to be a property of a message object.

Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
 src/b4/__init__.py | 48 ++++++++++++++++++++++++++--------------------
 1 file changed, 27 insertions(+), 21 deletions(-)

diff --git a/src/b4/__init__.py b/src/b4/__init__.py
index ffa7a5d5c7d9..c608a4d7956a 100644
--- a/src/b4/__init__.py
+++ b/src/b4/__init__.py
@@ -715,27 +715,7 @@ class LoreSeries:
             if lmsg is not None:
                 extras = list()
                 if addlink:
-                    linktrailer = None
-                    ltrmask = config.get('linktrailermask')
-                    if ltrmask and isinstance(ltrmask, str):
-                        if ltrmask.find(':'):
-                            lparts = ltrmask.split(':', maxsplit=1)
-                            llname = lparts[0].strip()
-                            llval = lparts[1].strip() % lmsg.msgid
-                            linktrailer = LoreTrailer(name=llname, value=llval)
-                        else:
-                            logger.critical('linktrailermask does not look like a valid trailer, using defaults')
-
-                    if not linktrailer:
-                        defmask = LOREADDR + '/r/%s'
-                        cfg_llval = config.get('linkmask', defmask)
-                        if isinstance(cfg_llval, str) and '%s' in cfg_llval:
-                            linktrailer = LoreTrailer(name='Link', value=cfg_llval % lmsg.msgid)
-                        else:
-                            logger.critical('linkmask does not look like a valid mask, using defaults')
-                            linktrailer = LoreTrailer(name='Link', value=defmask % lmsg.msgid)
-
-                    extras.append(linktrailer)
+                    extras.append(lmsg.linktrailer)
 
                 if attsame and not attcrit:
                     if attmark:
@@ -1258,6 +1238,7 @@ class LoreMessage:
         self._git_patch_id: Optional[str] = None
         self._pwhash: Optional[str] = None
         self._blob_indexes: Optional[Set[Tuple[str, str, str, str]]] = None
+        self._linktrailer: Optional[str] = None
 
         # Handle [PATCH 6/5]
         if self.counter > self.expected:
@@ -1399,6 +1380,31 @@ class LoreMessage:
                 self._blob_indexes = set()
         return self._blob_indexes
 
+    @property
+    def linktrailer(self) -> Optional[str]:
+        if self._linktrailer is None:
+            config = get_main_config()
+            ltrmask = config.get('linktrailermask')
+            if ltrmask and isinstance(ltrmask, str):
+                if ltrmask.find(':'):
+                    lparts = ltrmask.split(':', maxsplit=1)
+                    llname = lparts[0].strip()
+                    llval = lparts[1].strip() % self.msgid
+                    self._linktrailer = LoreTrailer(name=llname, value=llval)
+                else:
+                    logger.critical('linktrailermask does not look like a valid trailer, using defaults')
+
+            if self._linktrailer is None:
+                defmask = LOREADDR + '/r/%s'
+                cfg_llval = config.get('linkmask', defmask)
+                if isinstance(cfg_llval, str) and '%s' in cfg_llval:
+                    self._linktrailer = LoreTrailer(name='Link', value=cfg_llval % self.msgid)
+                else:
+                    logger.critical('linkmask does not look like a valid mask, using defaults')
+                    self._linktrailer = LoreTrailer(name='Link', value=defmask % self.msgid)
+
+        return self._linktrailer
+
     @property
     def attestors(self) -> List['LoreAttestor']:
         if self._attestors is not None:
-- 
2.51.0

[PATCH 2/3] b4, ty: Move git_get_rev_diff to __init__ for reuse in post processing shazam

[Dan Williams]

In support of adding link trailers as git notes, factor out
git_get_rev_diff() for reuse.

Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
 src/b4/__init__.py | 5 +++++
 src/b4/ty.py       | 7 +------
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/src/b4/__init__.py b/src/b4/__init__.py
index c608a4d7956a..c2cdb028fa54 100644
--- a/src/b4/__init__.py
+++ b/src/b4/__init__.py
@@ -2932,6 +2932,11 @@ def in_directory(dirname: str) -> Generator[bool, None, None]:
         os.chdir(cdir)
 
 
+def git_get_rev_diff(gitdir: Optional[str], rev: str) -> Tuple[int, str]:
+    args = ['diff', '%s~..%s' % (rev, rev)]
+    return git_run_command(gitdir, args)
+
+
 def setup_config(cmdargs: argparse.Namespace) -> None:
     """Setup configuration options. Needs to be called before accessing any of
     the config options."""
diff --git a/src/b4/ty.py b/src/b4/ty.py
index 9f01bd534c74..8d41a22e536b 100644
--- a/src/b4/ty.py
+++ b/src/b4/ty.py
@@ -70,11 +70,6 @@ def git_get_merge_id(gitdir: Optional[str], commit_id: str, branch: Optional[str
     return lines[-1]
 
 
-def git_get_rev_diff(gitdir: Optional[str], rev: str) -> Tuple[int, str]:
-    args = ['diff', '%s~..%s' % (rev, rev)]
-    return b4.git_run_command(gitdir, args)
-
-
 def git_get_commit_message(gitdir: Optional[str], rev: str) -> Tuple[int, str]:
     args = ['log', '--format=%B', '-1', rev]
     return b4.git_run_command(gitdir, args)
@@ -191,7 +186,7 @@ def get_all_commits(gitdir: Optional[str], branch: str, since: str = '1.week',
     # Get patch hash of each commit
     for line in lines:
         commit_id, subject = line.split(maxsplit=1)
-        ecode, out = git_get_rev_diff(gitdir, commit_id)
+        ecode, out = b4.git_get_rev_diff(gitdir, commit_id)
         pwhash = b4.LoreMessage.get_patchwork_hash(out)
         logger.debug('phash=%s', pwhash)
         # get all message-id or link trailers
-- 
2.51.0

[PATCH 3/3] mbox: Add a --add-link-note option to shazam

[Dan Williams]

While the Link: tag is disruptive to some top-level maintainer workflows
[1], it is also useful to a significant number of developers and subsystem
maintainers.

It is also the case that dynamic patch-id lookup [2] is an incomplete
replacement for having the submission Link: trailer readily available.
Specifically, navigating to a patch on gitweb or displaying the patch in
the local developer tree it is convenient to have the metadata inline.

A method to have that metadata available without polluting upstream is to
keep git notes locally.

Add a new option to shazam that annotates newly applied commits with the
Link: trailer of the submission. Honor the b4.linkmask option to use the
preferred namespace (patch.msgid.link) for these links.

Note: Claude Sonnet 4 was used to help early drafts of this patch, but all
submitted lines are authored by me or copied from other parts of b4.

Link: http://lore.kernel.org/CAHk-=whP2zoFm+-EmgQ69-00cxM5jgoEGWyAYVQ8bQYFbb2j=Q@mail.gmail.com [1]
Signed-off-by: Dan Williams <dan.j.williams@intel.com>
---
 src/b4/command.py |  2 ++
 src/b4/mbox.py    | 58 +++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 60 insertions(+)

diff --git a/src/b4/command.py b/src/b4/command.py
index 455124d9726a..678b0b53d6b9 100644
--- a/src/b4/command.py
+++ b/src/b4/command.py
@@ -211,6 +211,8 @@ def setup_parser() -> argparse.ArgumentParser:
     sp_sh = subparsers.add_parser('shazam', help='Like b4 am, but applies the series to your tree')
     cmd_retrieval_common_opts(sp_sh)
     cmd_am_common_opts(sp_sh)
+    sp_sh.add_argument('-L', '--add-link-note', dest='addlinknote', action='store_true', default=False,
+                       help='Add a git note with Link: trailer for every created commit')
     sh_g = sp_sh.add_mutually_exclusive_group()
     sh_g.add_argument('-H', '--make-fetch-head', dest='makefetchhead', action='store_true', default=False,
                       help='Attempt to treat series as a pull request and fetch it into FETCH_HEAD')
diff --git a/src/b4/mbox.py b/src/b4/mbox.py
index 8810ddd71b21..9479b8995019 100644
--- a/src/b4/mbox.py
+++ b/src/b4/mbox.py
@@ -354,6 +354,8 @@ def make_am(msgs: List[EmailMessage], cmdargs: argparse.Namespace, msgid: str) -
             logger.info(out.strip())
             if ecode == 0:
                 thanks_record_am(lser, cherrypick=cherrypick)
+                if cmdargs.addlinknote:
+                    shazam_notes(topdir, lser, 'HEAD')
             sys.exit(ecode)
 
         base_commit = get_base_commit(topdir, first_body, lser, cmdargs)
@@ -448,6 +450,9 @@ def make_am(msgs: List[EmailMessage], cmdargs: argparse.Namespace, msgid: str) -
             # We exec git-merge and let it take over
             os.execvp(mergecmd[0], mergecmd)
 
+        if cmdargs.addlinknote:
+            shazam_notes(topdir, lser, 'FETCH_HEAD')
+
         logger.info('You can now merge or checkout FETCH_HEAD')
         logger.info('  e.g.: %s', ' '.join(mergecmd))
         sys.exit(0)
@@ -547,6 +552,59 @@ def thanks_record_am(lser: b4.LoreSeries, cherrypick: Optional[List[int]]) -> No
         b4.patchwork_set_state(msgids, pwstate)
 
 
+def commits_by_patchid(gitdir: Optional[str], branch: str, num_patches: int) -> b4.Dict[str, str]:
+    """Create a patch-id to commit lookup for the top N commits"""
+
+    commits = dict()
+
+    args = ['log', '--no-abbrev', '--no-decorate', '--oneline', f'-{num_patches}', branch]
+    lines = b4.git_get_command_lines(gitdir, args)
+    if not lines:
+        return commits
+
+    for line in lines:
+        commit_id, subject = line.split(maxsplit=1)
+
+        ecode, diff_out = b4.git_get_rev_diff(gitdir, commit_id)
+        if ecode != 0 or not diff_out.strip():
+            continue
+
+        patch_id = b4.LoreMessage.get_patch_id(diff_out)
+        if patch_id:
+            commits[patch_id] = commit_id
+
+    return commits
+
+
+def shazam_notes(gitdir: Optional[str], lser: 'b4.LoreSeries', branch: str) -> None:
+    """Match commits to LoreMessages using git patch-id and emit debug info for later git notes processing."""
+    if not lser or not lser.patches:
+        return
+
+    lmsgs = [lmsg for lmsg in lser.patches if lmsg is not None and lmsg.has_diff]
+    if not lmsgs:
+        return
+
+    # Cache recently applied commits by patch-id (account for a merge commit)
+    commits = commits_by_patchid(gitdir, branch, len(lmsgs) + 1)
+    if not commits:
+        return
+
+    # Add link trailer notes
+    for lmsg in lmsgs:
+        patch_id = lmsg.git_patch_id
+        if not patch_id:
+            continue
+
+        if patch_id not in commits:
+            continue
+
+        commit_id = commits[patch_id]
+        linktrailer = lmsg.linktrailer
+        note_message = f"{linktrailer.name}: {linktrailer.value}"
+
+        b4.git_run_command(gitdir, ['notes', 'append', '-m', note_message, commit_id])
+
 def save_as_quilt(am_msgs: List[EmailMessage], q_dirname: str) -> None:
     if os.path.exists(q_dirname):
         logger.critical('ERROR: Directory %s exists, not saving quilt patches', q_dirname)
-- 
2.51.0

Re: [PATCH 3/3] mbox: Add a --add-link-note option to shazam

[dan.j.williams]

Dan Williams wrote:
> While the Link: tag is disruptive to some top-level maintainer workflows
> [1], it is also useful to a significant number of developers and subsystem
> maintainers.
> 
> It is also the case that dynamic patch-id lookup [2] is an incomplete
> replacement for having the submission Link: trailer readily available.
> Specifically, navigating to a patch on gitweb or displaying the patch in
> the local developer tree it is convenient to have the metadata inline.
> 
> A method to have that metadata available without polluting upstream is to
> keep git notes locally.
> 
> Add a new option to shazam that annotates newly applied commits with the
> Link: trailer of the submission. Honor the b4.linkmask option to use the
> preferred namespace (patch.msgid.link) for these links.
> 
> Note: Claude Sonnet 4 was used to help early drafts of this patch, but all
> submitted lines are authored by me or copied from other parts of b4.
> 
> Link: http://lore.kernel.org/CAHk-=whP2zoFm+-EmgQ69-00cxM5jgoEGWyAYVQ8bQYFbb2j=Q@mail.gmail.com [1]
> Signed-off-by: Dan Williams <dan.j.williams@intel.com>
> ---
>  src/b4/command.py |  2 ++
>  src/b4/mbox.py    | 58 +++++++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 60 insertions(+)
[..]
> diff --git a/src/b4/mbox.py b/src/b4/mbox.py
> index 8810ddd71b21..9479b8995019 100644
> --- a/src/b4/mbox.py
> +++ b/src/b4/mbox.py
[..]
> @@ -547,6 +552,59 @@ def thanks_record_am(lser: b4.LoreSeries, cherrypick: Optional[List[int]]) -> No
[..]
> +def shazam_notes(gitdir: Optional[str], lser: 'b4.LoreSeries', branch: str) -> None:
> +    """Match commits to LoreMessages using git patch-id and emit debug info for later git notes processing."""

Whoops, this stale comment is from a work-in-progress debug build.