git notes for the Linux kernel

git notes for the Linux kernel

[Vegard Nossum]

Hi cgit maintainers,

I've improved the support for git notes in cgit, including the ability
to load notes from a separate repository than the one you are viewing.

My use case is using a separate repository of git notes for the Linux
kernel to annotate commits with extra cross-referencing information such
as e.g.:

- lore links to patch submissions matching the patch,
- references to subsequent fixes (if the current commit is buggy)
- mitre links to CVEs
- references to backports in stable/LTS

My hope is that these notes can eventually be displayed on
git.kernel.org -- at least, we've found the notes invaluable and a huge
time saver in different types of kernel work. (I'm still in the process
of working out how to release these notes and/or the scripts generating
them, but that's a different topic.)

I tried to submit the git.git patches upstream, but they were rejected
by the maintainer for not being general enough:
https://lore.kernel.org/git/20220802075401.2393-1-vegard.nossum@oracle.com/

I will unfortunately not be able to implement the git maintainer's
suggestion, so I'll just offer up my git and cgit patches here, in case
cgit would still like this functionality; it would mean having to
maintain and carry the extra 2 git.git patches. I would be happy to
continue forward-porting those patches, however, as this is something I
will need to do for our internal use anyway.

Here is a screenshot of cgit with Linux kernel notes in action:

https://vegard.github.io/cgit/6399f1fae4ec.png

cgit patches will follow in replies to this email -- the two prerequisite
git.git patches are available at the git mailing list link above and
should apply cleanly to v2.38.0.

Thanks,


Vegard

[PATCH 1/6] Support notes from external repositories

[Vegard Nossum]

Before this commit, all git notes had to be part of the repository that
the notes are displayed for.

This adds an option to display notes from an external repository.

Signed-off-by: Vegard Nossum <vegard.nossum@oracle.com>
---
 cgit.c       | 27 ++++++++++++++++++++++++++-
 cgit.h       |  2 ++
 cgitrc.5.txt |  8 ++++++++
 shared.c     |  2 ++
 4 files changed, 38 insertions(+), 1 deletion(-)

diff --git a/cgit.c b/cgit.c
index 08d81a1..dc6b491 100644
--- a/cgit.c
+++ b/cgit.c
@@ -107,6 +107,10 @@ static void repo_config(struct cgit_repo *repo, const char *name, const char *va
 		repo->hide = atoi(value);
 	else if (!strcmp(name, "ignore"))
 		repo->ignore = atoi(value);
+	else if (!strcmp(name, "notes_repo"))
+		repo->notes_repo = xstrdup(value);
+	else if (!strcmp(name, "notes_ref"))
+		repo->notes_ref = xstrdup(value);
 	else if (ctx.cfg.enable_filter_overrides) {
 		if (!strcmp(name, "about-filter"))
 			repo->about_filter = cgit_new_filter(value, ABOUT);
@@ -570,6 +574,8 @@ static void print_no_repo_clone_urls(const char *url)
         html("</a></td></tr>\n");
 }
 
+static struct repository notes_repo;
+
 static void prepare_repo_env(int *nongit)
 {
 	/* The path to the git repository. */
@@ -579,7 +585,26 @@ static void prepare_repo_env(int *nongit)
 	 * load local configuration from the git repository, so we do them both while
 	 * the HOME variables are unset. */
 	setup_git_directory_gently(nongit);
-	load_display_notes(NULL);
+
+	if (ctx.repo->notes_repo) {
+		if (repo_init(&notes_repo, ctx.repo->notes_repo, NULL) == 0)
+			add_to_alternates_memory(notes_repo.objects->odb->path);
+	}
+
+	if (ctx.repo->notes_ref) {
+		struct display_notes_opt notes_opt;
+
+		init_display_notes(&notes_opt);
+		notes_opt.use_default_notes = 0;
+
+		if (ctx.repo->notes_repo)
+			notes_opt.repo = &notes_repo;
+
+		string_list_append(&notes_opt.extra_notes_refs, ctx.repo->notes_ref);
+		load_display_notes(&notes_opt);
+	} else {
+		load_display_notes(NULL);
+	}
 }
 
 static int prepare_repo_cmd(int nongit)
diff --git a/cgit.h b/cgit.h
index 69b5c13..50a5c73 100644
--- a/cgit.h
+++ b/cgit.h
@@ -113,6 +113,8 @@ struct cgit_repo {
 	struct string_list submodules;
 	int hide;
 	int ignore;
+	char *notes_repo;
+	char *notes_ref;
 };
 
 typedef void (*repo_config_fn)(struct cgit_repo *repo, const char *name,
diff --git a/cgitrc.5.txt b/cgitrc.5.txt
index 33a6a8c..f3f31ee 100644
--- a/cgitrc.5.txt
+++ b/cgitrc.5.txt
@@ -562,6 +562,14 @@ repo.max-stats::
 repo.name::
 	The value to show as repository name. Default value: <repo.url>.
 
+repo.notes_ref::
+	A ref or glob similar to the "notes.displayRef" git config option.
+	Default value: none.
+
+repo.notes_repo::
+	If you want to use notes from an external repository, set this
+	option to the path of the repository. Default value: none.
+
 repo.owner::
 	A value used to identify the owner of the repository. Default value:
 	none.
diff --git a/shared.c b/shared.c
index 8115469..d3d0fa0 100644
--- a/shared.c
+++ b/shared.c
@@ -79,6 +79,8 @@ struct cgit_repo *cgit_add_repo(const char *url)
 	ret->clone_url = ctx.cfg.clone_url;
 	ret->submodules.strdup_strings = 1;
 	ret->hide = ret->ignore = 0;
+	ret->notes_repo = NULL;
+	ret->notes_ref = NULL;
 	return ret;
 }
 
-- 
2.35.1.46.g38062e73e0

[PATCH 3/6] git: handle diff_queue_is_empty() changes

[Vegard Nossum]

The newer git version this requires changes for these upstream commits:

* 95433eeed9eac439eb21eb30105354b15e71302e
  diff: add ability to insert additional headers for paths

Signed-off-by: Vegard Nossum <vegard.nossum@oracle.com>
---
 ui-log.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/ui-log.c b/ui-log.c
index 20774bf..4d7c48d 100644
--- a/ui-log.c
+++ b/ui-log.c
@@ -159,7 +159,8 @@ static int show_commit(struct commit *commit, struct rev_info *revs)
 		      "", &revs->diffopt);
 	diffcore_std(&revs->diffopt);
 
-	found = !diff_queue_is_empty();
+	struct diff_options diffopt = { NULL };
+	found = !diff_queue_is_empty(&diffopt);
 	saved_fmt = revs->diffopt.output_format;
 	revs->diffopt.output_format = DIFF_FORMAT_CALLBACK;
 	revs->diffopt.format_callback = cgit_diff_tree_cb;
-- 
2.35.1.46.g38062e73e0

[PATCH 4/6] git: handle string_list_init_{nodup, dup}() changes for ctx.cfg.mimetypes

[Vegard Nossum]

The newer git version this requires changes for these upstream commits:

* bd4232fac3319890429ec303e2f7c3d287c8eaa3
  Merge branch 'ab/struct-init'

Signed-off-by: Vegard Nossum <vegard.nossum@oracle.com>
---
 cgit.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/cgit.c b/cgit.c
index dc6b491..e64694b 100644
--- a/cgit.c
+++ b/cgit.c
@@ -432,7 +432,7 @@ static void prepare_context(void)
 	ctx.page.modified = time(NULL);
 	ctx.page.expires = ctx.page.modified;
 	ctx.page.etag = NULL;
-	string_list_init(&ctx.cfg.mimetypes, 1);
+	string_list_init_dup(&ctx.cfg.mimetypes);
 	if (ctx.env.script_name)
 		ctx.cfg.script_name = xstrdup(ctx.env.script_name);
 	if (ctx.env.query_string)
-- 
2.35.1.46.g38062e73e0

[PATCH 5/6] cgit: add extra alternates

[Vegard Nossum]

This allows you to specify additional alternate repositories to be used
with the given repository to resolve trees and commits.

The main use for this is to be able to view commits that are not really
part of the current repository. In our setup, we have an "all.git" which
is really just an empty (bare) git repository with objects/info/alternates
pointing to every other relevant repository (mainline, stable, etc.) which
we then reference with this new config option.

This is not strictly necessary for notes support, but makes it easier to
linkify certain SHA1s in the notes if those commits are not usually
available from the currently selected repository.

Signed-off-by: Vegard Nossum <vegard.nossum@oracle.com>
---
 cgit.c       | 10 ++++++++++
 cgit.h       |  1 +
 cgitrc.5.txt |  8 ++++++++
 3 files changed, 19 insertions(+)

diff --git a/cgit.c b/cgit.c
index e64694b..e6a94e1 100644
--- a/cgit.c
+++ b/cgit.c
@@ -111,6 +111,8 @@ static void repo_config(struct cgit_repo *repo, const char *name, const char *va
 		repo->notes_repo = xstrdup(value);
 	else if (!strcmp(name, "notes_ref"))
 		repo->notes_ref = xstrdup(value);
+	else if (!strcmp(name, "alternates"))
+		string_list_append(&repo->alternates, xstrdup(value));
 	else if (ctx.cfg.enable_filter_overrides) {
 		if (!strcmp(name, "about-filter"))
 			repo->about_filter = cgit_new_filter(value, ABOUT);
@@ -578,6 +580,8 @@ static struct repository notes_repo;
 
 static void prepare_repo_env(int *nongit)
 {
+	struct string_list_item *alternates_path;
+
 	/* The path to the git repository. */
 	setenv("GIT_DIR", ctx.repo->path, 1);
 
@@ -605,6 +609,12 @@ static void prepare_repo_env(int *nongit)
 	} else {
 		load_display_notes(NULL);
 	}
+
+	for_each_string_list_item(alternates_path, &ctx.repo->alternates) {
+	        struct repository *extra_repo = malloc(sizeof(*extra_repo));
+	        if (repo_init(extra_repo, alternates_path->string, NULL) == 0)
+		        add_to_alternates_memory(extra_repo->objects->odb->path);
+	}
 }
 
 static int prepare_repo_cmd(int nongit)
diff --git a/cgit.h b/cgit.h
index 50a5c73..de5cd3d 100644
--- a/cgit.h
+++ b/cgit.h
@@ -115,6 +115,7 @@ struct cgit_repo {
 	int ignore;
 	char *notes_repo;
 	char *notes_ref;
+	struct string_list alternates;
 };
 
 typedef void (*repo_config_fn)(struct cgit_repo *repo, const char *name,
diff --git a/cgitrc.5.txt b/cgitrc.5.txt
index f3f31ee..1be1922 100644
--- a/cgitrc.5.txt
+++ b/cgitrc.5.txt
@@ -456,6 +456,14 @@ repo.about-filter::
 	Override the default about-filter. Default value: none. See also:
 	"enable-filter-overrides". See also: "FILTER API".
 
+repo.alternates::
+	Include additional repository object databases. This allows the
+	current repository to resolve commits from other repositories
+	(without includings its tags and branches), which is useful if you
+	want a single path that can resolve SHA1s from multiple repositories.
+	This should be an absolute path to the .git directory of the other
+	repository. Default value: none.
+
 repo.branch-sort::
 	Flag which, when set to "age", enables date ordering in the branch ref
 	list, and when set to "name" enables ordering by branch name. Default
-- 
2.35.1.46.g38062e73e0

Re: git notes for the Linux kernel

[Vegard Nossum]

[-- Attachment #1: Type: text/plain, Size: 595 bytes --]


On 10/17/22 13:50, Vegard Nossum wrote:
> cgit patches will follow in replies to this email -- the two prerequisite
> git.git patches are available at the git mailing list link above and
> should apply cleanly to v2.38.0.

Looks like the list rejected 2 of the patches for containing HTML, when
those were actually plaintext emails with patches tweaking cgit's HTML
generation code -- maybe something to look into, as that doesn't seem
too unusual for patches on this mailing list.

Anyway, also attaching a tarball with all the patches here, assuming
that makes it through the filter!


Vegard

[-- Attachment #2: notes.tar.bz2 --]
[-- Type: application/x-bzip, Size: 4989 bytes --]

Re: git notes for the Linux kernel

[John Keeping]

On Mon, Oct 17, 2022 at 01:50:04PM +0200, Vegard Nossum wrote:
> I've improved the support for git notes in cgit, including the ability
> to load notes from a separate repository than the one you are viewing.
> 
> My use case is using a separate repository of git notes for the Linux
> kernel to annotate commits with extra cross-referencing information such
> as e.g.:
> 
> - lore links to patch submissions matching the patch,
> - references to subsequent fixes (if the current commit is buggy)
> - mitre links to CVEs
> - references to backports in stable/LTS

This sounds useful, but...

> My hope is that these notes can eventually be displayed on
> git.kernel.org -- at least, we've found the notes invaluable and a huge
> time saver in different types of kernel work. (I'm still in the process
> of working out how to release these notes and/or the scripts generating
> them, but that's a different topic.)
> 
> I tried to submit the git.git patches upstream, but they were rejected
> by the maintainer for not being general enough:
> https://lore.kernel.org/git/20220802075401.2393-1-vegard.nossum@oracle.com/

... this likely blocks inclusion in CGit as I don't think there's any
desire to maintain a fork of git.git

Parts of this series look like they make sense regardless of the
separate repo option - patch 2 looks unrelated and the repo.notes_ref
config option is potentially useful to keep the CGit config separate
from gitconfig (although it should be "repo.notes-ref" for consistency
with other config keys).  Are you interested in splitting those parts
out?


Regards,
John