[musl] Current state and future of musl development infrastructure

[musl] Current state and future of musl development infrastructure

[jvoisin]

Hello everyone,

I had some small chitchat with dalias on irc, both on #musl and
privately about the ideal infrastructure for musl development, but it's
better to discuss those topics on the mailing list.

I think that everyone agrees that we need a bugtracker, and likely some
CI. Something lightweight (ho heavy dependencies), self-hosted, that can
integrate well with emails/mailing-lists/patch tracking/…

Here is a small selection, feel free to suggest more:

- https://bestpractical.com/request-tracker in Perl
- https://www.bugzilla.org/ no need to present it
- https://todo.sr.ht/~sircmpwn/todo.sr.ht from sourcehut, can be tightly
integrated with other pieces, like mailing lists, git commit/patches/…,
continuous integration, …

dalias is very adamant that CI & testing don't belong in the same repo
as software, but in a separate one.

Most of the forges providing continuous integration (gogs, gitea,
gitlab, …) don't play nice with emails, except sourcehut (
https://sourcehut.org/ ), which is "email first".

Henceforth, I think that a good move forward would be to go with sourcehut:

- easy to self-host (python+go)
- lightweight (both server-side and client-side)
- designed around mailing-list/patches first
- powerful continuous integration, on multiple OS and multiple architectures
- usage documented for newcomers (my generation is used to issuing
pull-requests: sending patches by email is completely alien)
- maintained by some musl users ;)

What do you people think?

-- 
Julien (jvoisin) Voisin
GPG: 04D041E8171901CC
dustri.org

Re: [musl] Current state and future of musl development infrastructure

[Zach van Rijn]

On Wed, 2021-12-15 at 22:22 +0100, jvoisin wrote:
> Hello everyone,
> 
> ...
>
> I think that everyone agrees that we need a bugtracker, and
> likely some CI. Something lightweight (ho heavy dependencies),
> self-hosted, that can integrate well with emails/mailing-
> lists/patch tracking/…

My primary concern is that this will create unnecessary noise and
maintenance burdens. Note IRC and the mailing list are 3rd party.

Having watched this space over the past few years I think there
is one actual problem we need to solve and many that we don't:


 1. Fixes and improvements are suggested in IRC or on the list,
then folks follow up months later to ask what is the status. The
matter is lost in the wash or it is unclear who had the baton.

Rich is busy. But is an issue tracker needed here? The rule of
thumb up to this point has been to punt it from IRC to the list.

Since Rich is the sole gatekeeper, a disciplined approach to
keeping "serious" topics/discussion to the list, and using IRC as
casual grounds for starting discussions might be sufficient. Then
a TODO list would be the right tool to ensure issues aren't lost.

For example: what of the Hexagon port?

Using an issue tracker would provide a public view of open/closed
issue status, and maybe subscriptions. Is this all that we need?

Would splitting the lists into 'musl-devel' and 'musl-support' be
enough? Even this seems like overkill.


Most of the other problems are mitigated by the sole-developer
model and general nature (libc) of this project:

  * Quality control -- each change is already carefully reviewed

  * Widespread testing by various distros

  * Relative stability and low churn -- ~50 commits in a year!


As for CI.

It would probably be helpful to be able to run libc-test and
perhaps other applications on a number of real hardware platforms
in some automated manner.

> ...
> 
> dalias is very adamant that CI & testing don't belong in the
> same repo as software, but in a separate one.

+1


> Most of the forges providing continuous integration (gogs,
> gitea, gitlab, …) don't play nice with emails, except sourcehut
> ( https://sourcehut.org/ ), which is "email first".
> 
> Henceforth, I think that a good move forward would be to go
> with sourcehut:

Very few people want to read documentation to submit patches.

I must say, I had to use Google to find the sourcehut docs. Can
you find it on https://sourcehut.org/ or https://sr.ht/ ?

The ~foo/bar+baz@example.com syntax is unintuitive for newcomers.


> ...
> 
>
> What do you people think?

Overall the volume of fixes/improvements/features is so low that
it might not make sense to set up and maintain any more than:

 1. A TODO list

 2. A brick-simple cluster of machines for testing


/$0.02

ZV

Re: [musl] Current state and future of musl development infrastructure

[Rich Felker]

On Wed, Dec 15, 2021 at 04:52:05PM -0600, Zach van Rijn wrote:
> On Wed, 2021-12-15 at 22:22 +0100, jvoisin wrote:
> > Hello everyone,
> > 
> > ...
> >
> > I think that everyone agrees that we need a bugtracker, and
> > likely some CI. Something lightweight (ho heavy dependencies),
> > self-hosted, that can integrate well with emails/mailing-
> > lists/patch tracking/…
> 
> My primary concern is that this will create unnecessary noise and
> maintenance burdens. Note IRC and the mailing list are 3rd party.
> 
> Having watched this space over the past few years I think there
> is one actual problem we need to solve and many that we don't:
> 
> 
>  1. Fixes and improvements are suggested in IRC or on the list,
> then folks follow up months later to ask what is the status. The
> matter is lost in the wash or it is unclear who had the baton.
> 
> Rich is busy. But is an issue tracker needed here? The rule of
> thumb up to this point has been to punt it from IRC to the list.
> 
> Since Rich is the sole gatekeeper, a disciplined approach to
> keeping "serious" topics/discussion to the list, and using IRC as
> casual grounds for starting discussions might be sufficient. Then
> a TODO list would be the right tool to ensure issues aren't lost.
> 
> For example: what of the Hexagon port?
> 
> Using an issue tracker would provide a public view of open/closed
> issue status, and maybe subscriptions. Is this all that we need?

This is roughly what we need: a place to represent long-term state for
the subset of issues that cannot be resolved immediately. Historically
that was very few, but what happens is that they start to accumulate
over time -- this is rather inherent in their nature.

Representing immediately fixable bugs would be fine too, but less
important.

> Would splitting the lists into 'musl-devel' and 'musl-support' be
> enough? Even this seems like overkill.

No, I don't think so. The problem is not that the list is overloaded
for multiple uses. The problem is that there's limited capability to
search for and reply into existing historical threads unless you're a
list member with your copy of the archive, and there's no way to link
new requests with old ones like "duplicate of..." relationship in a
bug tracker, nor any easy way to filter on "still open" vs "resolved",
etc.

> Most of the other problems are mitigated by the sole-developer
> model and general nature (libc) of this project:
> 
>   * Quality control -- each change is already carefully reviewed
> 
>   * Widespread testing by various distros
> 
>   * Relative stability and low churn -- ~50 commits in a year!
> 
> 
> As for CI.
> 
> It would probably be helpful to be able to run libc-test and
> perhaps other applications on a number of real hardware platforms
> in some automated manner.

Yes. Having this automated would be really nice. Even better if it
could be done not just for the official upstream repo but for proposed
patches/patchsets or third-party git branches (on some sort of
request).

I'd really love if the CI were optimized to do incremental builds and
reuse artifacts as much as possible for rapid results, but doing that
right probably requires fixing missing dependency tracking in the
build.

> > What do you people think?
> 
> Overall the volume of fixes/improvements/features is so low that
> it might not make sense to set up and maintain any more than:
> 
>  1. A TODO list
> 
>  2. A brick-simple cluster of machines for testing

A manually curated TODO list/wishlist on the wiki was how this was
managed for the last N years, and it wasn't really good because it
failed to link community discussion of the issue and didn't capture
things not already valued by me or community. Even with a bug tracker
it would be perfectly fine (and much more effective than it is now) to
have a TODO/roadmap, since it could *link to* the relevant tracker
items for more detail and record of what exactly is wanted or proposed
and why.

Rich

Re: [musl] Current state and future of musl development infrastructure

[Natanael Copa]

On Thu, 16 Dec 2021 00:02:20 -0500
Rich Felker <dalias@libc.org> wrote:

> On Wed, Dec 15, 2021 at 04:52:05PM -0600, Zach van Rijn wrote:
> > On Wed, 2021-12-15 at 22:22 +0100, jvoisin wrote:  
> > > Hello everyone,

...

> > As for CI.
> > 
> > It would probably be helpful to be able to run libc-test and
> > perhaps other applications on a number of real hardware platforms
> > in some automated manner.  
> 
> Yes. Having this automated would be really nice. Even better if it
> could be done not just for the official upstream repo but for proposed
> patches/patchsets or third-party git branches (on some sort of
> request).

I think it would be helpful to have the tests bundled in musl git
repository and in release tarballs. It is common practice for
downstream packagers to run `make && make check`, to run unit tests to
have some basic smoke testing (that does not require root permissions).

Depending on some external test suite (libc-test) is suboptimal because
they may not catch recent additions, and it may be difficult to keep
track of what is expected to pass or fail.

I think it would make sense to have both a lighter/faster integrated
set of unit tests, that are bundled with musl itself and have a more
extensive (external) integration tests that is run with the CI.

> I'd really love if the CI were optimized to do incremental builds and
> reuse artifacts as much as possible for rapid results, but doing that
> right probably requires fixing missing dependency tracking in the
> build.

Just an idea for incremental builds: Do out-of-tree builds. let the CI
store a tarball of the out-of-tree build in some key/value storage,
using the commit hash as the id. Subsequent builds tries fetch the old
heads community id from storage and extract it to the out-of-tree build
location.

It might also be a good idea to try get it work first and optimize
afterwards :)

-nc

Re: [musl] Current state and future of musl development infrastructure

[Rich Felker]

On Thu, Dec 16, 2021 at 12:40:05PM +0100, Natanael Copa wrote:
> On Thu, 16 Dec 2021 00:02:20 -0500
> Rich Felker <dalias@libc.org> wrote:
> 
> > On Wed, Dec 15, 2021 at 04:52:05PM -0600, Zach van Rijn wrote:
> > > On Wed, 2021-12-15 at 22:22 +0100, jvoisin wrote:  
> > > > Hello everyone,
> 
> ....
> 
> > > As for CI.
> > > 
> > > It would probably be helpful to be able to run libc-test and
> > > perhaps other applications on a number of real hardware platforms
> > > in some automated manner.  
> > 
> > Yes. Having this automated would be really nice. Even better if it
> > could be done not just for the official upstream repo but for proposed
> > patches/patchsets or third-party git branches (on some sort of
> > request).
> 
> I think it would be helpful to have the tests bundled in musl git
> repository and in release tarballs. It is common practice for
> downstream packagers to run `make && make check`, to run unit tests to
> have some basic smoke testing (that does not require root permissions).

I'm aware this is common practice, and I disagree with it. It *kinda*
makes sense for a project where the interfaces to be tested are
*defined* by the project itself. It does not make sense when they are
mostly or entirely defined by third parties (the relevant standards).

> Depending on some external test suite (libc-test) is suboptimal because
> they may not catch recent additions,

This would apply even if tests were integrated unless we had a
glibc-like policy that you're not allowed to fix bugs without a
matching test. In my book this is an extremely bad policy that's kept
bugs around for a long time (years?) beyond the date it was known how
to fix them just because nobody could write a good test.

> and it may be difficult to keep track of what is expected to pass
> or fail.

This could possibly be within the scope of CI?

> I think it would make sense to have both a lighter/faster integrated
> set of unit tests, that are bundled with musl itself and have a more
> extensive (external) integration tests that is run with the CI.

Aside from conceptual reasons not to do this, from a practical
standpoint I think it would mean duplicated work or gratuitous
competition, taking away effort from maintaining a strong cross-libc
test package that's incentivized not to do things in non-portable ways
or poke through abstraction layers. For example I'd love to have more
tests of things that are "hard to test" in musl, like hostname
resolution, but when they happen I want them to happen with namespaces
virtualizing resolv.conf and hosts file and port 53 (which is
future-proof and works with other libc's too), not by poking at
internal symbols in musl to attach a test harness to it (which would
not be future-proof and would create pressure to preserve
musl-internal interface boundaries).

> > I'd really love if the CI were optimized to do incremental builds and
> > reuse artifacts as much as possible for rapid results, but doing that
> > right probably requires fixing missing dependency tracking in the
> > build.
> 
> Just an idea for incremental builds: Do out-of-tree builds. let the CI
> store a tarball of the out-of-tree build in some key/value storage,
> using the commit hash as the id. Subsequent builds tries fetch the old
> heads community id from storage and extract it to the out-of-tree build
> location.

I'm not clear why you'd need to re-run CI for same commit hash. The
artifacts probably need to be cached by ccache type logic or
something, not commit hash, if you're going to do this.

> It might also be a good idea to try get it work first and optimize
> afterwards :)

I was expressing this more from a standpoint of shopping for good
existing CI tools rather than building one -- in that context it's
good to know a set of properties/features that might be helpful to
developers & maintaner.

Rich

Re: [musl] Current state and future of musl development infrastructure

[Zach van Rijn]

On Thu, 2021-12-16 at 00:02 -0500, Rich Felker wrote:
> ...
> 
> I'd really love if the CI were optimized to do incremental
> builds and reuse artifacts as much as possible for rapid
> results, but doing that right probably requires fixing missing
> dependency tracking in the build.

I'd recommend taking ccache for a spin before investing time in a
more involved solution. What type of security/access requirements
do you have? E.g. a mechanism to keep some code/tests private?


> A manually curated TODO list/wishlist on the wiki was how this
> was managed for the last N years, and it wasn't really good
> because it failed to link community discussion of the issue and
> didn't capture things not already valued by me or community.
> Even with a bug tracker it would be perfectly fine (and much
> more effective than it is now) to have a TODO/roadmap, since it
> could *link to* the relevant tracker items for more detail and
> record of what exactly is wanted or proposed and why.

Leah suggested Request Tracker [0] yesterday in IRC but setup is
pretty nasty [1] (quote):

    Some modules require user input or environment variables to
    install correctly, so it may be necessary to install them
    manually. Some modules also require external source
    libraries, so you may need to install additional packages.

Not to mention database backups, upgrades, and other maintenance.

One might consider Mantis [2]. It's brilliantly simple, so I've
put together a quick example of a minimal setup [3]. Someone will
need to figure out how to enable email support in this example.

A live demo (official Mantis bug tracker) is here [4].


ZV

[0]: https://bestpractical.com/download-page
[1]: https://docs.bestpractical.com/rt/5.0.2/README.html
[2]: https://www.mantisbt.org/
[3]: https://git.zv.io/t/musl-rt
[4]: https://mantisbt.org/bugs/my_view_page.php

Re: [musl] Current state and future of musl development infrastructure

[writeonce]

On 12/15/2021 22:22, jvoisin wrote:
> Hello everyone,
> 
> I had some small chitchat with dalias on irc, both on #musl and
> privately about the ideal infrastructure for musl development, but it's
> better to discuss those topics on the mailing list.
> 
> I think that everyone agrees that we need a bugtracker, and likely some
> CI. Something lightweight (ho heavy dependencies), self-hosted, that can
> integrate well with emails/mailing-lists/patch tracking/…
> 
> Here is a small selection, feel free to suggest more:
> 
> - https://bestpractical.com/request-tracker in Perl
> - https://www.bugzilla.org/ no need to present it
> - https://todo.sr.ht/~sircmpwn/todo.sr.ht from sourcehut, can be tightly
> integrated with other pieces, like mailing lists, git commit/patches/…,
> continuous integration, …

Another great option would be Pagure (https://pagure.io/pagure), which has a nice interface for online issue tracking alongside smooth email integration. To get an idea of what a self-hosted Pagure instance might look like, see https://dev.midipix.org/

A template for setting up a pagure instance can be found under:
- https://dev.midipix.org/hostcfg/culturestrings/blob/main/f/public/fs/etc/pagure

> 
> dalias is very adamant that CI & testing don't belong in the same repo
> as software, but in a separate one.
> 
> Most of the forges providing continuous integration (gogs, gitea,
> gitlab, …) don't play nice with emails, except sourcehut (
> https://sourcehut.org/ ), which is "email first".
> 
> Henceforth, I think that a good move forward would be to go with sourcehut:
> 
> - easy to self-host (python+go)
> - lightweight (both server-side and client-side)
> - designed around mailing-list/patches first
> - powerful continuous integration, on multiple OS and multiple architectures
> - usage documented for newcomers (my generation is used to issuing
> pull-requests: sending patches by email is completely alien)
> - maintained by some musl users ;)
> 
> What do you people think?
> 
> -- 
> Julien (jvoisin) Voisin
> GPG: 04D041E8171901CC
> dustri.org
> 

--