Hello everyone, I had some small chitchat with dalias on irc, both on #musl and privately about the ideal infrastructure for musl development, but it's better to discuss those topics on the mailing list. I think that everyone agrees that we need a bugtracker, and likely some CI. Something lightweight (ho heavy dependencies), self-hosted, that can integrate well with emails/mailing-lists/patch tracking/… Here is a small selection, feel free to suggest more: - https://bestpractical.com/request-tracker in Perl - https://www.bugzilla.org/ no need to present it - https://todo.sr.ht/~sircmpwn/todo.sr.ht from sourcehut, can be tightly integrated with other pieces, like mailing lists, git commit/patches/…, continuous integration, … dalias is very adamant that CI & testing don't belong in the same repo as software, but in a separate one. Most of the forges providing continuous integration (gogs, gitea, gitlab, …) don't play nice with emails, except sourcehut ( https://sourcehut.org/ ), which is "email first". Henceforth, I think that a good move forward would be to go with sourcehut: - easy to self-host (python+go) - lightweight (both server-side and client-side) - designed around mailing-list/patches first - powerful continuous integration, on multiple OS and multiple architectures - usage documented for newcomers (my generation is used to issuing pull-requests: sending patches by email is completely alien) - maintained by some musl users ;) What do you people think? -- Julien (jvoisin) Voisin GPG: 04D041E8171901CC dustri.org
On Wed, 2021-12-15 at 22:22 +0100, jvoisin wrote: > Hello everyone, > > ... > > I think that everyone agrees that we need a bugtracker, and > likely some CI. Something lightweight (ho heavy dependencies), > self-hosted, that can integrate well with emails/mailing- > lists/patch tracking/… My primary concern is that this will create unnecessary noise and maintenance burdens. Note IRC and the mailing list are 3rd party. Having watched this space over the past few years I think there is one actual problem we need to solve and many that we don't: 1. Fixes and improvements are suggested in IRC or on the list, then folks follow up months later to ask what is the status. The matter is lost in the wash or it is unclear who had the baton. Rich is busy. But is an issue tracker needed here? The rule of thumb up to this point has been to punt it from IRC to the list. Since Rich is the sole gatekeeper, a disciplined approach to keeping "serious" topics/discussion to the list, and using IRC as casual grounds for starting discussions might be sufficient. Then a TODO list would be the right tool to ensure issues aren't lost. For example: what of the Hexagon port? Using an issue tracker would provide a public view of open/closed issue status, and maybe subscriptions. Is this all that we need? Would splitting the lists into 'musl-devel' and 'musl-support' be enough? Even this seems like overkill. Most of the other problems are mitigated by the sole-developer model and general nature (libc) of this project: * Quality control -- each change is already carefully reviewed * Widespread testing by various distros * Relative stability and low churn -- ~50 commits in a year! As for CI. It would probably be helpful to be able to run libc-test and perhaps other applications on a number of real hardware platforms in some automated manner. > ... > > dalias is very adamant that CI & testing don't belong in the > same repo as software, but in a separate one. +1 > Most of the forges providing continuous integration (gogs, > gitea, gitlab, …) don't play nice with emails, except sourcehut > ( https://sourcehut.org/ ), which is "email first". > > Henceforth, I think that a good move forward would be to go > with sourcehut: Very few people want to read documentation to submit patches. I must say, I had to use Google to find the sourcehut docs. Can you find it on https://sourcehut.org/ or https://sr.ht/ ? The ~foo/bar+baz@example.com syntax is unintuitive for newcomers. > ... > > > What do you people think? Overall the volume of fixes/improvements/features is so low that it might not make sense to set up and maintain any more than: 1. A TODO list 2. A brick-simple cluster of machines for testing /$0.02 ZV
On Wed, Dec 15, 2021 at 04:52:05PM -0600, Zach van Rijn wrote: > On Wed, 2021-12-15 at 22:22 +0100, jvoisin wrote: > > Hello everyone, > > > > ... > > > > I think that everyone agrees that we need a bugtracker, and > > likely some CI. Something lightweight (ho heavy dependencies), > > self-hosted, that can integrate well with emails/mailing- > > lists/patch tracking/… > > My primary concern is that this will create unnecessary noise and > maintenance burdens. Note IRC and the mailing list are 3rd party. > > Having watched this space over the past few years I think there > is one actual problem we need to solve and many that we don't: > > > 1. Fixes and improvements are suggested in IRC or on the list, > then folks follow up months later to ask what is the status. The > matter is lost in the wash or it is unclear who had the baton. > > Rich is busy. But is an issue tracker needed here? The rule of > thumb up to this point has been to punt it from IRC to the list. > > Since Rich is the sole gatekeeper, a disciplined approach to > keeping "serious" topics/discussion to the list, and using IRC as > casual grounds for starting discussions might be sufficient. Then > a TODO list would be the right tool to ensure issues aren't lost. > > For example: what of the Hexagon port? > > Using an issue tracker would provide a public view of open/closed > issue status, and maybe subscriptions. Is this all that we need? This is roughly what we need: a place to represent long-term state for the subset of issues that cannot be resolved immediately. Historically that was very few, but what happens is that they start to accumulate over time -- this is rather inherent in their nature. Representing immediately fixable bugs would be fine too, but less important. > Would splitting the lists into 'musl-devel' and 'musl-support' be > enough? Even this seems like overkill. No, I don't think so. The problem is not that the list is overloaded for multiple uses. The problem is that there's limited capability to search for and reply into existing historical threads unless you're a list member with your copy of the archive, and there's no way to link new requests with old ones like "duplicate of..." relationship in a bug tracker, nor any easy way to filter on "still open" vs "resolved", etc. > Most of the other problems are mitigated by the sole-developer > model and general nature (libc) of this project: > > * Quality control -- each change is already carefully reviewed > > * Widespread testing by various distros > > * Relative stability and low churn -- ~50 commits in a year! > > > As for CI. > > It would probably be helpful to be able to run libc-test and > perhaps other applications on a number of real hardware platforms > in some automated manner. Yes. Having this automated would be really nice. Even better if it could be done not just for the official upstream repo but for proposed patches/patchsets or third-party git branches (on some sort of request). I'd really love if the CI were optimized to do incremental builds and reuse artifacts as much as possible for rapid results, but doing that right probably requires fixing missing dependency tracking in the build. > > What do you people think? > > Overall the volume of fixes/improvements/features is so low that > it might not make sense to set up and maintain any more than: > > 1. A TODO list > > 2. A brick-simple cluster of machines for testing A manually curated TODO list/wishlist on the wiki was how this was managed for the last N years, and it wasn't really good because it failed to link community discussion of the issue and didn't capture things not already valued by me or community. Even with a bug tracker it would be perfectly fine (and much more effective than it is now) to have a TODO/roadmap, since it could *link to* the relevant tracker items for more detail and record of what exactly is wanted or proposed and why. Rich
On Thu, 16 Dec 2021 00:02:20 -0500 Rich Felker <dalias@libc.org> wrote: > On Wed, Dec 15, 2021 at 04:52:05PM -0600, Zach van Rijn wrote: > > On Wed, 2021-12-15 at 22:22 +0100, jvoisin wrote: > > > Hello everyone, ... > > As for CI. > > > > It would probably be helpful to be able to run libc-test and > > perhaps other applications on a number of real hardware platforms > > in some automated manner. > > Yes. Having this automated would be really nice. Even better if it > could be done not just for the official upstream repo but for proposed > patches/patchsets or third-party git branches (on some sort of > request). I think it would be helpful to have the tests bundled in musl git repository and in release tarballs. It is common practice for downstream packagers to run `make && make check`, to run unit tests to have some basic smoke testing (that does not require root permissions). Depending on some external test suite (libc-test) is suboptimal because they may not catch recent additions, and it may be difficult to keep track of what is expected to pass or fail. I think it would make sense to have both a lighter/faster integrated set of unit tests, that are bundled with musl itself and have a more extensive (external) integration tests that is run with the CI. > I'd really love if the CI were optimized to do incremental builds and > reuse artifacts as much as possible for rapid results, but doing that > right probably requires fixing missing dependency tracking in the > build. Just an idea for incremental builds: Do out-of-tree builds. let the CI store a tarball of the out-of-tree build in some key/value storage, using the commit hash as the id. Subsequent builds tries fetch the old heads community id from storage and extract it to the out-of-tree build location. It might also be a good idea to try get it work first and optimize afterwards :) -nc
On Thu, Dec 16, 2021 at 12:40:05PM +0100, Natanael Copa wrote: > On Thu, 16 Dec 2021 00:02:20 -0500 > Rich Felker <dalias@libc.org> wrote: > > > On Wed, Dec 15, 2021 at 04:52:05PM -0600, Zach van Rijn wrote: > > > On Wed, 2021-12-15 at 22:22 +0100, jvoisin wrote: > > > > Hello everyone, > > .... > > > > As for CI. > > > > > > It would probably be helpful to be able to run libc-test and > > > perhaps other applications on a number of real hardware platforms > > > in some automated manner. > > > > Yes. Having this automated would be really nice. Even better if it > > could be done not just for the official upstream repo but for proposed > > patches/patchsets or third-party git branches (on some sort of > > request). > > I think it would be helpful to have the tests bundled in musl git > repository and in release tarballs. It is common practice for > downstream packagers to run `make && make check`, to run unit tests to > have some basic smoke testing (that does not require root permissions). I'm aware this is common practice, and I disagree with it. It *kinda* makes sense for a project where the interfaces to be tested are *defined* by the project itself. It does not make sense when they are mostly or entirely defined by third parties (the relevant standards). > Depending on some external test suite (libc-test) is suboptimal because > they may not catch recent additions, This would apply even if tests were integrated unless we had a glibc-like policy that you're not allowed to fix bugs without a matching test. In my book this is an extremely bad policy that's kept bugs around for a long time (years?) beyond the date it was known how to fix them just because nobody could write a good test. > and it may be difficult to keep track of what is expected to pass > or fail. This could possibly be within the scope of CI? > I think it would make sense to have both a lighter/faster integrated > set of unit tests, that are bundled with musl itself and have a more > extensive (external) integration tests that is run with the CI. Aside from conceptual reasons not to do this, from a practical standpoint I think it would mean duplicated work or gratuitous competition, taking away effort from maintaining a strong cross-libc test package that's incentivized not to do things in non-portable ways or poke through abstraction layers. For example I'd love to have more tests of things that are "hard to test" in musl, like hostname resolution, but when they happen I want them to happen with namespaces virtualizing resolv.conf and hosts file and port 53 (which is future-proof and works with other libc's too), not by poking at internal symbols in musl to attach a test harness to it (which would not be future-proof and would create pressure to preserve musl-internal interface boundaries). > > I'd really love if the CI were optimized to do incremental builds and > > reuse artifacts as much as possible for rapid results, but doing that > > right probably requires fixing missing dependency tracking in the > > build. > > Just an idea for incremental builds: Do out-of-tree builds. let the CI > store a tarball of the out-of-tree build in some key/value storage, > using the commit hash as the id. Subsequent builds tries fetch the old > heads community id from storage and extract it to the out-of-tree build > location. I'm not clear why you'd need to re-run CI for same commit hash. The artifacts probably need to be cached by ccache type logic or something, not commit hash, if you're going to do this. > It might also be a good idea to try get it work first and optimize > afterwards :) I was expressing this more from a standpoint of shopping for good existing CI tools rather than building one -- in that context it's good to know a set of properties/features that might be helpful to developers & maintaner. Rich
On Thu, 2021-12-16 at 00:02 -0500, Rich Felker wrote: > ... > > I'd really love if the CI were optimized to do incremental > builds and reuse artifacts as much as possible for rapid > results, but doing that right probably requires fixing missing > dependency tracking in the build. I'd recommend taking ccache for a spin before investing time in a more involved solution. What type of security/access requirements do you have? E.g. a mechanism to keep some code/tests private? > A manually curated TODO list/wishlist on the wiki was how this > was managed for the last N years, and it wasn't really good > because it failed to link community discussion of the issue and > didn't capture things not already valued by me or community. > Even with a bug tracker it would be perfectly fine (and much > more effective than it is now) to have a TODO/roadmap, since it > could *link to* the relevant tracker items for more detail and > record of what exactly is wanted or proposed and why. Leah suggested Request Tracker [0] yesterday in IRC but setup is pretty nasty [1] (quote): Some modules require user input or environment variables to install correctly, so it may be necessary to install them manually. Some modules also require external source libraries, so you may need to install additional packages. Not to mention database backups, upgrades, and other maintenance. One might consider Mantis [2]. It's brilliantly simple, so I've put together a quick example of a minimal setup [3]. Someone will need to figure out how to enable email support in this example. A live demo (official Mantis bug tracker) is here [4]. ZV [0]: https://bestpractical.com/download-page [1]: https://docs.bestpractical.com/rt/5.0.2/README.html [2]: https://www.mantisbt.org/ [3]: https://git.zv.io/t/musl-rt [4]: https://mantisbt.org/bugs/my_view_page.php
On 12/15/2021 22:22, jvoisin wrote: > Hello everyone, > > I had some small chitchat with dalias on irc, both on #musl and > privately about the ideal infrastructure for musl development, but it's > better to discuss those topics on the mailing list. > > I think that everyone agrees that we need a bugtracker, and likely some > CI. Something lightweight (ho heavy dependencies), self-hosted, that can > integrate well with emails/mailing-lists/patch tracking/… > > Here is a small selection, feel free to suggest more: > > - https://bestpractical.com/request-tracker in Perl > - https://www.bugzilla.org/ no need to present it > - https://todo.sr.ht/~sircmpwn/todo.sr.ht from sourcehut, can be tightly > integrated with other pieces, like mailing lists, git commit/patches/…, > continuous integration, … Another great option would be Pagure (https://pagure.io/pagure), which has a nice interface for online issue tracking alongside smooth email integration. To get an idea of what a self-hosted Pagure instance might look like, see https://dev.midipix.org/ A template for setting up a pagure instance can be found under: - https://dev.midipix.org/hostcfg/culturestrings/blob/main/f/public/fs/etc/pagure > > dalias is very adamant that CI & testing don't belong in the same repo > as software, but in a separate one. > > Most of the forges providing continuous integration (gogs, gitea, > gitlab, …) don't play nice with emails, except sourcehut ( > https://sourcehut.org/ ), which is "email first". > > Henceforth, I think that a good move forward would be to go with sourcehut: > > - easy to self-host (python+go) > - lightweight (both server-side and client-side) > - designed around mailing-list/patches first > - powerful continuous integration, on multiple OS and multiple architectures > - usage documented for newcomers (my generation is used to issuing > pull-requests: sending patches by email is completely alien) > - maintained by some musl users ;) > > What do you people think? > > -- > Julien (jvoisin) Voisin > GPG: 04D041E8171901CC > dustri.org > --