mailing list of musl libc
 help / color / mirror / code / Atom feed
* [musl] musl CI?
@ 2020-11-17  4:09 Rich Felker
  2020-11-17  6:52 ` Ruinland ChuanTzu Tsai
                   ` (2 more replies)
  0 siblings, 3 replies; 7+ messages in thread
From: Rich Felker @ 2020-11-17  4:09 UTC (permalink / raw)
  To: musl

The topic of CI for testing came up on #musl today, and although it's
not something I want to be responsible for running, it does sound very
good to have. We noted that libc-test currently has tests that are
failing and expected to fail, and that this is normal and not cause
for disabling tests, but it did suggest to me a methodology that seems
more appropriate than checking whether tests succeed: checking for
differences in test output (as well as other things, like symbol
table) vs suitable baselines like last release or last run.

I don't think anyone has concrete plans for going forward with this
yet, but I'm posting the outcome so far here for further discussion
and so it's not lost.

Rich

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [musl] musl CI?
  2020-11-17  4:09 [musl] musl CI? Rich Felker
@ 2020-11-17  6:52 ` Ruinland ChuanTzu Tsai
  2020-11-17 15:52   ` Rich Felker
  2020-11-17  8:39 ` Ruinland ChuanTzu Tsai
  2020-11-17 12:04 ` Leah Neukirchen
  2 siblings, 1 reply; 7+ messages in thread
From: Ruinland ChuanTzu Tsai @ 2020-11-17  6:52 UTC (permalink / raw)
  To: musl

Hi all,

Glad to know there are discussions about CI.
Sorry that I couldn't find the IRC log and thus I might be saying some-
thing which had already been disccussed.

Though might be irrevalent, I'm wondering which platforms are the
"golden standard" for such CI to run against for each musl supported
architecture ?

My rough hunch is something like : latest LTS Linux kernel
(which is 5.4 for now) running on QEMU "virt machine" with full-system
emulation or qemu-user ?

For full-system, there's an acceptance test matrix [1] which lists the
"machine" names and whether they could boot Linux successfully. And the
best part is that they provides qemu launch parameters and working
images (burried in their CI Python code), which saves people from the
burden of bringing up a bootable environment.

Currently, I've been running libc-test with some modifications on
Makefile so as to mimic the way how glibc perform cross-platform test
(their "test-wrapper" mechanism) on real RISC-V hardwares :
(1) mount the libc-test on native platform through NFS and keep the
directory hierarchy to be the same on the host.
(2) provide a hook script to invoke exection remotely with the help of
ssh remote command ( `ssh board "$@"` ).

Though I haven't tried this flow on other platforms, I believe this
method is quite gerneal and could be applied easily.

The reason why I prefer running testsuite under a fully-emulated Linux
instead of qemu-user is that I don't have too much faith on its syscall
-translation and signal-handling mechanism to work out perfectly when
the guest and host architecture are heterogeneous.
(This is just my prejudice, please correct me if I worry too much.)

Cordially yours,
Ruinland ChuanTzu Tsai

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [musl] musl CI?
  2020-11-17  4:09 [musl] musl CI? Rich Felker
  2020-11-17  6:52 ` Ruinland ChuanTzu Tsai
@ 2020-11-17  8:39 ` Ruinland ChuanTzu Tsai
  2020-11-17  9:44   ` Arnd Bergmann
  2020-11-17 12:04 ` Leah Neukirchen
  2 siblings, 1 reply; 7+ messages in thread
From: Ruinland ChuanTzu Tsai @ 2020-11-17  8:39 UTC (permalink / raw)
  To: musl

Oops, I forgot to append the reference link in the end of last email.
The QEMU acceptance test matrix is here :
https://wiki.qemu.org/Testing/Acceptance

For MIPS32, I'm not quite sure which ones are o32 and which ones are
n32. But I guess it won't take long to figure out.

Cordially Yours,
Ruinland ChuanTzu Tsai

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [musl] musl CI?
  2020-11-17  8:39 ` Ruinland ChuanTzu Tsai
@ 2020-11-17  9:44   ` Arnd Bergmann
  0 siblings, 0 replies; 7+ messages in thread
From: Arnd Bergmann @ 2020-11-17  9:44 UTC (permalink / raw)
  To: musl

On Tue, Nov 17, 2020 at 9:40 AM Ruinland ChuanTzu Tsai
<ruinland@andestech.com> wrote:
>
> Oops, I forgot to append the reference link in the end of last email.
> The QEMU acceptance test matrix is here :
> https://wiki.qemu.org/Testing/Acceptance
>
> For MIPS32, I'm not quite sure which ones are o32 and which ones are
> n32. But I guess it won't take long to figure out.

All 64-bit processors can run both o32 and n32 user space in compat mode
using a 64-bit kernel, but 32-bit kernels can only run o32 user space,
and the n32 user space (on 64-bit kernels) is rarely used in practice.

The 'nanomips' target uses an incompatible instruction set and ABI that
is supported by neither mainline kernels nor musl. There were patches
for it a few years ago, but I don't think any actual products running
Linux were built around the i7200 core, so it doesn't matter.

      Arnd

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [musl] musl CI?
  2020-11-17  4:09 [musl] musl CI? Rich Felker
  2020-11-17  6:52 ` Ruinland ChuanTzu Tsai
  2020-11-17  8:39 ` Ruinland ChuanTzu Tsai
@ 2020-11-17 12:04 ` Leah Neukirchen
  2020-11-17 15:48   ` Rich Felker
  2 siblings, 1 reply; 7+ messages in thread
From: Leah Neukirchen @ 2020-11-17 12:04 UTC (permalink / raw)
  To: Rich Felker; +Cc: musl

Rich Felker <dalias@libc.org> writes:

> The topic of CI for testing came up on #musl today, and although it's
> not something I want to be responsible for running, it does sound very
> good to have. We noted that libc-test currently has tests that are
> failing and expected to fail, and that this is normal and not cause
> for disabling tests, but it did suggest to me a methodology that seems
> more appropriate than checking whether tests succeed: checking for
> differences in test output (as well as other things, like symbol
> table) vs suitable baselines like last release or last run.

The tool abidiff from libabigail may be useful for that:

https://sourceware.org/libabigail/manual/abidiff.html

-- 
Leah Neukirchen  <leah@vuxu.org>  https://leahneukirchen.org/

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [musl] musl CI?
  2020-11-17 12:04 ` Leah Neukirchen
@ 2020-11-17 15:48   ` Rich Felker
  0 siblings, 0 replies; 7+ messages in thread
From: Rich Felker @ 2020-11-17 15:48 UTC (permalink / raw)
  To: musl

On Tue, Nov 17, 2020 at 01:04:07PM +0100, Leah Neukirchen wrote:
> Rich Felker <dalias@libc.org> writes:
> 
> > The topic of CI for testing came up on #musl today, and although it's
> > not something I want to be responsible for running, it does sound very
> > good to have. We noted that libc-test currently has tests that are
> > failing and expected to fail, and that this is normal and not cause
> > for disabling tests, but it did suggest to me a methodology that seems
> > more appropriate than checking whether tests succeed: checking for
> > differences in test output (as well as other things, like symbol
> > table) vs suitable baselines like last release or last run.
> 
> The tool abidiff from libabigail may be useful for that:
> 
> https://sourceware.org/libabigail/manual/abidiff.html

I seem to recall it coming up before and finding it overkill. Not
objecting if anyone wants to setup something using it and can, but
IIRC nsz has a simple ABI dumper too based on C++ name mangling, and
just the symbol table itself (without any info on further ABI for the
symbols) can be done with nm -D.

Rich

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [musl] musl CI?
  2020-11-17  6:52 ` Ruinland ChuanTzu Tsai
@ 2020-11-17 15:52   ` Rich Felker
  0 siblings, 0 replies; 7+ messages in thread
From: Rich Felker @ 2020-11-17 15:52 UTC (permalink / raw)
  To: musl

On Tue, Nov 17, 2020 at 02:52:13PM +0800, Ruinland ChuanTzu Tsai wrote:
> Hi all,
> 
> Glad to know there are discussions about CI.
> Sorry that I couldn't find the IRC log and thus I might be saying some-
> thing which had already been disccussed.
> 
> Though might be irrevalent, I'm wondering which platforms are the
> "golden standard" for such CI to run against for each musl supported
> architecture ?
> 
> My rough hunch is something like : latest LTS Linux kernel
> (which is 5.4 for now) running on QEMU "virt machine" with full-system
> emulation or qemu-user ?

This is a really good question. I might lean towards newer (latest
stable) and older to catch breakage from use of newly added syscalls
or broken fallback on much older kernels.

> The reason why I prefer running testsuite under a fully-emulated Linux
> instead of qemu-user is that I don't have too much faith on its syscall
> -translation and signal-handling mechanism to work out perfectly when
> the guest and host architecture are heterogeneous.
> (This is just my prejudice, please correct me if I worry too much.)

Same. qemu-user is not really good as anything beyond a smoke test; it
has too many false positives and negatives from broken emulation.

Rich

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2020-11-17 15:52 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-11-17  4:09 [musl] musl CI? Rich Felker
2020-11-17  6:52 ` Ruinland ChuanTzu Tsai
2020-11-17 15:52   ` Rich Felker
2020-11-17  8:39 ` Ruinland ChuanTzu Tsai
2020-11-17  9:44   ` Arnd Bergmann
2020-11-17 12:04 ` Leah Neukirchen
2020-11-17 15:48   ` Rich Felker

Code repositories for project(s) associated with this public inbox

	https://git.vuxu.org/mirror/musl/

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).