Hi and a few questions

Hi and a few questions

[Richard Pennington]

Hi,

I'm putting together a set of tools and libraries for doing embedded C and C++ 
development using clang/LLVM. 

The web site describing the project is http://ellcc.org

I just discovered musl (from a post on the minux mailing list) and it looks 
very cool.

I'm currently using a partial Linux port of the NetBSD libraries, but it looks 
like musl might be a better choice. I'm thinking of switching, and I have a 
few questions, but first a little background.

I want to target several processors, including i386, x86_64, arm, mips, 
microblaze, ppc, and ppc64 so it looks like musl support will have to be added 
for the currently unsupported processors.

I've done some preliminary testing by compiling the Open POSIX Test Suite 
(http://posixtest.sourceforge.net) three ways:
	1. with gcc/glibc, x86_64
	2. with clang/LLVM/glibc, x86_64
	3. with clang/LLVM/musl, x86_64

The results have been good enough that I'm pretty sure I want to switch:

[~/ellcc/posixtestsuite] main% grep PASS logfile.musl | wc
   5074   15286  275399
[~/ellcc/posixtestsuite] main% grep PASS logfile.ecc | wc
   5381   16143  294510
[~/ellcc/posixtestsuite] main% grep PASS logfile.gcc | wc
   5380   16140  294458

Now for my questions:
	1. Can musl be built out of the source tree? I'd like to be able to build
	    for different processors in different directories.
	2. Are the include/bits files the only include files that differ between
            processors?
	3. Are people actively working on other musl ports? I'd wouldn't want to
	    duplicate their efforts.

Sorry for the basic questions, but I just started looking at musl this 
morning. ;-)

-Rich

Re: Hi and a few questions

[Rich Felker]

On Sun, May 20, 2012 at 12:03:20PM -0500, Richard Pennington wrote:
> I want to target several processors, including i386, x86_64, arm,
> mips, microblaze, ppc, and ppc64 so it looks like musl support will
> have to be added for the currently unsupported processors.

Yes, and I'd be very happy to get support added. The reason for lack
of ports is not lack of portability but lack of knowledge about these
targets. I read up on ARM and did the ARM port using qemu / Aboriginal
Linux boot images just because I found it a bit shameful to only
support x86[_64], but I haven't gotten around to doing this with any
others.

> I've done some preliminary testing by compiling the Open POSIX Test Suite 
> (http://posixtest.sourceforge.net) three ways:
> 	1. with gcc/glibc, x86_64
> 	2. with clang/LLVM/glibc, x86_64
> 	3. with clang/LLVM/musl, x86_64
> 
> The results have been good enough that I'm pretty sure I want to switch:
> 
> [~/ellcc/posixtestsuite] main% grep PASS logfile.musl | wc
>    5074   15286  275399
> [~/ellcc/posixtestsuite] main% grep PASS logfile.ecc | wc
>    5381   16143  294510
> [~/ellcc/posixtestsuite] main% grep PASS logfile.gcc | wc
>    5380   16140  294458

I suspect the situation is somewhat better than these results
indicate. Open POSIX Test Suite, while useful, has a number of tests
that are incorrect/invalid (mostly, by virtue of invoking undefined
behavior then testing the behavior) that actually test for glibc
behavior rather than POSIX. As far as I know, all of the tests musl
fails to PASS are either (1) of this form, or (2) testing features
(like priority scheduling) that musl does not yet support.

> Now for my questions:
> 	1. Can musl be built out of the source tree? I'd like to be able to build
> 	    for different processors in different directories.

At present, no. Even if the trivial changes were made to put the .o
files somewhere else, there's also the issue of the include/bits
symlink (which could actually be removed since arch/$(ARCH) is also in
the -I path, but doing so would complicate the install rules and
preclude using musl "in-place" without "make install" which is
presently possible and a useful setup.

I'd welcome ideas (though I'd have to weigh whether to adopt them) for
making this possible, but the source is small enough that I wonder if
it's really necessary..

> 	2. Are the include/bits files the only include files that differ between
>             processors?

No, include/bits is just the _public_ differences to the interfaces.
There are also some internal headers in the arch/$(ARCH) directories,
but more importantly, musl's build system implicitly replaces any
source file foo.c with $(ARCH)/foo.s if the latter exists. Some of
these replacements (e.g. all the ones in math) are purely size/speed
optimizations, but most are essential, functions that cannot be
implemented except in assembly: setjmp/longjmp, clone, dlsym, and some
internal stuff:

- cancellation-point syscalls: needs to store stack/instruction
  pointer values so a thread can tell if it is blocked at a
  cancellation point when a cancellation request arrives.

- thread exiting: needs to call munmap on its own stack, obviously
  without touching the stack after the syscall returns.

- setting up the thread pointer register: this is arch-specific.

- startup code (crt/*): must convert the initial register/stack
  contents into arguments for __libc_start_main

> 	3. Are people actively working on other musl ports? I'd wouldn't want to
> 	    duplicate their efforts.

No. There's been some talk in the past, but no code.

> Sorry for the basic questions, but I just started looking at musl this 
> morning. ;-)

No problem. I don't mind answering at all; these are good questions
and it's useful to have the answers on the mailing list archive.

Rich

Re: Hi and a few questions

[Szabolcs Nagy]

* Richard Pennington <rich@pennware.com> [2012-05-20 12:03:20 -0500]:
> I want to target several processors, including i386, x86_64, arm, mips, 
> microblaze, ppc, and ppc64 so it looks like musl support will have to be added 
> for the currently unsupported processors.

yes

> I've done some preliminary testing by compiling the Open POSIX Test Suite 
> (http://posixtest.sourceforge.net) three ways:
> 	1. with gcc/glibc, x86_64
> 	2. with clang/LLVM/glibc, x86_64
> 	3. with clang/LLVM/musl, x86_64

nice, unfortunately it does not seem to be maintained
and it's not updated to posix 2008:
"... error: implicit declaration of function 'usleep'"

> The results have been good enough that I'm pretty sure I want to switch:
> 
> [~/ellcc/posixtestsuite] main% grep PASS logfile.musl | wc
>    5074   15286  275399
> [~/ellcc/posixtestsuite] main% grep PASS logfile.ecc | wc
>    5381   16143  294510
> [~/ellcc/posixtestsuite] main% grep PASS logfile.gcc | wc
>    5380   16140  294458
> 

many tests are broken and thus the build fails eg

"functional/threads/schedule/1-1.c:22:1: error: "_XOPEN_SOURCE" redefined"
they shouldn't define that in the source file (without undefing it first)

"conformance/interfaces/pthread_key_create/1-2.c:44: error: control reaches end of non-void function"
there are a couple of similar bad tests

"conformance/interfaces/aio_read/9-1.c: In function 'main':
conformance/interfaces/aio_read/9-1.c:59: error: implicit declaration of function 'open'"
they use open without including fcntl.h

etc

> Now for my questions:
> 	1. Can musl be built out of the source tree? I'd like to be able to build
> 	    for different processors in different directories.

what do you mean by out of the source tree?
you can set an install prefix for make install

if you want to have the .o files for each target in a separate build directory
then you can have several git clones or hack the Makefile

> 	2. Are the include/bits files the only include files that differ between
>             processors?

no, see eg src/thread/$ARCH or src/math/$ARCH

> 	3. Are people actively working on other musl ports? I'd wouldn't want to
> 	    duplicate their efforts.

i don't think so
mips port was mentioned on irc at some point but i don't think anyone took it up

Re: Hi and a few questions

[Richard Pennington]

On Sunday, May 20, 2012 01:21:16 PM Rich Felker wrote:
> On Sun, May 20, 2012 at 12:03:20PM -0500, Richard Pennington wrote:
> > I want to target several processors, including i386, x86_64, arm,
> > mips, microblaze, ppc, and ppc64 so it looks like musl support will
> > have to be added for the currently unsupported processors.
> 
> Yes, and I'd be very happy to get support added. The reason for lack
> of ports is not lack of portability but lack of knowledge about these
> targets. I read up on ARM and did the ARM port using qemu / Aboriginal
> Linux boot images just because I found it a bit shameful to only
> support x86[_64], but I haven't gotten around to doing this with any
> others.

Good. I'll start working on the ports, then. I also use QEMU for testing, 
generally using QEMU Linux user space emulation.


> I suspect the situation is somewhat better than these results
> indicate. Open POSIX Test Suite, while useful, has a number of tests
> that are incorrect/invalid (mostly, by virtue of invoking undefined
> behavior then testing the behavior) that actually test for glibc
> behavior rather than POSIX. As far as I know, all of the tests musl
> fails to PASS are either (1) of this form, or (2) testing features
> (like priority scheduling) that musl does not yet support.

I agree, I haven't done a complete analysis yet, but many of the errors are 
because -Werror is set and also because of the missing pthread_setschedparam 
and friends.

> 
> > Now for my questions:
> > 	1. Can musl be built out of the source tree? I'd like to be able to build
> > 	
> > 	    for different processors in different directories.
> 
> At present, no. Even if the trivial changes were made to put the .o
> files somewhere else, there's also the issue of the include/bits
> symlink (which could actually be removed since arch/$(ARCH) is also in
> the -I path, but doing so would complicate the install rules and
> preclude using musl "in-place" without "make install" which is
> presently possible and a useful setup.
> 
> I'd welcome ideas (though I'd have to weigh whether to adopt them) for
> making this possible, but the source is small enough that I wonder if
> it's really necessary..

Thinking about it, I agree that changing the build/install doesn't add much 
value. I think I'll use your normal build/install.

> 
> > 	2. Are the include/bits files the only include files that differ between
> > 	
> >             processors?
> 
> No, include/bits is just the _public_ differences to the interfaces.
> There are also some internal headers in the arch/$(ARCH) directories,
> but more importantly, musl's build system implicitly replaces any
> source file foo.c with $(ARCH)/foo.s if the latter exists. Some of
> these replacements (e.g. all the ones in math) are purely size/speed
> optimizations, but most are essential, functions that cannot be
> implemented except in assembly: setjmp/longjmp, clone, dlsym, and some
> internal stuff:
> 
> - cancellation-point syscalls: needs to store stack/instruction
>   pointer values so a thread can tell if it is blocked at a
>   cancellation point when a cancellation request arrives.
> 
> - thread exiting: needs to call munmap on its own stack, obviously
>   without touching the stack after the syscall returns.
> 
> - setting up the thread pointer register: this is arch-specific.
> 
> - startup code (crt/*): must convert the initial register/stack
>   contents into arguments for __libc_start_main
> 

Understood. I should be able to reuse some of the assembly language support 
stuff (startup, setjmp/longjmp, etc.) that I've already written for the other 
targets, I'll just model them after yours.

> > 	3. Are people actively working on other musl ports? I'd wouldn't want to
> > 	
> > 	    duplicate their efforts.
> 
> No. There's been some talk in the past, but no code.

Good. I'll get to work.

-Rich

Re: Hi and a few questions

[Richard Pennington]

On Sunday, May 20, 2012 07:53:16 PM Szabolcs Nagy wrote:
[snip]
> > (http://posixtest.sourceforge.net) three ways:
> > 	1. with gcc/glibc, x86_64
> > 	2. with clang/LLVM/glibc, x86_64
> > 	3. with clang/LLVM/musl, x86_64
> 
> nice, unfortunately it does not seem to be maintained
> and it's not updated to posix 2008:
> "... error: implicit declaration of function 'usleep'"
> 
Yeah, I noticed that right away. I've made a lot of changes to the test suite 
already to modernice it for gcc/glibc and clang/LLVM/glibc.

[snip]
> many tests are broken and thus the build fails eg
> 
> "functional/threads/schedule/1-1.c:22:1: error: "_XOPEN_SOURCE" redefined"
> they shouldn't define that in the source file (without undefing it first)
> 
> "conformance/interfaces/pthread_key_create/1-2.c:44: error: control reaches
> end of non-void function" there are a couple of similar bad tests
> 
> "conformance/interfaces/aio_read/9-1.c: In function 'main':
> conformance/interfaces/aio_read/9-1.c:59: error: implicit declaration of
> function 'open'" they use open without including fcntl.h
> 
> etc

Many of these problems have already been fixed in my copy.

> 
> > Now for my questions:
> > 	1. Can musl be built out of the source tree? I'd like to be able to build
> > 	
> > 	    for different processors in different directories.
> 
> what do you mean by out of the source tree?
> you can set an install prefix for make install
> 
> if you want to have the .o files for each target in a separate build
> directory then you can have several git clones or hack the Makefile

I've decided not to change the build rules much, if any. It doesn't take long 
to build the library anyway, so building several times after cleaning is not a 
big deal.

> 
> > 	2. Are the include/bits files the only include files that differ between
> > 	
> >             processors?
> 
> no, see eg src/thread/$ARCH or src/math/$ARCH

Will do.

> 
> > 	3. Are people actively working on other musl ports? I'd wouldn't want to
> > 	
> > 	    duplicate their efforts.
> 
> i don't think so
> mips port was mentioned on irc at some point but i don't think anyone took
> it up

Thanks for the info.

-Rich

Re: Hi and a few questions

[Szabolcs Nagy]

* Richard Pennington <rich@pennware.com> [2012-05-20 12:03:20 -0500]:
> I'm putting together a set of tools and libraries for doing embedded C and C++ 
> development using clang/LLVM. 
> 
> The web site describing the project is http://ellcc.org
> 

btw have you tried musl with c++?

i wonder how well that works
iirc libc++ had some issues with musl

Re: Hi and a few questions

[Richard Pennington]

On Sunday, May 20, 2012 10:28:59 PM Szabolcs Nagy wrote:
> * Richard Pennington <rich@pennware.com> [2012-05-20 12:03:20 -0500]:
> > I'm putting together a set of tools and libraries for doing embedded C and
> > C++ development using clang/LLVM.
> > 
> > The web site describing the project is http://ellcc.org
> 
> btw have you tried musl with c++?
> 
> i wonder how well that works
> iirc libc++ had some issues with musl

I haven't gotten that far yet. Thanks for the warning.

-Rich

Re: Hi and a few questions

[Richard Pennington]

On Sunday, May 20, 2012 07:53:16 PM Szabolcs Nagy wrote:
[snip] 
> "conformance/interfaces/pthread_key_create/1-2.c:44: error: control reaches
> end of non-void function" there are a couple of similar bad tests

This one might be real. I think the declaration of

  	void pthread_exit(void *);

should be

	void pthread_exit(void *) __attribute__ ((__noreturn__));

for gcc and clang. Does musl allow this type of attribute usage? I don't see 
many attributes used in the header files.

-Rich

Re: Hi and a few questions

[Isaac Dunham]

On Sun, 20 May 2012 13:21:16 -0400
Rich Felker <dalias@aerifal.cx> wrote:

> On Sun, May 20, 2012 at 12:03:20PM -0500, Richard Pennington wrote:
> > I want to target several processors, including i386, x86_64, arm,
> > mips, microblaze, ppc, and ppc64 so it looks like musl support will
> > have to be added for the currently unsupported processors.
> 
> Yes, and I'd be very happy to get support added. The reason for lack
> of ports is not lack of portability but lack of knowledge about these
> targets. I read up on ARM and did the ARM port using qemu / Aboriginal
> Linux boot images just because I found it a bit shameful to only
> support x86[_64], but I haven't gotten around to doing this with any
> others.

There was someone who was asking about portability previously; he has a
project that will use arm and mips cpus (this is the project that needs
libuv, hence the discussion on IRC) and-if musl works with libuv-he
thinks he could convince his boss to fund a port, if one isn't ready
ahead of time.

Microblaze is one of the oddball CPUs that you can configure without an
MMU. Would this project target MMU configurations only?

> > Now for my questions:
> > 	1. Can musl be built out of the source tree? I'd like to be
> > able to build for different processors in different directories.
> 
> At present, no. Even if the trivial changes were made to put the .o
> files somewhere else, there's also the issue of the include/bits
> symlink (which could actually be removed since arch/$(ARCH) is also in
> the -I path, but doing so would complicate the install rules and
> preclude using musl "in-place" without "make install" which is
> presently possible and a useful setup.
> 
> I'd welcome ideas (though I'd have to weigh whether to adopt them) for
> making this possible, but the source is small enough that I wonder if
> it's really necessary..

For what it's worth, a shadow tree (see lndir(1)) would probably do all
that's really needed, if you 
1 Get musl source code
2 lndir $MUSL_SOURCE ${MUSL_SOURCE}-${ARCH}
3 Configure and build in ${MUSL_SOURCE}-${ARCH}

You might have issues with shadowing after you have built musl (ie, do
3 in-tree, 2, repeat 3 in shadow tree), I wouldn't know for sure.

Isaac Dunham

Re: Hi and a few questions

[Rich Felker]

On Sun, May 20, 2012 at 03:44:24PM -0500, Richard Pennington wrote:
> On Sunday, May 20, 2012 07:53:16 PM Szabolcs Nagy wrote:
> [snip] 
> > "conformance/interfaces/pthread_key_create/1-2.c:44: error: control reaches
> > end of non-void function" there are a couple of similar bad tests
> 
> This one might be real. I think the declaration of
> 
>   	void pthread_exit(void *);
> 
> should be
> 
> 	void pthread_exit(void *) __attribute__ ((__noreturn__));
> 
> for gcc and clang. Does musl allow this type of attribute usage? I don't see 
> many attributes used in the header files.

In principle I'm not opposed to adding attributes under #ifdef
__GNUC__ when they have clear benefits to the generated code. If exit
functions are such a case, I would consider it.

However, on the uClibc list it was noted that the noreturn attribute
makes WORSE code from a debugging standpoint by making it impossible
to backtrace an exit in the debugger (which can be useful when there's
a breakpoint or crash in an atexit or pthread cleanup function).

Also, while POSIX does not permit conforming applications to do this,
musl allows longjmp out of cleanup handlers (which might have been
invoked via pthread_exit or cancellation) and uses this allowance
internally in the POSIX timer code (to allow reuse of the timer thread
for realtime robustness, i.e. no danger that a thread cannot be
created).

Rich

Re: Hi and a few questions

[Rich Felker]

On Sun, May 20, 2012 at 01:49:36PM -0700, Isaac Dunham wrote:
> There was someone who was asking about portability previously; he has a
> project that will use arm and mips cpus (this is the project that needs
> libuv, hence the discussion on IRC) and-if musl works with libuv-he
> thinks he could convince his boss to fund a port, if one isn't ready
> ahead of time.

Good to know!

> Microblaze is one of the oddball CPUs that you can configure without an
> MMU. Would this project target MMU configurations only?

Well POSIX requires an MMU (or a completely virtualized system that
has one), and as far as I know Linux does too (although there's
UCLinux or whatever too). I suspect the situation would be the same as
how we handle 2.4 kernels: "If some functionality works, great, but
let's not do all kinds of crazy things to make stuff work 'better'
when it's impossible to work 100% right anyway due to platform
limitations."

> For what it's worth, a shadow tree (see lndir(1)) would probably do all
> that's really needed, if you 
> 1 Get musl source code
> 2 lndir $MUSL_SOURCE ${MUSL_SOURCE}-${ARCH}
> 3 Configure and build in ${MUSL_SOURCE}-${ARCH}

Yes, this should work very well.

> You might have issues with shadowing after you have built musl (ie, do
> 3 in-tree, 2, repeat 3 in shadow tree), I wouldn't know for sure.

The only major problem I can think of is include/bits getting
shadowed, but "make clean" removes it anyway. So even this might work.

Rich

Re: Hi and a few questions

[Isaac Dunham]

On Sun, 20 May 2012 15:36 -0500
Richard Pennington <rich@pennware.com> wrote:

> On Sunday, May 20, 2012 10:28:59 PM Szabolcs Nagy wrote:
> > * Richard Pennington <rich@pennware.com> [2012-05-20 12:03:20
> > -0500]:
> > > I'm putting together a set of tools and libraries for doing
> > > embedded C and C++ development using clang/LLVM.
> > 
> > btw have you tried musl with c++?
> > 
> > i wonder how well that works
> > iirc libc++ had some issues with musl
> 
> I haven't gotten that far yet. Thanks for the warning.
AFAIK they were addressed last month.
(There are probably some residual issues though...)

Someone had a hacked clang + musl working several months ago, also.