Revised 1.0 wishlist

Revised 1.0 wishlist

[Rich Felker]

The following wishlist is a draft. I may have missed some items, and
the compatibility goals and testing goals are new ideas subject to
discussion.

Rich



Features wanted:
- Zoneinfo support.
- Korean, HK, and Taiwan legacy charset decoding in iconv.
- Stateful legacy charset decoding (ISO-2022) (undecided).

Documentation. (Details already posted in other threads)

Source-level compatibility goals:
- LAMP stack
- OpenWRT
- GUI stacks (GTK3, Qt, Webkit, deps like Pango, Cairo, etc.)
- Wine (?)
- Development stack (GCC, LLVM/clang, GDB, strace, ...)
- Xorg (with actual drivers)
- LFS stack (minus possibly some useless stuff)
- QEMU
- SIP/VOIP/telephony (PJSIP, Asterisk?, ...)
- Multimedia (FFmpeg/libav/x264/etc.)

Testing goals:
- Extend existing tests to cover everything that's likely to have
  arch-specific breakage; ensure that all archs pass.
- Review third-party tests (Open POSIX Test Suite, gnulib, etc.) and
  ensure that the only remaining failures are erroneous tests.
- Establish a pre-release testing procedure.

Re: Revised 1.0 wishlist

[Rich Felker]

On Tue, Dec 04, 2012 at 06:59:37PM -0500, Rich Felker wrote:
> The following wishlist is a draft. I may have missed some items, and
> the compatibility goals and testing goals are new ideas subject to
> discussion.

One further goal I'd like to add is increasing the level of ABI
stability. If it seems feasible to match the C++ ABI with glibc, I
think we should go ahead and do that before 1.0 rather than after
(where we would break our own C++ ABI). That means using matching
struct tags for structures that might be involved as arguments in C++
functions, dealing with some types that are defined differently (like
glibc using long instead of a pointer type for pthread_t), etc. In
really ugly cases like pthread_t, I think we could put the bad
definition for compatibility under #ifdef __cplusplus so that C
programs get the benefits of musl's superior definition.

Thoughts on this? Anybody up for auditing the differences, either
fully or at least partially to see if aligning them is feasible? The
reason I'm interested in this is that I suspect a decent portion of
the binaryware apps/libs we might want to support may be written in
C++ rather than C.

Rich

Status of 1.0 wishlist items?

[Rich Felker]

On Tue, Dec 04, 2012 at 06:59:37PM -0500, Rich Felker wrote:
> Features wanted:
> - Zoneinfo support.

I've begun looking at the file format again and it seems simple
enough.

> - Korean, HK, and Taiwan legacy charset decoding in iconv.
> - Stateful legacy charset decoding (ISO-2022) (undecided).

Haven't looked at these yet.

> Source-level compatibility goals:
> - LAMP stack

I'm guessing we may still have some issues here with mysql and perhaps
Apache. The other components work, I believe..

> - OpenWRT

Aside from the soft-float kernel issue, it sounds like OpenWRT is
working. Is this true?

> - GUI stacks (GTK3, Qt, Webkit, deps like Pango, Cairo, etc.)

As far as I know, these are working. Am I right?

> - Wine (?)

No idea.

> - Development stack (GCC, LLVM/clang, GDB, strace, ...)

Has anybody built LLVM/clang on musl? I believe the set of patches
needed for GDB has been reduced quite a bit now, but I'm not sure what
the status on strace is...

> - Xorg (with actual drivers)

What issues are remaining here? This is probably the single most
important compatibility problem we've got.

> - LFS stack (minus possibly some useless stuff)

No idea.

> - QEMU

My understanding is that QEMU is working now, with minimal or no
patching. Is this right?

> - SIP/VOIP/telephony (PJSIP, Asterisk?, ...)

Haven't experimented with it at all yet.

> - Multimedia (FFmpeg/libav/x264/etc.)

Unless there have been regressions, this stuff should all be working,
but it would be nice to check.

Reports would be very helpful.

Rich

Re: Status of 1.0 wishlist items?

[John Spencer]

On 12/13/2012 11:03 PM, Rich Felker wrote:
>
>> Source-level compatibility goals:
>> - LAMP stack
> I'm guessing we may still have some issues here with mysql and perhaps
> Apache. The other components work, I believe..

apache is a must-have... unfortunately.
i can look into it.
berkeley db and php are working so far.
>
>
>> - GUI stacks (GTK3, Qt, Webkit, deps like Pango, Cairo, etc.)
> As far as I know, these are working. Am I right?

Gtk+2, pango, cairo work without patches, i didnt test Qt, Webkit, Gtk+3.
>> - Wine (?)
> No idea.
would be nice if that compiled... maybe i'll look at it later.
>> - Development stack (GCC, LLVM/clang, GDB, strace, ...)
> Has anybody built LLVM/clang on musl? I believe the set of patches
> needed for GDB has been reduced quite a bit now, but I'm not sure what
> the status on strace is...
i built clang half a year ago, it required quite a lot of patches tho.
since that was a SVN version, i didnt collect the patches into a 
sabotage package.
i guess i'll wait for the next release to come out, or apply the calloc 
bugfix manually to 3.2.

gdb and strace are full of horrible ifdef orgies and glibc-assuming 
code, and still need a ton of patches.
need to look into the strace git to see if my patches were accepted...

>> - Xorg (with actual drivers)
> What issues are remaining here? This is probably the single most
> important compatibility problem we've got.
the X stuff requires knowing a lot of stuff and tricks about it, which i 
don't.
i still hope someone originating from the suckless or arch linux 
communities, (ppl that seem to love to play around with exotic window 
managers etc..)
tackles this.
maybe someone could follow the LFS recipes and try to make it work ?

>> - LFS stack (minus possibly some useless stuff)
> No idea.
well, sabotage is more or less based on LFS, so most things in there 
should build, except some dark X11 corners that sabotage hasnt touched yet.
>> - QEMU
> My understanding is that QEMU is working now, with minimal or no
> patching. Is this right?

the major remaining issue is the missing ifaddr.h functionality.
once that is patched out, qemu builds OOB with musl-git on x86_64 and ARM.
i guess for PPC and others we need to add some REG_XY values to signal.h 
to make it happy.
>> - Multimedia (FFmpeg/libav/x264/etc.)
> Unless there have been regressions, this stuff should all be working,
> but it would be nice to check.
alsa seems to work, however there are some really nasty patches needed, 
i.e. the one that hardcodes the mutex initializer:
sed -i 
's@PTHREAD_RECURSIVE_MUTEX_INITIALIZER_NP@{{{1,0,0,0,0,0,0,0,0,0}}}@' 
src/conf.c
sdl works as well, i haven't tested any video stuff though.

Re: Status of 1.0 wishlist items?

[Rich Felker]

On Thu, Dec 13, 2012 at 11:36:41PM +0100, John Spencer wrote:
> On 12/13/2012 11:03 PM, Rich Felker wrote:
> >
> >>Source-level compatibility goals:
> >>- LAMP stack
> >I'm guessing we may still have some issues here with mysql and perhaps
> >Apache. The other components work, I believe..
> 
> apache is a must-have... unfortunately.
> i can look into it.
> berkeley db and php are working so far.

What about mysql? IIRC it was a failure in (some of?) the pkgsrc
builds.

> >>- Development stack (GCC, LLVM/clang, GDB, strace, ...)
> >Has anybody built LLVM/clang on musl? I believe the set of patches
> >needed for GDB has been reduced quite a bit now, but I'm not sure what
> >the status on strace is...
> i built clang half a year ago, it required quite a lot of patches tho.
> since that was a SVN version, i didnt collect the patches into a
> sabotage package.
> i guess i'll wait for the next release to come out, or apply the
> calloc bugfix manually to 3.2.
> 
> gdb and strace are full of horrible ifdef orgies and glibc-assuming
> code, and still need a ton of patches.
> need to look into the strace git to see if my patches were accepted...

Okay.

> >>- Xorg (with actual drivers)
> >What issues are remaining here? This is probably the single most
> >important compatibility problem we've got.
> the X stuff requires knowing a lot of stuff and tricks about it,
> which i don't.
> i still hope someone originating from the suckless or arch linux
> communities, (ppl that seem to love to play around with exotic
> window managers etc..)
> tackles this.
> maybe someone could follow the LFS recipes and try to make it work ?

Okay.

> >>- LFS stack (minus possibly some useless stuff)
> >No idea.
> well, sabotage is more or less based on LFS, so most things in there
> should build, except some dark X11 corners that sabotage hasnt
> touched yet.

Sounds like it's mostly in good shape then.

> >>- QEMU
> >My understanding is that QEMU is working now, with minimal or no
> >patching. Is this right?
> 
> the major remaining issue is the missing ifaddr.h functionality.
> once that is patched out, qemu builds OOB with musl-git on x86_64 and ARM.

Great.

> i guess for PPC and others we need to add some REG_XY values to
> signal.h to make it happy.

Yes, that still needs some attention..

> >>- Multimedia (FFmpeg/libav/x264/etc.)
> >Unless there have been regressions, this stuff should all be working,
> >but it would be nice to check.
> alsa seems to work, however there are some really nasty patches
> needed, i.e. the one that hardcodes the mutex initializer:
> sed -i
> 's@PTHREAD_RECURSIVE_MUTEX_INITIALIZER_NP@{{{1,0,0,0,0,0,0,0,0,0}}}@'
> src/conf.c
> sdl works as well, i haven't tested any video stuff though.

This is a bad "fix"; it's treating the structure as part of the ABI,
which it's presently not considered to be. If we want to support these
things, then the mutex structure should be reordered to match the
ordering it has in glibc so that the ABI matches. If not, alsa should
be fixed not to depend on them. But encoding assumptions about opaque
structures which are likely to change into binaries is a very bad
idea.

Rich

Re: Status of 1.0 wishlist items?

[Szabolcs Nagy]

* Rich Felker <dalias@aerifal.cx> [2012-12-13 18:48:53 -0500]:
> On Thu, Dec 13, 2012 at 11:36:41PM +0100, John Spencer wrote:
> > On 12/13/2012 11:03 PM, Rich Felker wrote:
> > >>- LAMP stack
> > >I'm guessing we may still have some issues here with mysql and perhaps
> > >Apache. The other components work, I believe..
> > 
> > apache is a must-have... unfortunately.
> > i can look into it.
> > berkeley db and php are working so far.
> 
> What about mysql? IIRC it was a failure in (some of?) the pkgsrc
> builds.

this reminds me that there is a difference between the
glibc and musl (==posix) fsync semantics

this may cause performance issues in storage related code
(which often incorrectly use fsync when they mean
fdatasync or O_SYNC vs O_DSYNC)

so if anyone plans to benchmark some web service thing on
musl vs glibc should know about this..

Re: Status of 1.0 wishlist items?

[Luca Barbato]

On 12/13/2012 11:03 PM, Rich Felker wrote:
>> - Multimedia (FFmpeg/libav/x264/etc.)
> 
> Unless there have been regressions, this stuff should all be working,
> but it would be nice to check.

libav will get a fate instance soon, the only problem I experienced was
due threads with small stack but you fixed it long ago.

lu

Re: Status of 1.0 wishlist items?

[ojab]

[-- Attachment #1: Type: text/plain, Size: 1570 bytes --]

On 14.12.2012 02:03, Rich Felker wrote:
>> - Development stack (GCC, LLVM/clang, GDB, strace, ...)
> Has anybody built LLVM/clang on musl? I believe the set of patches
> needed for GDB has been reduced quite a bit now, but I'm not sure what
> the status on strace is...
>

LLVM/clang (x86_64) builds (on sabotage) with the small patch (in the 
attached file), but there are 15 test failing (mostly segfaults at 
include/llvm/Type.h:133, see attached log):

Failing Tests (15):
     LLVM :: Instrumentation/AddressSanitizer/instrument_global.ll
     LLVM :: 
Instrumentation/AddressSanitizer/instrument_initializer_metadata.ll
     LLVM :: Instrumentation/ThreadSanitizer/atomic.ll
     LLVM :: Instrumentation/ThreadSanitizer/read_before_write.ll
     LLVM :: Instrumentation/ThreadSanitizer/read_from_global.ll
     LLVM :: Instrumentation/ThreadSanitizer/tsan_basic.ll
     LLVM :: Instrumentation/ThreadSanitizer/vptr_update.ll
     LLVM :: Other/close-stderr.ll
     LLVM :: Transforms/InstCombine/fprintf-1.ll
     LLVM :: Transforms/InstCombine/fputs-1.ll
     LLVM :: Transforms/InstCombine/osx-names.ll
     LLVM :: Transforms/InstCombine/stpcpy-1.ll
     LLVM :: Transforms/InstCombine/stpcpy_chk-1.ll
     LLVM :: Transforms/InstCombine/strcmp-1.ll
     LLVM :: Transforms/InstCombine/strcpy_chk-1.ll


In "[musl] Summary of 1.0 marketing plan/scheme/nefarious plot from 
IRC." thread there is pt. "- push "musl support" patches to other 
projects upstream all at once" — it's better to send patches to musl@ 
mail-list and leave it here or what?


//wbr ojab

[-- Attachment #2: llvm-musl.diff --]
[-- Type: text/plain, Size: 974 bytes --]

Index: lib/Support/DynamicLibrary.cpp
===================================================================
--- lib/Support/DynamicLibrary.cpp	(revision 170294)
+++ lib/Support/DynamicLibrary.cpp	(working copy)
@@ -155,7 +155,7 @@
 
 // This macro returns the address of a well-known, explicit symbol
 #define EXPLICIT_SYMBOL(SYM) \
-   if (!strcmp(symbolName, #SYM)) return &SYM
+   if (!strcmp(symbolName, #SYM)) return (void *) &SYM
 
 // On linux we have a weird situation. The stderr/out/in symbols are both
 // macros and global variables because of standards requirements. So, we
Index: utils/unittest/googletest/gtest.cc
===================================================================
--- utils/unittest/googletest/gtest.cc	(revision 170294)
+++ utils/unittest/googletest/gtest.cc	(working copy)
@@ -120,6 +120,7 @@
 
 #if GTEST_CAN_STREAM_RESULTS_
 # include <arpa/inet.h>  // NOLINT
+# include <sys/socket.h>  // NOLINT
 # include <netdb.h>  // NOLINT
 #endif
 

[-- Attachment #3: llvm_fails.log --]
[-- Type: text/plain, Size: 7491 bytes --]

~ # /root/llvm-build/Release+Asserts/bin/opt < /root/llvm/test/Instrumentation/AddressSanitizer/instrument_global.ll -asan -asan-module -S
Assertion failed: (ST->isOpaque() || ST->getNumElements() == V.size()) && "Incorrect # elements specified to ConstantStruct::get" (/root/llvm/lib/VMCore/Constants.cpp: get: 846)
Stack dump:
0.    Program arguments: /root/llvm-build/Release+Asserts/bin/opt -asan -asan-module -S 
1.    Running pass 'AddressSanitizerModule' on module '<stdin>'.
Killed


~ # /root/llvm-build/Release+Asserts/bin/opt < /root/llvm/test/Instrumentation/AddressSanitizer/instrument_initializer_metadata.ll -asan -asan-module -S
Stack dump:
0.    Program arguments: /root/llvm-build/Release+Asserts/bin/opt -asan -asan-module -S 
1.    Running pass 'Function Pass Manager' on module '<stdin>'.
2.    Running pass 'AddressSanitizerFunctionPass' on function '@_GLOBAL__I_a'
Segmentation fault (core dumped)
...
#0  0x00000000009315ba in llvm::Type::getTypeID (this=0x606060600000000) at /root/llvm/include/llvm/Type.h:133
No locals.
#1  0x0000000001270a26 in llvm::Type::isFirstClassType (this=0x606060600000000) at /root/llvm/include/llvm/Type.h:236
No locals.
#2  0x00000000013cf6cc in llvm::FunctionType::isValidArgumentType (ArgTy=0x606060600000000) at /root/llvm/lib/VMCore/Type.cpp:396
No locals.
#3  0x00000000013cf3fb in llvm::FunctionType::FunctionType (this=0x389c8e0, Result=0x38944a0, Params=..., IsVarArgs=false) at /root/llvm/lib/VMCore/Type.cpp:351
        i = 2
        e = 6
        SubTys = 0x389c8f8


~ # /root/llvm-build/Release+Asserts/bin/opt < /root/llvm/test/Instrumentation/ThreadSanitizer/atomic.ll -tsan -S
Stack dump:
0.    Program arguments: /root/llvm-build/Release+Asserts/bin/opt -tsan -S 
1.    Running pass 'Function Pass Manager' on module '<stdin>'.
2.    Running pass 'ThreadSanitizer' on function '@atomic8_load_unordered'
Segmentation fault (core dumped)
#0  0x00000000009315ba in llvm::Type::getTypeID (this=0xffffffff00000000) at /root/llvm/include/llvm/Type.h:133


~ # /root/llvm-build/Release+Asserts/bin/opt < /root/llvm/test/Instrumentation/ThreadSanitizer/read_before_write.ll -tsan -S
Stack dump:
0.    Program arguments: /root/llvm-build/Release+Asserts/bin/opt -tsan -S 
1.    Running pass 'Function Pass Manager' on module '<stdin>'.
2.    Running pass 'ThreadSanitizer' on function '@IncrementMe'
Segmentation fault (core dumped)
#0  0x00000000009315ba in llvm::Type::getTypeID (this=0xffffffff00000000) at /root/llvm/include/llvm/Type.h:133


~ # /root/llvm-build/Release+Asserts/bin/opt < /root/llvm/test/Instrumentation/ThreadSanitizer/read_from_global.ll -tsan -S
Stack dump:
0.    Program arguments: /root/llvm-build/Release+Asserts/bin/opt -tsan -S 
1.    Running pass 'Function Pass Manager' on module '<stdin>'.
2.    Running pass 'ThreadSanitizer' on function '@read_from_const_global'
Segmentation fault (core dumped)
#0  0x00000000009315ba in llvm::Type::getTypeID (this=0xffffffff00000000) at /root/llvm/include/llvm/Type.h:133


~ # /root/llvm-build/Release+Asserts/bin/opt < /root/llvm/test/Instrumentation/ThreadSanitizer/tsan_basic.ll -tsan -S
Stack dump:
0.    Program arguments: /root/llvm-build/Release+Asserts/bin/opt -tsan -S 
1.    Running pass 'Function Pass Manager' on module '<stdin>'.
2.    Running pass 'ThreadSanitizer' on function '@read_4_bytes'
Segmentation fault (core dumped)
#0  0x00000000009315ba in llvm::Type::getTypeID (this=0xffffffff00000000) at /root/llvm/include/llvm/Type.h:133


~ # /root/llvm-build/Release+Asserts/bin/opt < /root/llvm/test/Instrumentation/ThreadSanitizer/vptr_update.ll -tsan -S
Stack dump:
0.    Program arguments: /root/llvm-build/Release+Asserts/bin/opt -tsan -S 
1.    Running pass 'Function Pass Manager' on module '<stdin>'.
2.    Running pass 'ThreadSanitizer' on function '@Foo'
Segmentation fault (core dumped)
#0  0x00000000009315ba in llvm::Type::getTypeID (this=0xffffffff00000000) at /root/llvm/include/llvm/Type.h:133


LLVM :: Other/close-stderr.ll
hangs


~ # /root/llvm-build/Release+Asserts/bin/opt < /root/llvm/test/Transforms/InstCombine/fprintf-1.ll  -instcombine -S
Stack dump:
0.    Program arguments: /root/llvm-build/Release+Asserts/bin/opt -instcombine -S 
1.    Running pass 'Function Pass Manager' on module '<stdin>'.
2.    Running pass 'Combine redundant instructions' on function '@test_simplify1'
Segmentation fault (core dumped)
#0  0x00000000009315ba in llvm::Type::getTypeID (this=0x7fff00000000) at /root/llvm/include/llvm/Type.h:133


~ # /root/llvm-build/Release+Asserts/bin/opt < /root/llvm/test/Transforms/InstCombine/fputs-1.ll  -instcombine -S
Stack dump:
0.    Program arguments: /root/llvm-build/Release+Asserts/bin/opt -instcombine -S 
1.    Running pass 'Function Pass Manager' on module '<stdin>'.
2.    Running pass 'Combine redundant instructions' on function '@test_simplify1'
Segmentation fault (core dumped)
#0  0x00000000009315ba in llvm::Type::getTypeID (this=0x7fff00000000) at /root/llvm/include/llvm/Type.h:133


~ # /root/llvm-build/Release+Asserts/bin/opt < /root/llvm/test/Transforms/InstCombine/osx-names.ll  -instcombine -S
Stack dump:
0.    Program arguments: /root/llvm-build/Release+Asserts/bin/opt -instcombine -S 
1.    Running pass 'Function Pass Manager' on module '<stdin>'.
2.    Running pass 'Combine redundant instructions' on function '@test1'
Segmentation fault (core dumped)
#0  0x00000000009315ba in llvm::Type::getTypeID (this=0x7fff00000000) at /root/llvm/include/llvm/Type.h:133


~ # /root/llvm-build/Release+Asserts/bin/opt < /root/llvm/test/Transforms/InstCombine/stpcpy-1.ll  -instcombine -S
Stack dump:
0.    Program arguments: /root/llvm-build/Release+Asserts/bin/opt -instcombine -S 
1.    Running pass 'Function Pass Manager' on module '<stdin>'.
2.    Running pass 'Combine redundant instructions' on function '@test_simplify2'
Segmentation fault (core dumped)
#0  0x00000000009315ba in llvm::Type::getTypeID (this=0x1700000000) at /root/llvm/include/llvm/Type.h:133


~ # /root/llvm-build/Release+Asserts/bin/opt < /root/llvm/test/Transforms/InstCombine/stpcpy_chk-1.ll  -instcombine -S
Stack dump:
0.    Program arguments: /root/llvm-build/Release+Asserts/bin/opt -instcombine -S 
1.    Running pass 'Function Pass Manager' on module '<stdin>'.
2.    Running pass 'Combine redundant instructions' on function '@test_simplify1'
Segmentation fault (core dumped)
#0  0x00000000009315ba in llvm::Type::getTypeID (this=0xffffffff00000000) at /root/llvm/include/llvm/Type.h:133


~ # /root/llvm-build/Release+Asserts/bin/opt < /root/llvm/test/Transforms/InstCombine/strcmp-1.ll  -instcombine -S
Stack dump:
0.    Program arguments: /root/llvm-build/Release+Asserts/bin/opt -instcombine -S 
1.    Running pass 'Function Pass Manager' on module '<stdin>'.
2.    Running pass 'Combine redundant instructions' on function '@test5'
Segmentation fault (core dumped)
#0  0x00000000009315ba in llvm::Type::getTypeID (this=0x7fff00000000) at /root/llvm/include/llvm/Type.h:133


~ # /root/llvm-build/Release+Asserts/bin/opt < /root/llvm/test/Transforms/InstCombine/strcpy_chk-1.ll  -instcombine -S
Stack dump:
0.    Program arguments: /root/llvm-build/Release+Asserts/bin/opt -instcombine -S 
1.    Running pass 'Function Pass Manager' on module '<stdin>'.
2.    Running pass 'Combine redundant instructions' on function '@test_simplify1'
Segmentation fault (core dumped)
#0  0x00000000009315ba in llvm::Type::getTypeID (this=0xffffffff00000000) at /root/llvm/include/llvm/Type.h:133

Re: Status of 1.0 wishlist items?

[John Spencer]

On 12/14/2012 12:48 AM, Rich Felker wrote:
> On Thu, Dec 13, 2012 at 11:36:41PM +0100, John Spencer wrote:
>> On 12/13/2012 11:03 PM, Rich Felker wrote:
>>>> Source-level compatibility goals:
>>>> - LAMP stack
>>

all functional now in sabotage

wine now works as well, however one needs to build it on i386 sabotage 
to use it on x86_64:
https://github.com/rofl0r/sabotage/wiki/wine

i'll put a binary tarball for x86_64 online (and link it on that wiki 
page) as soon as moritz fixed my upload account...