Timeline for 1.1.20?

Timeline for 1.1.20?

[ardi]

Hi!

Is there a timeline for the 1.1.20 release? I'm starting a project,
and I'd find it very convenient to know if 1.1.20 will be released
this month, or if 1.1.19 is going to be kept as the current release
for some months to come.

Kind Regards,

ardi

Re: Timeline for 1.1.20?

[Rich Felker]

On Wed, Jul 11, 2018 at 09:17:01AM +0200, ardi wrote:
> Hi!
> 
> Is there a timeline for the 1.1.20 release? I'm starting a project,
> and I'd find it very convenient to know if 1.1.20 will be released
> this month, or if 1.1.19 is going to be kept as the current release
> for some months to come.

I don't have a date in mind, but letting it get drawn out for "some
months" again is definitely not something I want to have happen.
There's definitely enough time left in this month that a release
before the end should be possible. Assistance with testing/checks for
regressions would help a lot.

Rich

Re: Timeline for 1.1.20?

[ardi]

On Mon, Jul 16, 2018 at 7:20 PM, Rich Felker <dalias@libc.org> wrote:
> On Wed, Jul 11, 2018 at 09:17:01AM +0200, ardi wrote:
>> Hi!
>>
>> Is there a timeline for the 1.1.20 release? I'm starting a project,
>> and I'd find it very convenient to know if 1.1.20 will be released
>> this month, or if 1.1.19 is going to be kept as the current release
>> for some months to come.
>
> I don't have a date in mind, but letting it get drawn out for "some
> months" again is definitely not something I want to have happen.
> There's definitely enough time left in this month that a release
> before the end should be possible. Assistance with testing/checks for
> regressions would help a lot.

I'm not the best one for doing testing, as I don't have any previous
code based on musl, and moreover, my way of using it is going to be
non-obvious/marginal, as I'm working in being OS-independent, as I
wrote in the list months ago. There's a possibility that I end up
combining code from newlib and musl, but that's not certain at this
moment (the only certain thing for the moment is that musl has
everything I need, including a complete ELF dynamic linker/loader, and
that newlib follows the approach I want -syscall isolation- but it's
less complete and has no linker/loader). So, I need to dig deeply at
building both libraries, and finding my way during the process.

Said this, I'm starting my efforts through an Alpine Linux VM (for
obvious reasons, as musl is default there, and llvm is also there,
which is the compiler suite I use), so I'll be happy to test in that
VM any 1.1.20 release candidates you prepare, although I assume that
Alpine will be a platform for which you'll have testers already.

Thanks!

ardi

Re: Timeline for 1.1.20?

[Rob Landley]

On 07/16/2018 12:20 PM, Rich Felker wrote:
> On Wed, Jul 11, 2018 at 09:17:01AM +0200, ardi wrote:
>> Hi!
>>
>> Is there a timeline for the 1.1.20 release? I'm starting a project,
>> and I'd find it very convenient to know if 1.1.20 will be released
>> this month, or if 1.1.19 is going to be kept as the current release
>> for some months to come.
> 
> I don't have a date in mind, but letting it get drawn out for "some
> months" again is definitely not something I want to have happen.
> There's definitely enough time left in this month that a release
> before the end should be possible. Assistance with testing/checks for
> regressions would help a lot.

What kind of tests? I built an mcm-buildall.sh toolchain set and all the mkroot
targets with it and booted 'em to a shell prompt shortly after m68k went in. I
can do that again. But I'm not running a whole lot of test loads under the
result yet...

I added an m68k target to mcm-buildall.sh a _month_ ago, on the assumption that
a release with m68k was forthcoming:

  https://github.com/landley/mkroot/commit/b34fd3014357

The build has been broken for anything but musl-git ever since. (Also, should I
stop testing gcc 7.2? Is there a reason the default hasn't updated?)

Rob

Re: Timeline for 1.1.20?

[Christopher Friedt]

[-- Attachment #1: Type: text/plain, Size: 554 bytes --]

Rich,

have you considered a CI environment of some kind (e.g. Travis, GitLab)?

GitLab is my favourite, particularly because it can be hosted locally (i.e.
can interact with hardware if you so desire).

That means you can e.g. run quick unit tests inside of a Docker image for
various arches, and then run integration & system tests natively before a
release.

I'm particularly a big fan of the TDD approach, where unit tests are
written and passing for new features (and unit tests continue to pass for
old features) before a pull request is merge.

C

[-- Attachment #2: Type: text/html, Size: 859 bytes --]

Re: Timeline for 1.1.20?

[Rich Felker]

On Sat, Jul 28, 2018 at 01:43:34PM -0400, Christopher Friedt wrote:
> Rich,
> 
> have you considered a CI environment of some kind (e.g. Travis, GitLab)?
> 
> GitLab is my favourite, particularly because it can be hosted locally (i.e.
> can interact with hardware if you so desire).
> 
> That means you can e.g. run quick unit tests inside of a Docker image for
> various arches, and then run integration & system tests natively before a
> release.
> 
> I'm particularly a big fan of the TDD approach, where unit tests are
> written and passing for new features (and unit tests continue to pass for
> old features) before a pull request is merge.

Yes, but I don't know how to set it up, and any proper approach to
setting it up really shouldn't require the project maintainer to know
how, since it should revolve around a separate CI project pulling
musl, libc-test, and possibly other sources (e.g. mcm) as either
subrepos or part of the build scripts, then evaluating the resutls.

As for the immediate need, though, it's *not* fancy CI processes but
actual test coverage. I'm really wary of projects that have fancy
process frameworks but nothing to show for it.

In particular, coverage for changes since 1.1.19 would include:

- getrandom/getentropy basic functionality check
- setvbuf non-stub inplementation: basic functionality, check for
  writes outside the buffer, etc.
- malloc interposition: check that partial replacement doesn't result
  in unsafe behavior.
- pthread_create: confirm that scheduling and other attributes still
  work as expected after refactoring work.
- getddrinfo AI_ADDRCONFIG (can't really be tested without network
  namespaces though)

Particular bugfixes that call for functionality or regression tests:

b123f23 fix getopt wrongly treating colons in optstring as valid option chars
0cf5058 fix nl_langinfo_l(CODESET, loc) reporting wrong locale's value
282b1cd fix fmaf wrong result
ae2a01d fix wrong result in casin and many related complex functions
10e4bd3 fix incorrect results for catan with some inputs
4bf0717 fix return value of nice function
3f6dc30 fix out of bounds write for zero length buffer in gethostname
9be4ed5 getopt_long_only: don't prefix-match long-options that match short ones
55a661f fix iconv buffer overflow converting to legacy JIS-based encodings
99f4237 fix iconv conversion to UTF-32 with implicit (big) endianness
165a1e3 fix iconv mapping of big5-hkscs characters that map to two unicode chars
029c622 fix output size handling for multi-unicode-char big5-hkscs characters
5c8e692 inet_ntop: do not compress single zeros in IPv6
8b8fb7f correctly handle non-matching symbols in dladdr
9cad27a fix writes outside buffer by ungetc after setvbuf
b3fa0f2 fix regression in alignment of dirent structs produced by readdir

Some fixes that would not be confirmed with just libc-test, but that
needs more of a framework for covering multiple build configurations:

a7c53e0 fix out-of-tree build of crt files with stack protector enabled
e3c682a work around arm gcc's rejection of r7 asm constraints in thumb mode

For now it will have to suffice that we tested them by hand at the
time of fixing.

Rich

Re: Timeline for 1.1.20?

[Christopher Friedt]

On Sat, Jul 28, 2018 at 3:55 PM Rich Felker <dalias@libc.org> wrote:
> Yes, but I don't know how to set it up, and any proper approach to
> setting it up really shouldn't require the project maintainer to know
> how, since it should revolve around a separate CI project pulling
> musl, libc-test, and possibly other sources (e.g. mcm) as either
> subrepos or part of the build scripts, then evaluating the resutls.

I'll set up a .gitlab-ci.yml file - likely will use one 'pipelilne' to
trigger other pipelines that exercise all combinations here:

https://github.com/richfelker/musl-cross-make#supported-targets

Speaking of which, one of my eventual use-cases is an rtos that uses
the same syscall numbers as linux for each arch. Originally I was
using bionic libc, but it's just difficult to maintain a permanent
fork. Beyond syscall numbers, is there any specific reason that musl
requires linux?

> As for the immediate need, though, it's *not* fancy CI processes but
> actual test coverage. I'm really wary of projects that have fancy
> process frameworks but nothing to show for it.

Reports are useful. The fancy processes (automation, etc) just make
life easier in the end and they're pretty easy to set up after one or
two projects. I usually prefer to get it out of the way early. The
low-hanging fruit. For unit testing, it's helpful to adopt a
framework. gtest is good, although it assumes you have a working c++
compiler.

I get that you want to keep libc-test and musl separate for now, but
normally test code is part of the same repository as the source it's
testing. Maybe consider that in the future.

> In particular, coverage for changes since 1.1.19 would include:
>
> - getrandom/getentropy basic functionality check
> - setvbuf non-stub inplementation: basic functionality, check for
>   writes outside the buffer, etc.
> - malloc interposition: check that partial replacement doesn't result
>   in unsafe behavior.
> - pthread_create: confirm that scheduling and other attributes still
>   work as expected after refactoring work.
> - getddrinfo AI_ADDRCONFIG (can't really be tested without network
>   namespaces though)
>
> Particular bugfixes that call for functionality or regression tests:
>
> b123f23 fix getopt wrongly treating colons in optstring as valid option chars
> 0cf5058 fix nl_langinfo_l(CODESET, loc) reporting wrong locale's value
> 282b1cd fix fmaf wrong result
> ae2a01d fix wrong result in casin and many related complex functions
> 10e4bd3 fix incorrect results for catan with some inputs
> 4bf0717 fix return value of nice function
> 3f6dc30 fix out of bounds write for zero length buffer in gethostname
> 9be4ed5 getopt_long_only: don't prefix-match long-options that match short ones
> 55a661f fix iconv buffer overflow converting to legacy JIS-based encodings
> 99f4237 fix iconv conversion to UTF-32 with implicit (big) endianness
> 165a1e3 fix iconv mapping of big5-hkscs characters that map to two unicode chars
> 029c622 fix output size handling for multi-unicode-char big5-hkscs characters
> 5c8e692 inet_ntop: do not compress single zeros in IPv6
> 8b8fb7f correctly handle non-matching symbols in dladdr
> 9cad27a fix writes outside buffer by ungetc after setvbuf
> b3fa0f2 fix regression in alignment of dirent structs produced by readdir
>
> Some fixes that would not be confirmed with just libc-test, but that
> needs more of a framework for covering multiple build configurations:
>
> a7c53e0 fix out-of-tree build of crt files with stack protector enabled
> e3c682a work around arm gcc's rejection of r7 asm constraints in thumb mode

You've just listed off a number of work tickets ;-) I'll set something
up and then show you the results - my server is limited in horsepower
though, and (in terms of actual hardware), I'm limited to x86_64,
arm64, arm32 (various arch versions), and a mips32.

C

[RFC/RFT] musl: 1.1.20 prelease testing

[Christian Lamparter]

Rich Felker requested some field testing of his 1.1.20 release:
<http://www.openwall.com/lists/musl/2018/04/18/3>

"The biggest areas that need testing are:

- stdio locking after commits c21f750727515602a9e84f2a190ee8a0a2aeb2a1
  and c1014a812c90bab3c9c989863e4ebb129e987de6.

- reclaim_gaps since commit ce7ae11acfd9db8eb92cc6823c132e1825918d92.
  this could be tested by abusing __malloc_donate as if it were a
  public api.

- malloc interposition -- seeing if replacement allocators actually
  work. (not so important since it wouldn't be a regression if it's
  broken, but would be nice to know it works before announcing)"

<http://www.openwall.com/lists/musl/2018/07/28/4>
"In particular, coverage for changes since 1.1.19 would include:

- getrandom/getentropy basic functionality check
- setvbuf non-stub inplementation: basic functionality, check for
  writes outside the buffer, etc.
- malloc interposition: check that partial replacement doesn't result
  in unsafe behavior.
- pthread_create: confirm that scheduling and other attributes still
  work as expected after refactoring work.
- getddrinfo AI_ADDRCONFIG (can't really be tested without network
  namespaces though)

Particular bugfixes that call for functionality or regression tests:

b123f23 fix getopt wrongly treating colons in optstring as valid option chars
0cf5058 fix nl_langinfo_l(CODESET, loc) reporting wrong locale's value
282b1cd fix fmaf wrong result
ae2a01d fix wrong result in casin and many related complex functions
10e4bd3 fix incorrect results for catan with some inputs
4bf0717 fix return value of nice function
3f6dc30 fix out of bounds write for zero length buffer in gethostname
9be4ed5 getopt_long_only: don't prefix-match long-options that match short ones
55a661f fix iconv buffer overflow converting to legacy JIS-based encodings
99f4237 fix iconv conversion to UTF-32 with implicit (big) endianness
165a1e3 fix iconv mapping of big5-hkscs characters that map to two unicode chars
029c622 fix output size handling for multi-unicode-char big5-hkscs characters
5c8e692 inet_ntop: do not compress single zeros in IPv6
8b8fb7f correctly handle non-matching symbols in dladdr
9cad27a fix writes outside buffer by ungetc after setvbuf
b3fa0f2 fix regression in alignment of dirent structs produced by readdir"

Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
---
 toolchain/musl/common.mk                      |  6 +-
 ...ocket.h-fix-SO_PEERSEC-value-on-MIPS.patch | 59 -------------------
 .../patches/200-add_libssp_nonshared.patch    | 39 ++++++------
 toolchain/musl/patches/300-relative.patch     |  2 +-
 ...ribute-to-some-function-declarations.patch |  8 +--
 .../musl/patches/900-iconv_size_hack.patch    |  6 +-
 6 files changed, 32 insertions(+), 88 deletions(-)
 delete mode 100644 toolchain/musl/patches/010-sys-socket.h-fix-SO_PEERSEC-value-on-MIPS.patch

diff --git a/toolchain/musl/common.mk b/toolchain/musl/common.mk
index 87424646c3..3d58439a01 100644
--- a/toolchain/musl/common.mk
+++ b/toolchain/musl/common.mk
@@ -8,13 +8,13 @@ include $(TOPDIR)/rules.mk
 include $(INCLUDE_DIR)/target.mk
 
 PKG_NAME:=musl
-PKG_VERSION:=1.1.19
+PKG_VERSION:=1.1.20-pre
 PKG_RELEASE=1
 
 PKG_SOURCE_PROTO:=git
 PKG_SOURCE_SUBDIR:=$(PKG_NAME)-$(PKG_VERSION)
-PKG_SOURCE_VERSION:=55df09bfccbfe21fc9dd7d8f94550c0ff25ace04
-PKG_MIRROR_HASH:=eb94e4e7e94221dd8890afd9b29e2562c36cf5585649035349ca1c6c1c354f2b
+PKG_SOURCE_VERSION:=f2c6dbe2442027ed8fe0fa869918e41f495534d8
+PKG_MIRROR_HASH:=2ec8320cea2c560c9e9a01f7e3b8681f0113b380838194bcdf90562a38c847e4
 PKG_SOURCE_URL:=git://git.musl-libc.org/musl
 PKG_SOURCE:=$(PKG_NAME)-$(PKG_VERSION)-$(PKG_SOURCE_VERSION).tar.xz
 
diff --git a/toolchain/musl/patches/010-sys-socket.h-fix-SO_PEERSEC-value-on-MIPS.patch b/toolchain/musl/patches/010-sys-socket.h-fix-SO_PEERSEC-value-on-MIPS.patch
deleted file mode 100644
index 2319d9cb68..0000000000
--- a/toolchain/musl/patches/010-sys-socket.h-fix-SO_PEERSEC-value-on-MIPS.patch
+++ /dev/null
@@ -1,59 +0,0 @@
-From 4e0877a604bad684be020f68e96a05156131fd44 Mon Sep 17 00:00:00 2001
-From: Matthias Schiffer <mschiffer@universe-factory.net>
-Date: Sun, 24 Jun 2018 17:05:31 +0200
-Subject: [PATCH] sys/socket.h: fix SO_PEERSEC value on MIPS
-
-Differing from all other archs supported by musl, MIPS defines SO_PEERSEC
-to 30 instead of 31.
-
-Reported-by: Andrey Jr. Mlenikov <temnota.am@gmail.com>
----
- arch/mips/bits/socket.h    | 2 ++
- arch/mips64/bits/socket.h  | 2 ++
- arch/mipsn32/bits/socket.h | 2 ++
- include/sys/socket.h       | 3 +++
- 4 files changed, 9 insertions(+)
-
---- a/arch/mips/bits/socket.h
-+++ b/arch/mips/bits/socket.h
-@@ -48,5 +48,7 @@ struct cmsghdr {
- #define SO_SNDBUFFORCE  31
- #define SO_RCVBUFFORCE  33
- 
-+#define SO_PEERSEC      30
-+
- #define SOCK_NONBLOCK     0200
- #define SOCK_CLOEXEC  02000000
---- a/arch/mips64/bits/socket.h
-+++ b/arch/mips64/bits/socket.h
-@@ -64,5 +64,7 @@ struct cmsghdr {
- #define SO_SNDBUFFORCE  31
- #define SO_RCVBUFFORCE  33
- 
-+#define SO_PEERSEC      30
-+
- #define SOCK_NONBLOCK     0200
- #define SOCK_CLOEXEC  02000000
---- a/arch/mipsn32/bits/socket.h
-+++ b/arch/mipsn32/bits/socket.h
-@@ -48,5 +48,7 @@ struct cmsghdr {
- #define SO_SNDBUFFORCE  31
- #define SO_RCVBUFFORCE  33
- 
-+#define SO_PEERSEC      30
-+
- #define SOCK_NONBLOCK     0200
- #define SOCK_CLOEXEC  02000000
---- a/include/sys/socket.h
-+++ b/include/sys/socket.h
-@@ -201,7 +201,10 @@ struct linger {
- #define SO_TIMESTAMP            29
- #define SCM_TIMESTAMP           SO_TIMESTAMP
- 
-+#ifndef SO_PEERSEC
- #define SO_PEERSEC              31
-+#endif
-+
- #define SO_PASSSEC              34
- #define SO_TIMESTAMPNS          35
- #define SCM_TIMESTAMPNS         SO_TIMESTAMPNS
diff --git a/toolchain/musl/patches/200-add_libssp_nonshared.patch b/toolchain/musl/patches/200-add_libssp_nonshared.patch
index 7a2909461b..b8fa7b4b4f 100644
--- a/toolchain/musl/patches/200-add_libssp_nonshared.patch
+++ b/toolchain/musl/patches/200-add_libssp_nonshared.patch
@@ -4,11 +4,6 @@ Date: Mon, 22 Jun 2015 11:01:56 +0200
 Subject: [PATCH] Add libssp_nonshared.a so GCC's is not needed
 
 Signed-off-by: Steven Barth <steven@midlink.org>
----
- Makefile                                  | 10 ++++++++--
- libssp_nonshared/__stack_chk_fail_local.c |  2 ++
- 2 files changed, 10 insertions(+), 2 deletions(-)
- create mode 100644 libssp_nonshared/__stack_chk_fail_local.c
 
 --- a/Makefile
 +++ b/Makefile
@@ -21,21 +16,29 @@ Signed-off-by: Steven Barth <steven@midlink.org>
  ALL_TOOLS = obj/musl-gcc
  
  WRAPCC_GCC = gcc
-@@ -125,7 +125,8 @@ NOSSP_SRCS = $(wildcard crt/*.c) \
- 	src/thread/__set_thread_area.c src/thread/$(ARCH)/__set_thread_area.c \
- 	src/string/memset.c src/string/$(ARCH)/memset.c \
- 	src/string/memcpy.c src/string/$(ARCH)/memcpy.c \
--	ldso/dlstart.c ldso/dynlink.c
-+	ldso/dlstart.c ldso/dynlink.c \
-+	src/libssp_nonshared/__stack_chk_fail_local.c
- $(NOSSP_SRCS:%.c=obj/%.o) $(NOSSP_SRCS:%.c=obj/%.lo): CFLAGS_ALL += $(CFLAGS_NOSSP)
- 
- $(CRT_OBJS): CFLAGS_ALL += -DCRT
-@@ -168,6 +169,11 @@ lib/libc.a: $(AOBJS)
+@@ -86,7 +86,7 @@ else
+ 
+ all: $(ALL_LIBS) $(ALL_TOOLS)
+ 
+-OBJ_DIRS = $(sort $(patsubst %/,%,$(dir $(ALL_LIBS) $(ALL_TOOLS) $(ALL_OBJS) $(GENH) $(GENH_INT))) obj/include)
++OBJ_DIRS = $(sort $(patsubst %/,%,$(dir $(ALL_LIBS) $(ALL_TOOLS) $(ALL_OBJS) $(GENH) $(GENH_INT))) obj/include obj/libssp_nonshared)
+ 
+ $(ALL_LIBS) $(ALL_TOOLS) $(ALL_OBJS) $(ALL_OBJS:%.o=%.lo) $(GENH) $(GENH_INT): | $(OBJ_DIRS)
+
+@@ -113,6 +113,8 @@ obj/crt/rcrt1.o: $(srcdir)/ldso/dlstart.
+ 
+ obj/crt/Scrt1.o obj/crt/rcrt1.o: CFLAGS_ALL += -fPIC
+ 
++obj/libssp_nonshared/__stack_chk_fail_local.o: CFLAGS_ALL += $(CFLAGS_NOSSP)
++
+ OPTIMIZE_SRCS = $(wildcard $(OPTIMIZE_GLOBS:%=$(srcdir)/src/%))
+ $(OPTIMIZE_SRCS:$(srcdir)/%.c=obj/%.o) $(OPTIMIZE_SRCS:$(srcdir)/%.c=obj/%.lo): CFLAGS += -O3
+ 
+@@ -165,6 +166,11 @@ lib/libc.a: $(AOBJS)
  	$(AR) rc $@ $(AOBJS)
  	$(RANLIB) $@
  
-+lib/libssp_nonshared.a: obj/src/libssp_nonshared/__stack_chk_fail_local.o
++lib/libssp_nonshared.a: obj/libssp_nonshared/__stack_chk_fail_local.o
 +	rm -f $@
 +	$(AR) rc $@ $<
 +	$(RANLIB) $@
@@ -44,7 +47,7 @@ Signed-off-by: Steven Barth <steven@midlink.org>
  	rm -f $@
  	$(AR) rc $@
 --- /dev/null
-+++ b/src/libssp_nonshared/__stack_chk_fail_local.c
++++ b/libssp_nonshared/__stack_chk_fail_local.c
 @@ -0,0 +1,2 @@
 +#include "atomic.h"
 +void __attribute__((visibility ("hidden"))) __stack_chk_fail_local(void) { a_crash(); }
diff --git a/toolchain/musl/patches/300-relative.patch b/toolchain/musl/patches/300-relative.patch
index 7e1eb7d6bc..e34e60a09d 100644
--- a/toolchain/musl/patches/300-relative.patch
+++ b/toolchain/musl/patches/300-relative.patch
@@ -1,6 +1,6 @@
 --- a/Makefile
 +++ b/Makefile
-@@ -217,7 +217,7 @@ $(DESTDIR)$(includedir)/%: $(srcdir)/inc
+@@ -215,7 +215,7 @@ $(DESTDIR)$(includedir)/%: $(srcdir)/inc
  	$(INSTALL) -D -m 644 $< $@
  
  $(DESTDIR)$(LDSO_PATHNAME): $(DESTDIR)$(libdir)/libc.so
diff --git a/toolchain/musl/patches/400-Add-format-attribute-to-some-function-declarations.patch b/toolchain/musl/patches/400-Add-format-attribute-to-some-function-declarations.patch
index 915b0b7b47..f7eff9141f 100644
--- a/toolchain/musl/patches/400-Add-format-attribute-to-some-function-declarations.patch
+++ b/toolchain/musl/patches/400-Add-format-attribute-to-some-function-declarations.patch
@@ -102,7 +102,7 @@ Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
  #ifdef __cplusplus
  #define NULL 0L
  #else
-@@ -102,19 +110,19 @@ int puts(const char *);
+@@ -103,19 +111,19 @@ int puts(const char *);
  int printf(const char *__restrict, ...);
  int fprintf(FILE *__restrict, const char *__restrict, ...);
  int sprintf(char *__restrict, const char *__restrict, ...);
@@ -127,7 +127,7 @@ Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
  
  void perror(const char *);
  
-@@ -135,8 +143,8 @@ int pclose(FILE *);
+@@ -136,8 +144,8 @@ int pclose(FILE *);
  int fileno(FILE *);
  int fseeko(FILE *, off_t, int);
  off_t ftello(FILE *);
@@ -138,7 +138,7 @@ Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
  void flockfile(FILE *);
  int ftrylockfile(FILE *);
  void funlockfile(FILE *);
-@@ -175,8 +183,8 @@ int fileno_unlocked(FILE *);
+@@ -176,8 +184,8 @@ int fileno_unlocked(FILE *);
  int getw(FILE *);
  int putw(int, FILE *);
  char *fgetln(FILE *, size_t *);
@@ -149,7 +149,7 @@ Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
  #endif
  
  #ifdef _GNU_SOURCE
-@@ -198,6 +206,9 @@ typedef struct _IO_cookie_io_functions_t
+@@ -199,6 +207,9 @@ typedef struct _IO_cookie_io_functions_t
  FILE *fopencookie(void *, const char *, cookie_io_functions_t);
  #endif
  
diff --git a/toolchain/musl/patches/900-iconv_size_hack.patch b/toolchain/musl/patches/900-iconv_size_hack.patch
index 6200262b1d..461a204a4c 100644
--- a/toolchain/musl/patches/900-iconv_size_hack.patch
+++ b/toolchain/musl/patches/900-iconv_size_hack.patch
@@ -56,7 +56,7 @@
  		case SHIFT_JIS:
  			if (c < 128) break;
  			if (c-0xa1 <= 0xdf-0xa1) {
-@@ -510,6 +517,7 @@ size_t iconv(iconv_t cd, char **restrict
+@@ -518,6 +525,7 @@ size_t iconv(iconv_t cd, char **restrict
  			c = ksc[c][d];
  			if (!c) goto ilseq;
  			break;
@@ -64,7 +64,7 @@
  		default:
  			if (!c) break;
  			c = legacy_map(map, c);
-@@ -550,6 +558,7 @@ size_t iconv(iconv_t cd, char **restrict
+@@ -559,6 +567,7 @@ size_t iconv(iconv_t cd, char **restrict
  				}
  			}
  			goto subst;
@@ -72,7 +72,7 @@
  		case SHIFT_JIS:
  			if (c < 128) goto revout;
  			if (c == 0xa5) {
-@@ -623,6 +632,7 @@ size_t iconv(iconv_t cd, char **restrict
+@@ -632,6 +641,7 @@ size_t iconv(iconv_t cd, char **restrict
  			*(*out)++ = 'B';
  			*outb -= 8;
  			break;
-- 
2.18.0

musl "Linux-dependencies" info [was Re: [musl] Timeline for 1.1.20?]

[Rich Felker]

On Sun, Jul 29, 2018 at 07:40:25AM -0400, Christopher Friedt wrote:
> On Sat, Jul 28, 2018 at 3:55 PM Rich Felker <dalias@libc.org> wrote:
> > Yes, but I don't know how to set it up, and any proper approach to
> > setting it up really shouldn't require the project maintainer to know
> > how, since it should revolve around a separate CI project pulling
> > musl, libc-test, and possibly other sources (e.g. mcm) as either
> > subrepos or part of the build scripts, then evaluating the resutls.
> 
> I'll set up a .gitlab-ci.yml file - likely will use one 'pipelilne' to
> trigger other pipelines that exercise all combinations here:
> 
> https://github.com/richfelker/musl-cross-make#supported-targets
> 
> Speaking of which, one of my eventual use-cases is an rtos that uses
> the same syscall numbers as linux for each arch. Originally I was
> using bionic libc, but it's just difficult to maintain a permanent
> fork. Beyond syscall numbers, is there any specific reason that musl
> requires linux?

The other "Linux" interfaces it uses are parts of /proc (needed for
filling in gaps of some syscalls' functionality), some netlink
functionality (for network interface enumeration interfaces), and some
ioctls (for determining if something is a tty, some socket operations,
etc.).

There are also some de facto standard pathnames used for configuration
(resolv.conf, passwd, etc.) on top of the standard ones specified by
POSIX (/dev/null, etc.).

Some details are in the (incomplete, outdated, but IMO very good on
what it does have) musl documentation:

https://www.musl-libc.org/manual.html

Rich

Re: Timeline for 1.1.20?

[Rich Felker]

On Sun, Jul 29, 2018 at 07:40:25AM -0400, Christopher Friedt wrote:
> > As for the immediate need, though, it's *not* fancy CI processes but
> > actual test coverage. I'm really wary of projects that have fancy
> > process frameworks but nothing to show for it.
> 
> Reports are useful. The fancy processes (automation, etc) just make
> life easier in the end and they're pretty easy to set up after one or
> two projects. I usually prefer to get it out of the way early. The
> low-hanging fruit. For unit testing, it's helpful to adopt a
> framework. gtest is good, although it assumes you have a working c++
> compiler.
> 
> I get that you want to keep libc-test and musl separate for now, but
> normally test code is part of the same repository as the source it's
> testing. Maybe consider that in the future.

It's highly intentional that it's not done this way.

1. Having tests separately versioned from the code they're testing
   allows validation in both directions. By using an older version of
   the code being tested with the latest version of the tests, you can
   validate that the tests actually catch bugs which were known to be
   present in the older version.

2. For the most part, the tests are against interfaces, not a specific
   implementation of those interfaces. They're equally valid to run
   against other libcs, and by doing so you can compare correctness
   properties.

3. Storage/download size burden. roughly 1/3 of the size of gcc is
   testsuite, it's hundreds of MB, and the vast majority of that only
   makes sense to be run by people developing gcc, not end users
   building it. Making everyone download it is not friendly. For musl
   the numbers are smaller but the same principle applies.

4. Putting both in the same package is not really cross-compile
   friendly, since in general you need the package source on the build
   system but the tests on the host/target system.

Probably also some others I'm forgetting...

> > In particular, coverage for changes since 1.1.19 would include:
> >
> > - getrandom/getentropy basic functionality check
> > - setvbuf non-stub inplementation: basic functionality, check for
> >   writes outside the buffer, etc.
> > - malloc interposition: check that partial replacement doesn't result
> >   in unsafe behavior.
> > - pthread_create: confirm that scheduling and other attributes still
> >   work as expected after refactoring work.
> > - getddrinfo AI_ADDRCONFIG (can't really be tested without network
> >   namespaces though)
> >
> > Particular bugfixes that call for functionality or regression tests:
> >
> > b123f23 fix getopt wrongly treating colons in optstring as valid option chars
> > 0cf5058 fix nl_langinfo_l(CODESET, loc) reporting wrong locale's value
> > 282b1cd fix fmaf wrong result
> > ae2a01d fix wrong result in casin and many related complex functions
> > 10e4bd3 fix incorrect results for catan with some inputs
> > 4bf0717 fix return value of nice function
> > 3f6dc30 fix out of bounds write for zero length buffer in gethostname
> > 9be4ed5 getopt_long_only: don't prefix-match long-options that match short ones
> > 55a661f fix iconv buffer overflow converting to legacy JIS-based encodings
> > 99f4237 fix iconv conversion to UTF-32 with implicit (big) endianness
> > 165a1e3 fix iconv mapping of big5-hkscs characters that map to two unicode chars
> > 029c622 fix output size handling for multi-unicode-char big5-hkscs characters
> > 5c8e692 inet_ntop: do not compress single zeros in IPv6
> > 8b8fb7f correctly handle non-matching symbols in dladdr
> > 9cad27a fix writes outside buffer by ungetc after setvbuf
> > b3fa0f2 fix regression in alignment of dirent structs produced by readdir
> >
> > Some fixes that would not be confirmed with just libc-test, but that
> > needs more of a framework for covering multiple build configurations:
> >
> > a7c53e0 fix out-of-tree build of crt files with stack protector enabled
> > e3c682a work around arm gcc's rejection of r7 asm constraints in thumb mode
> 
> You've just listed off a number of work tickets ;-) I'll set something
> up and then show you the results - my server is limited in horsepower
> though, and (in terms of actual hardware), I'm limited to x86_64,
> arm64, arm32 (various arch versions), and a mips32.

Great.

Rich