mailing list of musl libc
 help / color / Atom feed
* Issue with musl and valgrind
@ 2020-01-06 11:01 d.dorau
  2020-01-06 13:56 ` Rich Felker
  0 siblings, 1 reply; 4+ messages in thread
From: d.dorau @ 2020-01-06 11:01 UTC (permalink / raw)
  To: musl

[-- Attachment #1: Type: text/plain, Size: 3003 bytes --]

Hello and a happy new year!

I'm looking for some help/advice regarding the use of valgrind with musl.
In my attempts to use valgrind with musl I ran into a reproducible crash. 
I found others
experienced this earlier as well, but with no apparent solution:

https://forum.archive.openwrt.org/viewtopic.php?id=70169

==3915== Invalid read of size 4
==3915==    at 0x48C3154: free (in 
/usr/lib/valgrind/vgpreload_memcheck-mips32-linux.so)
==3915==    by 0x4088C04: ??? (in /lib/libc.so)
==3915==  Address  is not stack'd, malloc'd or (recently) free'd

I did some debugging and came to a plausible reason, and with a quick
hack I could get it to run.

My findings are that when musl loads vgpreload_memcheck-mips32-linux.so,
malloc/free are replaced by the implementation provided from valgrind, as 
expected.

The free() call that leads to the crash originates from musl's 
dynlink.c:782 in
map_library().

The crash in valgrind's vgpreload_memcheck-mips32-linux.so is located in 
coregrind/m_replacemalloc/vg_replace_malloc.c:184 as part of the macro 
DO_INIT


static int init_done;
#define DO_INIT if (UNLIKELY(!init_done)) init()
[...]
#define FREE(soname, fnname, vg_replacement) \
   \
   void VG_REPLACE_FUNCTION_EZU(10050,soname,fnname) (void *p); \
   void VG_REPLACE_FUNCTION_EZU(10050,soname,fnname) (void *p)  \
   { \
      DO_INIT; \
      MALLOC_TRACE(#fnname "(%p)\n", p ); \
      if (p == NULL)  \
         return; \
      (void)VALGRIND_NON_SIMD_CALL1( info.tl_##vg_replacement, p ); \
   }

that is called in the free function provided by valgrind.

Calculating back the crash address 0x1e0d8 from the example above leads to 
the
address of the variable "init_done" of the valgrind code in the ELF file 
itself!

My interpretation is that although the relocation of the new free function 
has
apprently been done, the variables (and maybe other called functions) used 
*inside*
the valgrind library have not been relocated yet, so that their addresses 
still
point to the bare address in the ELF file. (I hope the explanation makes 
sense.)

I could avoid the crash by patching musl so that dynlink.c always calls 
musl's
own malloc/free function (by renaming and an additional wrapper for the 
exported
symbols).

Since I don't have in-depth knowledge about the inner workings of musl's 
shared
library loader, I'm now looking for advice on how to solve this issue.

At first glance the fact that functions inside the loaded library are 
called before
all relocations are done looks problematic to me. 
Assuming this would be solved, wouldn't it be desirable to ensure that 
malloc/free
calls inside musl won't be replaced by external libraries so that we don't 
get
stale allocations left from the old allocator?


Best regards,
Daniel





--
AVM Audiovisuelles Marketing und Computersysteme GmbH
Alt-Moabit 95, 10559 Berlin
HRB 23075 AG Charlottenburg
Geschäftsführer: Johannes Nill
 

[-- Attachment #2: Type: text/html, Size: 5564 bytes --]

<font size=2 face="Courier New">Hello and a happy new year!</font>
<br>
<br><font size=2 face="Courier New">I'm looking for some help/advice regarding
the use of valgrind with musl.</font>
<br><font size=2 face="Courier New">In my attempts to use valgrind with
musl I ran into a reproducible crash. I found others</font>
<br><font size=2 face="Courier New">experienced this earlier as well, but
with no apparent solution:</font>
<br>
<br><a href="https://forum.archive.openwrt.org/viewtopic.php?id=70169"><font size=2 face="Courier New">https://forum.archive.openwrt.org/viewtopic.php?id=70169</font></a>
<br>
<br><font size=2 face="Courier New">==3915== Invalid read of size 4<br>
==3915== &nbsp; &nbsp;at 0x48C3154: free (in /usr/lib/valgrind/vgpreload_memcheck-mips32-linux.so)<br>
==3915== &nbsp; &nbsp;by 0x4088C04: ??? (in /lib/libc.so)<br>
==3915== &nbsp;Address &nbsp;is not stack'd, malloc'd or (recently) free'd</font>
<br>
<br><font size=2 face="Courier New">I did some debugging and came to a
plausible reason, and with a quick</font>
<br><font size=2 face="Courier New">hack I could get it to run.</font>
<br>
<br><font size=2 face="Courier New">My findings are that when musl loads
vgpreload_memcheck-mips32-linux.so,</font>
<br><font size=2 face="Courier New">malloc/free are replaced by the implementation
provided from valgrind, as expected.</font>
<br>
<br><font size=2 face="Courier New">The free() call that leads to the crash
originates from musl's dynlink.c:782 in</font>
<br><font size=2 face="Courier New">map_library().</font>
<br>
<br><font size=2 face="Courier New">The crash in valgrind's vgpreload_memcheck-mips32-linux.so
is located in</font><font size=2 face="Courier New"> </font>
<br><font size=2 face="Courier New">coregrind/m_replacemalloc/vg_replace_malloc.c:184
as part of the macro DO_INIT</font>
<br>
<br>
<br><font size=2 face="Courier New">static int init_done;</font>
<br><font size=2 face="Courier New">#define DO_INIT if (UNLIKELY(!init_done))
init()</font>
<br><font size=2 face="Courier New">[...]</font>
<br><font size=2 face="Courier New">#define FREE(soname, fnname, vg_replacement)
\</font>
<br><font size=2 face="Courier New">&nbsp; &nbsp;\</font>
<br><font size=2 face="Courier New">&nbsp; &nbsp;void VG_REPLACE_FUNCTION_EZU(10050,soname,fnname)
(void *p); \</font>
<br><font size=2 face="Courier New">&nbsp; &nbsp;void VG_REPLACE_FUNCTION_EZU(10050,soname,fnname)
(void *p) &nbsp;\</font>
<br><font size=2 face="Courier New">&nbsp; &nbsp;{ \</font>
<br><font size=2 face="Courier New">&nbsp; &nbsp; &nbsp; DO_INIT; \</font>
<br><font size=2 face="Courier New">&nbsp; &nbsp; &nbsp; MALLOC_TRACE(#fnname
&quot;(%p)\n&quot;, p ); \</font>
<br><font size=2 face="Courier New">&nbsp; &nbsp; &nbsp; if (p == NULL)
&nbsp;\</font>
<br><font size=2 face="Courier New">&nbsp; &nbsp; &nbsp; &nbsp; &nbsp;return;
\</font>
<br><font size=2 face="Courier New">&nbsp; &nbsp; &nbsp; (void)VALGRIND_NON_SIMD_CALL1(
info.tl_##vg_replacement, p ); \</font>
<br><font size=2 face="Courier New">&nbsp; &nbsp;}</font>
<br>
<br><font size=2 face="Courier New">that is called in the free function
provided by valgrind.</font>
<br>
<br><font size=2 face="Courier New">Calculating back the crash address
0x1e0d8 from the example above leads to the</font>
<br><font size=2 face="Courier New">address of the variable &quot;init_done&quot;
of the valgrind code in the ELF file itself!</font>
<br>
<br><font size=2 face="Courier New">My interpretation is that although
the relocation of the new free function has</font>
<br><font size=2 face="Courier New">apprently been done, the variables
(and maybe other called functions) used *inside*</font>
<br><font size=2 face="Courier New">the valgrind library have not been
relocated yet, so that their addresses still</font>
<br><font size=2 face="Courier New">point to the bare address in the ELF
file. (I hope the explanation makes sense.)</font>
<br>
<br><font size=2 face="Courier New">I could avoid the crash by patching
musl so that dynlink.c always calls musl's</font>
<br><font size=2 face="Courier New">own malloc/free function (by renaming
and an additional wrapper for the exported</font>
<br><font size=2 face="Courier New">symbols).</font>
<br>
<br><font size=2 face="Courier New">Since I don't have in-depth knowledge
about the inner workings of musl's shared</font>
<br><font size=2 face="Courier New">library loader, I'm now looking for
advice on how to solve this issue.</font>
<br>
<br><font size=2 face="Courier New">At first glance the fact that functions
inside the loaded library are called before</font>
<br><font size=2 face="Courier New">all relocations are done looks problematic
to me. </font>
<br><font size=2 face="Courier New">Assuming this would be solved, wouldn't
it be desirable to ensure that malloc/free</font>
<br><font size=2 face="Courier New">calls inside musl won't be replaced
by external libraries so that we don't get</font>
<br><font size=2 face="Courier New">stale allocations left from the old
allocator?</font>
<br>
<br>
<br><font size=2 face="Courier New">Best regards,</font>
<br><font size=2 face="Courier New">Daniel</font>
<br>
<br>
<br>
<br>
<br>
<br><font size=2 face="Courier New">--<br>
AVM Audiovisuelles Marketing und Computersysteme GmbH<br>
Alt-Moabit 95, 10559 Berlin<br>
HRB 23075 AG Charlottenburg<br>
Geschäftsführer: Johannes Nill</font><font size=2 face="sans-serif"><br>
</font><font size=2 face="Courier New"> </font>

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Issue with musl and valgrind
  2020-01-06 11:01 Issue with musl and valgrind d.dorau
@ 2020-01-06 13:56 ` Rich Felker
  2020-01-06 16:24   ` Re[2]: " d.dorau
  0 siblings, 1 reply; 4+ messages in thread
From: Rich Felker @ 2020-01-06 13:56 UTC (permalink / raw)
  To: musl; +Cc: d.dorau

On Mon, Jan 06, 2020 at 12:01:34PM +0100, d.dorau@avm.de wrote:
> Hello and a happy new year!
> 
> I'm looking for some help/advice regarding the use of valgrind with musl.
> In my attempts to use valgrind with musl I ran into a reproducible crash. 
> I found others
> experienced this earlier as well, but with no apparent solution:
> 
> https://forum.archive.openwrt.org/viewtopic.php?id=70169
> 
> ==3915== Invalid read of size 4
> ==3915==    at 0x48C3154: free (in 
> /usr/lib/valgrind/vgpreload_memcheck-mips32-linux.so)
> ==3915==    by 0x4088C04: ??? (in /lib/libc.so)
> ==3915==  Address  is not stack'd, malloc'd or (recently) free'd
> 
> I did some debugging and came to a plausible reason, and with a quick
> hack I could get it to run.
> 
> My findings are that when musl loads vgpreload_memcheck-mips32-linux.so,
> malloc/free are replaced by the implementation provided from valgrind, as 
> expected.
> 
> The free() call that leads to the crash originates from musl's 
> dynlink.c:782 in
> map_library().
> 
> The crash in valgrind's vgpreload_memcheck-mips32-linux.so is located in 
> coregrind/m_replacemalloc/vg_replace_malloc.c:184 as part of the macro 
> DO_INIT
> 
> 
> static int init_done;
> #define DO_INIT if (UNLIKELY(!init_done)) init()
> [...]
> #define FREE(soname, fnname, vg_replacement) \
>    \
>    void VG_REPLACE_FUNCTION_EZU(10050,soname,fnname) (void *p); \
>    void VG_REPLACE_FUNCTION_EZU(10050,soname,fnname) (void *p)  \
>    { \
>       DO_INIT; \
>       MALLOC_TRACE(#fnname "(%p)\n", p ); \
>       if (p == NULL)  \
>          return; \
>       (void)VALGRIND_NON_SIMD_CALL1( info.tl_##vg_replacement, p ); \
>    }
> 
> that is called in the free function provided by valgrind.
> 
> Calculating back the crash address 0x1e0d8 from the example above leads to 
> the
> address of the variable "init_done" of the valgrind code in the ELF file 
> itself!
> 
> My interpretation is that although the relocation of the new free function 
> has
> apprently been done, the variables (and maybe other called functions) used 
> *inside*
> the valgrind library have not been relocated yet, so that their addresses 
> still
> point to the bare address in the ELF file. (I hope the explanation makes 
> sense.)
> 
> I could avoid the crash by patching musl so that dynlink.c always calls 
> musl's
> own malloc/free function (by renaming and an additional wrapper for the 
> exported
> symbols).
> 
> Since I don't have in-depth knowledge about the inner workings of musl's 
> shared
> library loader, I'm now looking for advice on how to solve this issue.
> 
> At first glance the fact that functions inside the loaded library are 
> called before
> all relocations are done looks problematic to me. 
> Assuming this would be solved, wouldn't it be desirable to ensure that 
> malloc/free
> calls inside musl won't be replaced by external libraries so that we don't 
> get
> stale allocations left from the old allocator?

This is really a bug in valgrind, that it's relying on a mix of the
dynamic linker and its own mechanism for interposing malloc. Either
the interposition should not take place until after line 1913 of
__dls3 in dynlink.c (this is what would happen if you did it via
LD_PRELOAD without valgrind), or valgrind should do its own loading
and relocation of vgpreload_memcheck-*.so independent of the tracee's
dynamic linker.

On most archs, the problem does not manifest as things are setup so
that it's safe to call free in an unrelocated context. However
realloc is gratuitously not safe and Adélie Linux has a patch for it
to do it like free:

https://code.foxkit.us/adelie/packages/blob/master/user/valgrind/realloc.patch

But on MIPS, where there are no PC-relative references and everything
goes through the GOT, this blows up.

I'm not sure how practical it is to get valgrind to fix this upstream.
In the either/or above, if the first course of action is taken, the
mechanism should probably be by inserting a breakpoint (or valgrind's
equivalent) at the main program's e_entry address and not doing the
replacement hack until then. Alternatively, it could just rely on
LD_PRELOAD working without trying to do additional tricks, but maybe
they have a good reason they're not doing that.

Rich


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re[2]: Issue with musl and valgrind
  2020-01-06 13:56 ` Rich Felker
@ 2020-01-06 16:24   ` " d.dorau
  2020-01-06 16:32     ` Rich Felker
  0 siblings, 1 reply; 4+ messages in thread
From: d.dorau @ 2020-01-06 16:24 UTC (permalink / raw)
  To: Rich Felker; +Cc: musl

[-- Attachment #1: Type: text/plain, Size: 2550 bytes --]

"Rich Felker" <dalias@aerifal.cx> schrieb am 06.01.2020 14:56:23:
[...]
> 
> This is really a bug in valgrind, that it's relying on a mix of the
> dynamic linker and its own mechanism for interposing malloc. Either
> the interposition should not take place until after line 1913 of
> __dls3 in dynlink.c (this is what would happen if you did it via
> LD_PRELOAD without valgrind), or valgrind should do its own loading
> and relocation of vgpreload_memcheck-*.so independent of the tracee's
> dynamic linker.
> 
> On most archs, the problem does not manifest as things are setup so
> that it's safe to call free in an unrelocated context. However
> realloc is gratuitously not safe and Adélie Linux has a patch for it
> to do it like free:
> 
> 
https://code.foxkit.us/adelie/packages/blob/master/user/valgrind/realloc.patch

> 
> But on MIPS, where there are no PC-relative references and everything
> goes through the GOT, this blows up.
> 
> I'm not sure how practical it is to get valgrind to fix this upstream.
> In the either/or above, if the first course of action is taken, the
> mechanism should probably be by inserting a breakpoint (or valgrind's
> equivalent) at the main program's e_entry address and not doing the
> replacement hack until then. Alternatively, it could just rely on
> LD_PRELOAD working without trying to do additional tricks, but maybe
> they have a good reason they're not doing that.
> 
> Rich

I was not aware of this when debugging the crash, but after looking at 
additional
debug statements and valgrind code my current understanding is that 
valgrind
actually inserts

LD_PRELOAD="
/usr/lib/valgrind/vgpreload_core-mips32-linux.so:/usr/lib/valgrind/vgpreload_memcheck-mips32-linux.so"

into the environment before loading the executable to test.
(coregrind/m_initimg/initimg-linux.c:123)

425:11:32:696 --4757:1:    main Split up command line
425:11:32:696 --4757:1:    main (early_) Process Valgrind's command line 
options
425:11:32:696 --4757:1:    main Create initial image
425:11:32:712 --4757:1: initimg Loading client
425:11:32:712 --4757:1: initimg Setup client env
425:11:32:712 --4757:2: initimg   preload_string:
425:11:32:728 --4757:2: initimg 
"/usr/lib/valgrind/vgpreload_core-mips32-linux.so:/usr/lib/valgrind/vgpreload_memcheck-mips32-linux.so"
425:11:32:728 --4757:1: initimg Setup client stack: size will be 1048576


Is my understanding correct that you would expect it to not run into
the crash I described earlier then?


Daniel

[-- Attachment #2: Type: text/html, Size: 3606 bytes --]

<tt><font size=2>&quot;Rich Felker&quot; &lt;dalias@aerifal.cx&gt; schrieb
am 06.01.2020 14:56:23:<br>
[...]<br>
&gt; <br>
&gt; This is really a bug in valgrind, that it's relying on a mix of the<br>
&gt; dynamic linker and its own mechanism for interposing malloc. Either<br>
&gt; the interposition should not take place until after line 1913 of<br>
&gt; __dls3 in dynlink.c (this is what would happen if you did it via<br>
&gt; LD_PRELOAD without valgrind), or valgrind should do its own loading<br>
&gt; and relocation of vgpreload_memcheck-*.so independent of the tracee's<br>
&gt; dynamic linker.<br>
&gt; <br>
&gt; On most archs, the problem does not manifest as things are setup so<br>
&gt; that it's safe to call free in an unrelocated context. However<br>
&gt; realloc is gratuitously not safe and Adélie Linux has a patch for
it<br>
&gt; to do it like free:<br>
&gt; <br>
&gt; </font></tt><a href=https://code.foxkit.us/adelie/packages/blob/master/user/valgrind/realloc.patch><tt><font size=2>https://code.foxkit.us/adelie/packages/blob/master/user/valgrind/realloc.patch</font></tt></a><tt><font size=2><br>
&gt; <br>
&gt; But on MIPS, where there are no PC-relative references and everything<br>
&gt; goes through the GOT, this blows up.<br>
&gt; <br>
&gt; I'm not sure how practical it is to get valgrind to fix this upstream.<br>
&gt; In the either/or above, if the first course of action is taken, the<br>
&gt; mechanism should probably be by inserting a breakpoint (or valgrind's<br>
&gt; equivalent) at the main program's e_entry address and not doing the<br>
&gt; replacement hack until then. Alternatively, it could just rely on<br>
&gt; LD_PRELOAD working without trying to do additional tricks, but maybe<br>
&gt; they have a good reason they're not doing that.<br>
&gt; <br>
&gt; Rich<br>
</font></tt>
<br><tt><font size=2>I was not aware of this when debugging the crash,
but after looking at additional</font></tt>
<br><tt><font size=2>debug statements and valgrind code my current understanding
is that valgrind</font></tt>
<br><tt><font size=2>actually inserts</font></tt>
<br><tt><font size=2><br>
LD_PRELOAD=&quot;/usr/lib/valgrind/vgpreload_core-mips32-linux.so:/usr/lib/valgrind/vgpreload_memcheck-mips32-linux.so&quot;</font></tt>
<br>
<br><tt><font size=2>into the environment before loading the executable
to test.</font></tt>
<br><tt><font size=2>(coregrind/m_initimg/initimg-linux.c:123)</font></tt>
<br>
<br><tt><font size=2>425:11:32:696 --4757:1: &nbsp; &nbsp;main Split up
command line</font></tt>
<br><tt><font size=2>425:11:32:696 --4757:1: &nbsp; &nbsp;main (early_)
Process Valgrind's command line options</font></tt>
<br><tt><font size=2>425:11:32:696 --4757:1: &nbsp; &nbsp;main Create initial
image</font></tt>
<br><tt><font size=2>425:11:32:712 --4757:1: initimg Loading client</font></tt>
<br><tt><font size=2>425:11:32:712 --4757:1: initimg Setup client env</font></tt>
<br><tt><font size=2>425:11:32:712 --4757:2: initimg &nbsp; preload_string:</font></tt>
<br><tt><font size=2>425:11:32:728 --4757:2: initimg &nbsp; &nbsp; &quot;/usr/lib/valgrind/vgpreload_core-mips32-linux.so:/usr/lib/valgrind/vgpreload_memcheck-mips32-linux.so&quot;</font></tt>
<br><tt><font size=2>425:11:32:728 --4757:1: initimg Setup client stack:
size will be 1048576</font></tt>
<br>
<br>
<br><tt><font size=2>Is my understanding correct that you would expect
it to not run into</font></tt>
<br><tt><font size=2>the crash I described earlier then?</font></tt>
<br>
<br><tt><font size=2><br>
Daniel</font></tt>
<br>

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: Issue with musl and valgrind
  2020-01-06 16:24   ` Re[2]: " d.dorau
@ 2020-01-06 16:32     ` Rich Felker
  0 siblings, 0 replies; 4+ messages in thread
From: Rich Felker @ 2020-01-06 16:32 UTC (permalink / raw)
  To: d.dorau; +Cc: musl

On Mon, Jan 06, 2020 at 05:24:35PM +0100, d.dorau@avm.de wrote:
> "Rich Felker" <dalias@aerifal.cx> schrieb am 06.01.2020 14:56:23:
> [...]
> > 
> > This is really a bug in valgrind, that it's relying on a mix of the
> > dynamic linker and its own mechanism for interposing malloc. Either
> > the interposition should not take place until after line 1913 of
> > __dls3 in dynlink.c (this is what would happen if you did it via
> > LD_PRELOAD without valgrind), or valgrind should do its own loading
> > and relocation of vgpreload_memcheck-*.so independent of the tracee's
> > dynamic linker.
> > 
> > On most archs, the problem does not manifest as things are setup so
> > that it's safe to call free in an unrelocated context. However
> > realloc is gratuitously not safe and Adélie Linux has a patch for it
> > to do it like free:
> > 
> > 
> https://code.foxkit.us/adelie/packages/blob/master/user/valgrind/realloc.patch
> 
> > 
> > But on MIPS, where there are no PC-relative references and everything
> > goes through the GOT, this blows up.
> > 
> > I'm not sure how practical it is to get valgrind to fix this upstream.
> > In the either/or above, if the first course of action is taken, the
> > mechanism should probably be by inserting a breakpoint (or valgrind's
> > equivalent) at the main program's e_entry address and not doing the
> > replacement hack until then. Alternatively, it could just rely on
> > LD_PRELOAD working without trying to do additional tricks, but maybe
> > they have a good reason they're not doing that.
> > 
> > Rich
> 
> I was not aware of this when debugging the crash, but after looking at 
> additional
> debug statements and valgrind code my current understanding is that 
> valgrind
> actually inserts
> 
> LD_PRELOAD="
> /usr/lib/valgrind/vgpreload_core-mips32-linux.so:/usr/lib/valgrind/vgpreload_memcheck-mips32-linux.so"
> 
> into the environment before loading the executable to test.
> (coregrind/m_initimg/initimg-linux.c:123)
> 
> 425:11:32:696 --4757:1:    main Split up command line
> 425:11:32:696 --4757:1:    main (early_) Process Valgrind's command line 
> options
> 425:11:32:696 --4757:1:    main Create initial image
> 425:11:32:712 --4757:1: initimg Loading client
> 425:11:32:712 --4757:1: initimg Setup client env
> 425:11:32:712 --4757:2: initimg   preload_string:
> 425:11:32:728 --4757:2: initimg 
> "/usr/lib/valgrind/vgpreload_core-mips32-linux.so:/usr/lib/valgrind/vgpreload_memcheck-mips32-linux.so"
> 425:11:32:728 --4757:1: initimg Setup client stack: size will be 1048576
> 
> 
> Is my understanding correct that you would expect it to not run into
> the crash I described earlier then?

No, it does something explicitly wrong that causes the crash, and now
I'm starting to remember. What it does is hook the mmap of
vgpreload_memcheck-*.so and activate its redirections as soon as it's
mapped into memory. This activation should instead be moved to e_entry
point time.

BTW you accidentally un-CC'd the list; I bounced the email back on to
reply. Please keep replies on-list so others can follow.

Rich


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, back to index

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-01-06 11:01 Issue with musl and valgrind d.dorau
2020-01-06 13:56 ` Rich Felker
2020-01-06 16:24   ` Re[2]: " d.dorau
2020-01-06 16:32     ` Rich Felker

mailing list of musl libc

Archives are clonable: git clone --mirror http://inbox.vuxu.org/musl

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://inbox.vuxu.org/vuxu.archive.musl


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git