Re: [musl] m68k - malloc causing 'out of memory: my_alloc caller' in rsync

[Rich Felker <dalias@libc.org>]

On Mon, Jun 26, 2023 at 08:34:04PM +0100, Immolo wrote:
> Hi,
> 
> I've been testing using Gentoo on m68k with musl -1.2.4 over the last few
> days and hit an interesting issue when running `emerge --sync` to update
> the portage tree would cause the following error:
> 
> [receiver] out of memory: my_alloc caller (file=flist.c, line=311)
> rsync error: error allocating core memory buffers (code 22) at util2.c(123)
> [receiver=3.2.7]
> [generator] out of memory: my_alloc caller (file=flist.c, line=311)
> rsync error: error allocating core memory buffers (code 22) at util2.c(123)
> [generator=3.2.7]
> rsync: [receiver] write error: Broken pipe (32)
> Full output here: https://bpa.st/3VDNM
> 
> Looking at the source code of the files it highlights it seems to be an
> issue in the malloc:
> https://github.com/WayneD/rsync/blob/master/util2.c#L123
> https://github.com/WayneD/rsync/blob/master/flist.c#L311
> https://github.com/WayneD/rsync/blob/master/fileio.c#L159
> 
> I've tested to see if a local rsync mirror will cause the same error with
> random files and found it happens at around 62 files being mirrored but
> size of the file does not matter.
> I run musll on multiple architectures and this is the first time I've run
> across it and have confirmed the Gentoo glibc m68k install does not run
> into this.

I guess you should try putting a breakpoint on that line in flist.c
and see what values are being passed to realloc_array.

Looking at the definition of realloc_array (at first I mistook this
for the new reallocarray function, but it's a custom thing in terms of
their my_alloc), this code does not look good. Unless max_alloc has
been set, there is no overflow checking, so aside from whatever is
going on with m68k, there is probably an exploitable integer overflow
bug:

https://github.com/WayneD/rsync/blob/6f3c5eccee6cf4dead68b9f3fda8fc2ff90dc311/util2.c#L87

I'm guessing the underlying problem is either some mismatched function
call signature that only happens to mismatch the call ABI on m68k, or
perhaps some weird effect of structs having almost no alignment on
m68k. But I without actually seeing the values involved at the point
of failure, it's hard to narrow it down.

Rich