Re: [musl] m68k - malloc causing 'out of memory: my_alloc caller' in rsync

[Immolo <immoloism@googlemail.com>]

So after further investigation the cause of this issue was found to be
in QEMU and the fix is now in 8.0.3
(https://gitlab.com/qemu-project/qemu/-/commit/df1e45c9dfc8d5e9f8c19677799e8a77c601ce29).

On a side note I now have a full m68k musl stage3 working in Gentoo so
if anyone is interesting in testing then feel free to ping me as
immolo in either #musl or #gentoo-releng

On Mon, 26 Jun 2023 at 22:06, Rich Felker <dalias@libc.org> wrote:
>
> On Mon, Jun 26, 2023 at 08:34:04PM +0100, Immolo wrote:
> > Hi,
> >
> > I've been testing using Gentoo on m68k with musl -1.2.4 over the last few
> > days and hit an interesting issue when running `emerge --sync` to update
> > the portage tree would cause the following error:
> >
> > [receiver] out of memory: my_alloc caller (file=flist.c, line=311)
> > rsync error: error allocating core memory buffers (code 22) at util2.c(123)
> > [receiver=3.2.7]
> > [generator] out of memory: my_alloc caller (file=flist.c, line=311)
> > rsync error: error allocating core memory buffers (code 22) at util2.c(123)
> > [generator=3.2.7]
> > rsync: [receiver] write error: Broken pipe (32)
> > Full output here: https://bpa.st/3VDNM
> >
> > Looking at the source code of the files it highlights it seems to be an
> > issue in the malloc:
> > https://github.com/WayneD/rsync/blob/master/util2.c#L123
> > https://github.com/WayneD/rsync/blob/master/flist.c#L311
> > https://github.com/WayneD/rsync/blob/master/fileio.c#L159
> >
> > I've tested to see if a local rsync mirror will cause the same error with
> > random files and found it happens at around 62 files being mirrored but
> > size of the file does not matter.
> > I run musll on multiple architectures and this is the first time I've run
> > across it and have confirmed the Gentoo glibc m68k install does not run
> > into this.
>
> I guess you should try putting a breakpoint on that line in flist.c
> and see what values are being passed to realloc_array.
>
> Looking at the definition of realloc_array (at first I mistook this
> for the new reallocarray function, but it's a custom thing in terms of
> their my_alloc), this code does not look good. Unless max_alloc has
> been set, there is no overflow checking, so aside from whatever is
> going on with m68k, there is probably an exploitable integer overflow
> bug:
>
> https://github.com/WayneD/rsync/blob/6f3c5eccee6cf4dead68b9f3fda8fc2ff90dc311/util2.c#L87
>
> I'm guessing the underlying problem is either some mismatched function
> call signature that only happens to mismatch the call ABI on m68k, or
> perhaps some weird effect of structs having almost no alignment on
> m68k. But I without actually seeing the values involved at the point
> of failure, it's hard to narrow it down.
>
> Rich