[musl] m68k - malloc causing 'out of memory: my_alloc caller' in rsync

[musl] m68k - malloc causing 'out of memory: my_alloc caller' in rsync

[Immolo]

[-- Attachment #1: Type: text/plain, Size: 1216 bytes --]

Hi,

I've been testing using Gentoo on m68k with musl -1.2.4 over the last few
days and hit an interesting issue when running `emerge --sync` to update
the portage tree would cause the following error:

[receiver] out of memory: my_alloc caller (file=flist.c, line=311)
rsync error: error allocating core memory buffers (code 22) at util2.c(123)
[receiver=3.2.7]
[generator] out of memory: my_alloc caller (file=flist.c, line=311)
rsync error: error allocating core memory buffers (code 22) at util2.c(123)
[generator=3.2.7]
rsync: [receiver] write error: Broken pipe (32)
Full output here: https://bpa.st/3VDNM

Looking at the source code of the files it highlights it seems to be an
issue in the malloc:
https://github.com/WayneD/rsync/blob/master/util2.c#L123
https://github.com/WayneD/rsync/blob/master/flist.c#L311
https://github.com/WayneD/rsync/blob/master/fileio.c#L159

I've tested to see if a local rsync mirror will cause the same error with
random files and found it happens at around 62 files being mirrored but
size of the file does not matter.
I run musll on multiple architectures and this is the first time I've run
across it and have confirmed the Gentoo glibc m68k install does not run
into this.

[-- Attachment #2: Type: text/html, Size: 2295 bytes --]

Re: [musl] m68k - malloc causing 'out of memory: my_alloc caller' in rsync

[Rich Felker]

On Mon, Jun 26, 2023 at 08:34:04PM +0100, Immolo wrote:
> Hi,
> 
> I've been testing using Gentoo on m68k with musl -1.2.4 over the last few
> days and hit an interesting issue when running `emerge --sync` to update
> the portage tree would cause the following error:
> 
> [receiver] out of memory: my_alloc caller (file=flist.c, line=311)
> rsync error: error allocating core memory buffers (code 22) at util2.c(123)
> [receiver=3.2.7]
> [generator] out of memory: my_alloc caller (file=flist.c, line=311)
> rsync error: error allocating core memory buffers (code 22) at util2.c(123)
> [generator=3.2.7]
> rsync: [receiver] write error: Broken pipe (32)
> Full output here: https://bpa.st/3VDNM
> 
> Looking at the source code of the files it highlights it seems to be an
> issue in the malloc:
> https://github.com/WayneD/rsync/blob/master/util2.c#L123
> https://github.com/WayneD/rsync/blob/master/flist.c#L311
> https://github.com/WayneD/rsync/blob/master/fileio.c#L159
> 
> I've tested to see if a local rsync mirror will cause the same error with
> random files and found it happens at around 62 files being mirrored but
> size of the file does not matter.
> I run musll on multiple architectures and this is the first time I've run
> across it and have confirmed the Gentoo glibc m68k install does not run
> into this.

I guess you should try putting a breakpoint on that line in flist.c
and see what values are being passed to realloc_array.

Looking at the definition of realloc_array (at first I mistook this
for the new reallocarray function, but it's a custom thing in terms of
their my_alloc), this code does not look good. Unless max_alloc has
been set, there is no overflow checking, so aside from whatever is
going on with m68k, there is probably an exploitable integer overflow
bug:

https://github.com/WayneD/rsync/blob/6f3c5eccee6cf4dead68b9f3fda8fc2ff90dc311/util2.c#L87

I'm guessing the underlying problem is either some mismatched function
call signature that only happens to mismatch the call ABI on m68k, or
perhaps some weird effect of structs having almost no alignment on
m68k. But I without actually seeing the values involved at the point
of failure, it's hard to narrow it down.

Rich

Re: [musl] m68k - malloc causing 'out of memory: my_alloc caller' in rsync

[Immolo]

So after further investigation the cause of this issue was found to be
in QEMU and the fix is now in 8.0.3
(https://gitlab.com/qemu-project/qemu/-/commit/df1e45c9dfc8d5e9f8c19677799e8a77c601ce29).

On a side note I now have a full m68k musl stage3 working in Gentoo so
if anyone is interesting in testing then feel free to ping me as
immolo in either #musl or #gentoo-releng

On Mon, 26 Jun 2023 at 22:06, Rich Felker <dalias@libc.org> wrote:
>
> On Mon, Jun 26, 2023 at 08:34:04PM +0100, Immolo wrote:
> > Hi,
> >
> > I've been testing using Gentoo on m68k with musl -1.2.4 over the last few
> > days and hit an interesting issue when running `emerge --sync` to update
> > the portage tree would cause the following error:
> >
> > [receiver] out of memory: my_alloc caller (file=flist.c, line=311)
> > rsync error: error allocating core memory buffers (code 22) at util2.c(123)
> > [receiver=3.2.7]
> > [generator] out of memory: my_alloc caller (file=flist.c, line=311)
> > rsync error: error allocating core memory buffers (code 22) at util2.c(123)
> > [generator=3.2.7]
> > rsync: [receiver] write error: Broken pipe (32)
> > Full output here: https://bpa.st/3VDNM
> >
> > Looking at the source code of the files it highlights it seems to be an
> > issue in the malloc:
> > https://github.com/WayneD/rsync/blob/master/util2.c#L123
> > https://github.com/WayneD/rsync/blob/master/flist.c#L311
> > https://github.com/WayneD/rsync/blob/master/fileio.c#L159
> >
> > I've tested to see if a local rsync mirror will cause the same error with
> > random files and found it happens at around 62 files being mirrored but
> > size of the file does not matter.
> > I run musll on multiple architectures and this is the first time I've run
> > across it and have confirmed the Gentoo glibc m68k install does not run
> > into this.
>
> I guess you should try putting a breakpoint on that line in flist.c
> and see what values are being passed to realloc_array.
>
> Looking at the definition of realloc_array (at first I mistook this
> for the new reallocarray function, but it's a custom thing in terms of
> their my_alloc), this code does not look good. Unless max_alloc has
> been set, there is no overflow checking, so aside from whatever is
> going on with m68k, there is probably an exploitable integer overflow
> bug:
>
> https://github.com/WayneD/rsync/blob/6f3c5eccee6cf4dead68b9f3fda8fc2ff90dc311/util2.c#L87
>
> I'm guessing the underlying problem is either some mismatched function
> call signature that only happens to mismatch the call ABI on m68k, or
> perhaps some weird effect of structs having almost no alignment on
> m68k. But I without actually seeing the values involved at the point
> of failure, it's hard to narrow it down.
>
> Rich