Re: incompatibility between libtheora/mmx and musl ?

[Szabolcs Nagy <nsz@port70.net>]

* u-uy74@aetey.se <u-uy74@aetey.se> [2016-09-14 12:32:53 +0200]:
> On Tue, Sep 13, 2016 at 04:43:04PM -0400, Rich Felker wrote:
> > > Libtheora contains an assembler part for ia32 and x86_64 which has been
> > > in use for many years, with at least both glibc and uclibc.
> > > 
> > > Now, musl-based builds of libtheora for ia32 with this code enabled
> > > lead to the encoder segfaulting. No problem when using the C version.
> 
> > >  https://trac.xiph.org/ticket/2287
> > > 
> > > There is also a corresponding
> > > 
> > >  https://bugs.alpinelinux.org/issues/6132
> > 
> > The most likely explanation is that they're overflowing a heap buffer.
> > Perhaps it would be possible to pad all their malloc/realloc calls
> > with +64 or so to see if that makes the problem go away. If so that
> > gives a good starting point for tracking down the bug.
> 
> Thanks for the suggestion, indeed libtheora seems to consistently use
> ogg_malloc() which is redefinable.
> 
> Building with
> 
>  #define _ogg_malloc(x)  malloc((x)+256)
>  #define _ogg_calloc(x,y)  calloc((x)+256,(y))
>  #define _ogg_realloc(y,x) realloc((y),(x)+256)
>  #define _ogg_free    free
> 
> instead of the default
> 
>  #define _ogg_malloc  malloc
>  #define _ogg_calloc  calloc
>  #define _ogg_realloc realloc
>  #define _ogg_free    free
> 
> did not make any difference. The crash on a test file occurs in the same
> way and the resulting partial output file is as long as otherwise.
> 
> This may mean that this is not a simple overflowing but rather
> overwriting or reading distant "random" places (?) (register corruption?)
> 

can be underflow (or the way they align the pointer returned by malloc)

you can increase/decrease alignment of musl's alloc by
changing SIZE_ALIGN in src/malloc/malloc.c
(or you can try some hack in _ogg_malloc/free if you are
sure that's what they are using)

there can be some call abi issue (register clobbering,
stack alignment,..) because of the asm, but that's hard
to check.

you may try tracing malloc calls (i don't know an easy
way other than instrumenting musl, you can try python
scripting gdb, the default gdb command language is not
enough for reporting malloc args and return values).