Re: incompatibility between libtheora/mmx and musl ?

[u-uy74@aetey.se]

On Wed, Sep 14, 2016 at 01:24:00PM +0200, Szabolcs Nagy wrote:
> >  #define _ogg_malloc(x)  malloc((x)+256)
> >  #define _ogg_calloc(x,y)  calloc((x)+256,(y))
> >  #define _ogg_realloc(y,x) realloc((y),(x)+256)
> >  #define _ogg_free    free
> > 
> > instead of the default
> > 
> >  #define _ogg_malloc  malloc
> >  #define _ogg_calloc  calloc
> >  #define _ogg_realloc realloc
> >  #define _ogg_free    free
> > 
> > did not make any difference. The crash on a test file occurs in the same
> > way and the resulting partial output file is as long as otherwise.
> > 
> > This may mean that this is not a simple overflowing but rather
> > overwriting or reading distant "random" places (?) (register corruption?)

> can be underflow (or the way they align the pointer returned by malloc)

Pointer alignment yes they do in some cases but in a different layer,
inside the malloc()-ed buffers, it is plain C and looks harmless to me.

> you can increase/decrease alignment of musl's alloc by
> changing SIZE_ALIGN in src/malloc/malloc.c

Doubling the alignment did not apparently change the crashing.

Reducing the alignment in half did not apparently change the crashing.

(A single test file with a single quality setting tested
crashed the same way, at the same place in the output stream)

> (or you can try some hack in _ogg_malloc/free if you are
> sure that's what they are using)

Yes it is present/used for this very purpose, to enable easy "hijacking".

OTOH when I checked the arguments in gdb they looked always sane, up to
the last and crashing realloc() call. That's why I do not expect seeing
anything unusual there.

Valgrind did not see any bad free()s either.

> there can be some call abi issue (register clobbering,
> stack alignment,..) because of the asm, but that's hard
> to check.

Is musl in any way special compared to glibc/uclibc in its register usage?

> you may try tracing malloc calls (i don't know an easy
> way other than instrumenting musl, you can try python
> scripting gdb, the default gdb command language is not
> enough for reporting malloc args and return values).

This is something I wished to avoid. It does not promise much either,
but I may possibly try this if nothing else helps.

Thanks everyone for the help!

Rune