Re: incompatibility between libtheora/mmx and musl ?

[u-uy74@aetey.se]

On Tue, Sep 13, 2016 at 04:43:04PM -0400, Rich Felker wrote:
> > Libtheora contains an assembler part for ia32 and x86_64 which has been
> > in use for many years, with at least both glibc and uclibc.
> > 
> > Now, musl-based builds of libtheora for ia32 with this code enabled
> > lead to the encoder segfaulting. No problem when using the C version.

> >  https://trac.xiph.org/ticket/2287
> > 
> > There is also a corresponding
> > 
> >  https://bugs.alpinelinux.org/issues/6132
> 
> The most likely explanation is that they're overflowing a heap buffer.
> Perhaps it would be possible to pad all their malloc/realloc calls
> with +64 or so to see if that makes the problem go away. If so that
> gives a good starting point for tracking down the bug.

Thanks for the suggestion, indeed libtheora seems to consistently use
ogg_malloc() which is redefinable.

Building with

 #define _ogg_malloc(x)  malloc((x)+256)
 #define _ogg_calloc(x,y)  calloc((x)+256,(y))
 #define _ogg_realloc(y,x) realloc((y),(x)+256)
 #define _ogg_free    free

instead of the default

 #define _ogg_malloc  malloc
 #define _ogg_calloc  calloc
 #define _ogg_realloc realloc
 #define _ogg_free    free

did not make any difference. The crash on a test file occurs in the same
way and the resulting partial output file is as long as otherwise.

This may mean that this is not a simple overflowing but rather
overwriting or reading distant "random" places (?) (register corruption?)

Rune