Re: incompatibility between libtheora/mmx and musl ?

[Rich Felker <dalias@libc.org>]

On Tue, Sep 13, 2016 at 08:06:49PM +0200, u-uy74@aetey.se wrote:
> Hello,
> 
> Libtheora contains an assembler part for ia32 and x86_64 which has been
> in use for many years, with at least both glibc and uclibc.
> 
> Now, musl-based builds of libtheora for ia32 with this code enabled
> lead to the encoder segfaulting. No problem when using the C version.
> 
> This is seen when building at Aetey. Ffmpeg in Alpine exhibits the same
> problem, too (strictly speaking, possibly another problem but ffmpeg
> crashes there with the same pattern, when encoding to theora).
> 
> People at Xiph do not have any musl-based system and want really detailed
> debugging info which implies tracing through musl and following malloc's
> internal data (which is what seems to become corrupted).
> 
> In other words, the detailed knowledge (xiph and musl) is located
> separately from each other and also from the motivation (aetey, alpine
> and possibly others).
> 
> I would appreciate some help and suggestions. The alternative is diving
> into mmx-assembler and musl code or resorting to C-only routines which
> give roughly half of the performance of the assembler ones.
> 
> More details are there at
> 
>  https://trac.xiph.org/ticket/2287
> 
> There is also a corresponding
> 
>  https://bugs.alpinelinux.org/issues/6132

The most likely explanation is that they're overflowing a heap buffer.
Perhaps it would be possible to pad all their malloc/realloc calls
with +64 or so to see if that makes the problem go away. If so that
gives a good starting point for tracking down the bug.

Rich