Re: incompatibility between libtheora/mmx and musl ?

[Szabolcs Nagy <nsz@port70.net>]

* Rich Felker <dalias@libc.org> [2016-09-14 10:28:42 -0400]:
> On Wed, Sep 14, 2016 at 04:04:50PM +0200, u-uy74@aetey.se wrote:
> > On Wed, Sep 14, 2016 at 01:24:00PM +0200, Szabolcs Nagy wrote:
> > > there can be some call abi issue (register clobbering,
> > > stack alignment,..) because of the asm, but that's hard
> > > to check.
> > 
> > Is musl in any way special compared to glibc/uclibc in its register usage?
> 
> Not in principle; this is mandated by the ABI. But it's possible that
> their violation of ABI contracts is visible with some implementations
> but not others. For example if they're calling malloc from code that's
> using asm it's possible that they assume the floating point registers
> (or mmx state) are call-saved rather than call-clobbered. This is an
> invalid assumption that might happen to actively break on musl but not
> glibc. IIRC you need some special instructions to switch between x87
> and (original) mmx usage; perhaps they're missing this somewhere.
> 

this might be an issue:
musl uses float instructions in malloc,
if mmx needs different fpu state and
they don't change it back for a malloc
call that can corrupt the heap.

to test this, try to use the 'non-float bin index'
code in musl from here:
http://port70.net/~nsz/musl/malloc.c