From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/10441 Path: news.gmane.org!.POSTED!not-for-mail From: Szabolcs Nagy Newsgroups: gmane.linux.lib.musl.general Subject: Re: incompatibility between libtheora/mmx and musl ? Date: Wed, 14 Sep 2016 13:24:00 +0200 Message-ID: <20160914112400.GS1280@port70.net> References: <20160913180649.GN16436@example.net> <20160913204304.GY15995@brightrain.aerifal.cx> <20160914103253.GO16436@example.net> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii X-Trace: blaine.gmane.org 1473852275 31575 195.159.176.226 (14 Sep 2016 11:24:35 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Wed, 14 Sep 2016 11:24:35 +0000 (UTC) User-Agent: Mutt/1.6.0 (2016-04-01) To: musl@lists.openwall.com Original-X-From: musl-return-10454-gllmg-musl=m.gmane.org@lists.openwall.com Wed Sep 14 13:24:28 2016 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by blaine.gmane.org with smtp (Exim 4.84_2) (envelope-from ) id 1bk8IX-0006Ek-Oq for gllmg-musl@m.gmane.org; Wed, 14 Sep 2016 13:24:13 +0200 Original-Received: (qmail 22004 invoked by uid 550); 14 Sep 2016 11:24:13 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Original-Received: (qmail 21978 invoked from network); 14 Sep 2016 11:24:12 -0000 Mail-Followup-To: musl@lists.openwall.com Content-Disposition: inline In-Reply-To: <20160914103253.GO16436@example.net> Xref: news.gmane.org gmane.linux.lib.musl.general:10441 Archived-At: * u-uy74@aetey.se [2016-09-14 12:32:53 +0200]: > On Tue, Sep 13, 2016 at 04:43:04PM -0400, Rich Felker wrote: > > > Libtheora contains an assembler part for ia32 and x86_64 which has been > > > in use for many years, with at least both glibc and uclibc. > > > > > > Now, musl-based builds of libtheora for ia32 with this code enabled > > > lead to the encoder segfaulting. No problem when using the C version. > > > > https://trac.xiph.org/ticket/2287 > > > > > > There is also a corresponding > > > > > > https://bugs.alpinelinux.org/issues/6132 > > > > The most likely explanation is that they're overflowing a heap buffer. > > Perhaps it would be possible to pad all their malloc/realloc calls > > with +64 or so to see if that makes the problem go away. If so that > > gives a good starting point for tracking down the bug. > > Thanks for the suggestion, indeed libtheora seems to consistently use > ogg_malloc() which is redefinable. > > Building with > > #define _ogg_malloc(x) malloc((x)+256) > #define _ogg_calloc(x,y) calloc((x)+256,(y)) > #define _ogg_realloc(y,x) realloc((y),(x)+256) > #define _ogg_free free > > instead of the default > > #define _ogg_malloc malloc > #define _ogg_calloc calloc > #define _ogg_realloc realloc > #define _ogg_free free > > did not make any difference. The crash on a test file occurs in the same > way and the resulting partial output file is as long as otherwise. > > This may mean that this is not a simple overflowing but rather > overwriting or reading distant "random" places (?) (register corruption?) > can be underflow (or the way they align the pointer returned by malloc) you can increase/decrease alignment of musl's alloc by changing SIZE_ALIGN in src/malloc/malloc.c (or you can try some hack in _ogg_malloc/free if you are sure that's what they are using) there can be some call abi issue (register clobbering, stack alignment,..) because of the asm, but that's hard to check. you may try tracing malloc calls (i don't know an easy way other than instrumenting musl, you can try python scripting gdb, the default gdb command language is not enough for reporting malloc args and return values).