From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/6171 Path: news.gmane.org!not-for-mail From: Scott Valentine Newsgroups: gmane.linux.lib.musl.general Subject: RE: LUA + musl, garbage collection issue? Date: Sat, 20 Sep 2014 23:58:49 -1000 Message-ID: References: ,<20140921043831.GF23797@brightrain.aerifal.cx> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="_62f4b58d-6480-4ec9-906a-12ae0fde0e32_" X-Trace: ger.gmane.org 1411293562 15863 80.91.229.3 (21 Sep 2014 09:59:22 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Sun, 21 Sep 2014 09:59:22 +0000 (UTC) To: "musl@lists.openwall.com" Original-X-From: musl-return-6184-gllmg-musl=m.gmane.org@lists.openwall.com Sun Sep 21 11:59:17 2014 Return-path: Envelope-to: gllmg-musl@plane.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by plane.gmane.org with smtp (Exim 4.69) (envelope-from ) id 1XVdvJ-0005ka-54 for gllmg-musl@plane.gmane.org; Sun, 21 Sep 2014 11:59:17 +0200 Original-Received: (qmail 30217 invoked by uid 550); 21 Sep 2014 09:59:03 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: Original-Received: (qmail 30209 invoked from network); 21 Sep 2014 09:59:02 -0000 X-TMN: [B7JaWOpteZH97mVz0zBI1PXXtziORO9I] X-Originating-Email: [scottvalen@hotmail.com] Importance: Normal In-Reply-To: <20140921043831.GF23797@brightrain.aerifal.cx> X-OriginalArrivalTime: 21 Sep 2014 09:58:50.0029 (UTC) FILETIME=[A48E55D0:01CFD582] Xref: news.gmane.org gmane.linux.lib.musl.general:6171 Archived-At: --_62f4b58d-6480-4ec9-906a-12ae0fde0e32_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable > Date: Sun=2C 21 Sep 2014 00:38:31 -0400 > From: dalias@libc.org > To: musl@lists.openwall.com > Subject: Re: [musl] LUA + musl=2C garbage collection issue? >=20 > On Sat=2C Sep 20=2C 2014 at 04:41:14PM -1000=2C Scott Valentine wrote: > > I noticed that in order to free memory=2C it basically calls realloc > > with 0 as the new size. Is this something musl doesn't handle well? > >=20 > > I'm trying a rebuild with a check for n =3D=3D 0 in musl's realloc > > function to just free the pointer=2C and I'll report back. > >=20 > > What is "the right thing to do" to fix this? Should lua not be using > > realloc to free memory=2C or should musl handle the case better=2C if= =2C > > in fact this is the problem? >=20 > This is a bug in lua=3B it's depending on a bug in glibc. POSIX attempts > to allow the glibc behavior (see the text here: > http://pubs.opengroup.org/onlinepubs/9699919799/functions/realloc.html) > but this allowance is invalid since it conflicts with the requirements > of ISO C (and=2C as you can see in the above link=2C "The functionality > described on this reference page is aligned with the ISO C standard. > Any conflict between the requirements described here and the ISO C > standard is unintentional. This volume of POSIX.1-2008 defers to the > ISO C standard.") The relevant ISO C requirement is: >=20 > "The realloc function returns a pointer to the new object (which may > have the same value as a pointer to the old object)=2C or a null pointer > if the new object could not be allocated." >=20 > In particular=2C the only way realloc is permitted to return 0 is if the > operation failed=2C in which case the old pointer is still valid (not > freed). >=20 > But even if the glibc behavior weren't conflicting with the C > standard=2C it's still not valid for an application to assume it=2C and > it's still undesirable behavior (because it's inconsistent with > glibc's malloc=2C which returns a non-null=2C unique pointer for each > malloc(0)=2C and because it makes it impossible for applications to > reliably determine whether the operation succeeded or failed). >=20 > The whole situation is a big enough mess that I feel I can safely say > that any application that ever passes 0 as the size argument to > realloc has potentially-serious bugs. So if lua wants to free memory=2C > it just needs to call free. >=20 > Rich Interestingly=2C I added the explicit free and return 0 at the top of musl'= s realloc function=2C but I am still seeing the same behavior=2C so now I a= m scratching my head. I've eliminated most pieces of the stack as sources o= f the problem. I can use scp to copy the file without issues=2C uhttpd pret= ty much never mallocs anything=2C I turned off ssl=2C and luci is almost en= tire written in lua. It does use the nixio library to actually write the fi= le=2C but that doesn't really malloc anything and is really just a lightwei= ght lua to native wrapper. I'm going to try valgrind on the mips target to see who is causing trouble. Could this possibly have something to do with the read/write stdio function= s in musl? I can't really see how=2C but at present that is the only other = implementation specific feature I can find that is relevant. Well=2C that a= nd musl uses BUFSIZ =3D 1024 and glibc uses BUFSIZ =3D 8192. LUA references= this variable directly=2C but it is really only a default. Thanks=2C-Scott V. = --_62f4b58d-6480-4ec9-906a-12ae0fde0e32_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable


>=3B Date: Sun=2C= 21 Sep 2014 00:38:31 -0400
>=3B From: dalias@libc.org
>=3B To: m= usl@lists.openwall.com
>=3B Subject: Re: [musl] LUA + musl=2C garbage = collection issue?
>=3B
>=3B On Sat=2C Sep 20=2C 2014 at 04:41:14= PM -1000=2C Scott Valentine wrote:
>=3B >=3B I noticed that in order= to free memory=2C it basically calls realloc
>=3B >=3B with 0 as th= e new size. Is this something musl doesn't handle well?
>=3B >=3B >=3B >=3B I'm trying a rebuild with a check for n =3D=3D 0 in musl's = realloc
>=3B >=3B function to just free the pointer=2C and I'll repo= rt back.
>=3B >=3B
>=3B >=3B What is "the right thing to do"= to fix this? Should lua not be using
>=3B >=3B realloc to free memo= ry=2C or should musl handle the case better=2C if=2C
>=3B >=3B in fa= ct this is the problem?
>=3B
>=3B This is a bug in lua=3B it's d= epending on a bug in glibc. POSIX attempts
>=3B to allow the glibc beh= avior (see the text here:
>=3B http://pubs.opengroup.org/onlinepubs/96= 99919799/functions/realloc.html)
>=3B but this allowance is invalid si= nce it conflicts with the requirements
>=3B of ISO C (and=2C as you ca= n see in the above link=2C "The functionality
>=3B described on this r= eference page is aligned with the ISO C standard.
>=3B Any conflict be= tween the requirements described here and the ISO C
>=3B standard is u= nintentional. This volume of POSIX.1-2008 defers to the
>=3B ISO C sta= ndard.") The relevant ISO C requirement is:
>=3B
>=3B "The reall= oc function returns a pointer to the new object (which may
>=3B have t= he same value as a pointer to the old object)=2C or a null pointer
>= =3B if the new object could not be allocated."
>=3B
>=3B In part= icular=2C the only way realloc is permitted to return 0 is if the
>=3B= operation failed=2C in which case the old pointer is still valid (not
&= gt=3B freed).
>=3B
>=3B But even if the glibc behavior weren't c= onflicting with the C
>=3B standard=2C it's still not valid for an app= lication to assume it=2C and
>=3B it's still undesirable behavior (bec= ause it's inconsistent with
>=3B glibc's malloc=2C which returns a non= -null=2C unique pointer for each
>=3B malloc(0)=2C and because it make= s it impossible for applications to
>=3B reliably determine whether th= e operation succeeded or failed).
>=3B
>=3B The whole situation = is a big enough mess that I feel I can safely say
>=3B that any applic= ation that ever passes 0 as the size argument to
>=3B realloc has pote= ntially-serious bugs. So if lua wants to free memory=2C
>=3B it just n= eeds to call free.
>=3B
>=3B Rich

Inter= estingly=2C I added the explicit free and return 0 at the top of musl's rea= lloc function=2C but I am still seeing the same behavior=2C so now I am scr= atching my head. I've eliminated most pieces of the stack as sources of the= problem. I can use scp to copy the file without issues=2C uhttpd pretty mu= ch never mallocs anything=2C I turned off ssl=2C and luci is almost entire = written in lua. It does use the nixio library to actually write the file=2C= but that doesn't really malloc anything and is really just a lightweight l= ua to native wrapper.

I'm going to try valgrind on= the mips target to see who is causing trouble.

Co= uld this possibly have something to do with the read/write stdio functions = in musl? I can't really see how=2C but at present that is the only other im= plementation specific feature I can find that is relevant. Well=2C that and= musl uses BUFSIZ =3D 1024 and glibc uses BUFSIZ =3D 8192. LUA references t= his variable directly=2C but it is really only a default.

Thanks=2C
-Scott V.
= --_62f4b58d-6480-4ec9-906a-12ae0fde0e32_--