From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/6945 Path: news.gmane.org!not-for-mail From: stephen Turner Newsgroups: gmane.linux.lib.musl.general Subject: Re: GNU Emacs LD_PRELOAD build hack Date: Tue, 3 Feb 2015 07:21:38 -0500 Message-ID: References: <20150203035407.GA14795@brightrain.aerifal.cx> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: multipart/alternative; boundary=089e013a011ad6e33e050e2e1ca6 X-Trace: ger.gmane.org 1422966119 26244 80.91.229.3 (3 Feb 2015 12:21:59 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Tue, 3 Feb 2015 12:21:59 +0000 (UTC) To: musl@lists.openwall.com Original-X-From: musl-return-6958-gllmg-musl=m.gmane.org@lists.openwall.com Tue Feb 03 13:21:58 2015 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by plane.gmane.org with smtp (Exim 4.69) (envelope-from ) id 1YIcUQ-0003z3-Af for gllmg-musl@m.gmane.org; Tue, 03 Feb 2015 13:21:58 +0100 Original-Received: (qmail 5541 invoked by uid 550); 3 Feb 2015 12:21:51 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: Original-Received: (qmail 5526 invoked from network); 3 Feb 2015 12:21:50 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=Bv15ZtihXxTwa2IuKirrmz9NYp6h9iKCu2ftbyEFvf4=; b=l0MWE+lbOyr+7373m7C8e1/t6q9/urqteha1HKmUyUkmhlh1rFhm+4qDaPLtUTUdQd JDckOST0m+lo1q7geC/iYhIUmUE9Y912DoOeny2g26DNJ2SZYROZBSPOTDTNus41RAff bxKMeHTzs8KDtO3IS9CqUsbRYqg/6s8hphNoBZD2lXsRZ+yRwTt5uoTx5oRUW1mfEU/b dLQEOWQFcVTgru2kqQOR3+rne3Cy73OVlsxbyV1kLsbi3K/KVJNkrdVhrohzj77Xmto1 uQixc2xXZjkWKjJwkfXeZoUfLswXj1HHc3GKateG6l4+MIzb5eiQBSZHTq/KoJCX858A 2Fng== X-Received: by 10.236.23.132 with SMTP id v4mr9826418yhv.160.1422966098681; Tue, 03 Feb 2015 04:21:38 -0800 (PST) In-Reply-To: <20150203035407.GA14795@brightrain.aerifal.cx> Xref: news.gmane.org gmane.linux.lib.musl.general:6945 Archived-At: --089e013a011ad6e33e050e2e1ca6 Content-Type: text/plain; charset=UTF-8 On Feb 2, 2015 10:54 PM, "Rich Felker" wrote: > > Background: GNU Emacs' build process depends on the ability of the > build-stage binary (temacs) to "dump" itself to a new executable file > containing preloaded lisp objects/state in its .data segment. This > process is highly non-portable even in principle; in practice, the big > issue is where malloc allocations end up. They need to all be > contiguous just above the .data/.bss in the original binary so that > they can become part of the .data mapping. Against musl's malloc, this > has two major ways it can fail: > > 1. musl uses mmap for large allocations (roughly, > 128-256k) and has > no mechanism for obtaining such large objects from the main > brk-based heap or even requesting such (whereas glibc has mallopt > and/or an environment variable to control the mmap threshold, and > emacs cheats and uses that to control glibc). > > 2. musl reclaims the gaps around the edges of writable mappings in the > main program and shared libraries and uses them for malloc. If > these are in shared libraries, they won't be dumped at all, and if > they're in the main program, they actually overlap with .text on > disk (the same page is mapped twice; this is the cause of the gaps) > and thus the .text, not the heap data, gets written out to disk by > the dumper. > > Emacs provides its own malloc replacement and tries to use it by > default, but this has to be disabled with musl, since replacing malloc > in dynamic programs doesn't work (and static binaries don't work right > at all with emacs' dumper because libc state would get included in the > dump -- state which is "intentionally lost" when it resides in a > shared library whose state isn't dumped). > > The right solution: As I discussed on the emacs-devel list nearly a > year ago, the right solution is to get rid of the non-portable code in > emacs, dumping the lisp heap and its data (rather than the whole > program) to a file and either mmapping it at runtime (and possibly > relocating pointers in it, if the new location it's loaded at differs) > or converting it to a C source file that's compiled and linked and for > which the (static or dynamic) linker can perform relocations at > link/load time. This solution also solves a number of other serious > issues related to the dumper, including its incompatibility with PIE > binaries. > > Unfortunately, the right solution requires a significant overhaul by > someone with expertise in emacs internals, and it's not practical in > the short term. Meanwhile, we have users wanting emacs on musl-based > distros (myself included). > > So, here's an alternate solution. I think there was a guy on the toybox list working on editors and might have a emacs for you. Onefang comes to mind. --089e013a011ad6e33e050e2e1ca6 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable


On Feb 2, 2015 10:54 PM, "Rich Felker" <dalias@libc.org> wrote:
>
> Background: GNU Emacs' build process depends on the ability of the=
> build-stage binary (temacs) to "dump" itself to a new execut= able file
> containing preloaded lisp objects/state in its .data segment. This
> process is highly non-portable even in principle; in practice, the big=
> issue is where malloc allocations end up. They need to all be
> contiguous just above the .data/.bss in the original binary so that > they can become part of the .data mapping. Against musl's malloc, = this
> has two major ways it can fail:
>
> 1. musl uses mmap for large allocations (roughly, > 128-256k) and h= as
> =C2=A0 =C2=A0no mechanism for obtaining such large objects from the ma= in
> =C2=A0 =C2=A0brk-based heap or even requesting such (whereas glibc has= mallopt
> =C2=A0 =C2=A0and/or an environment variable to control the mmap thresh= old, and
> =C2=A0 =C2=A0emacs cheats and uses that to control glibc).
>
> 2. musl reclaims the gaps around the edges of writable mappings in the=
> =C2=A0 =C2=A0main program and shared libraries and uses them for mallo= c. If
> =C2=A0 =C2=A0these are in shared libraries, they won't be dumped a= t all, and if
> =C2=A0 =C2=A0they're in the main program, they actually overlap wi= th .text on
> =C2=A0 =C2=A0disk (the same page is mapped twice; this is the cause of= the gaps)
> =C2=A0 =C2=A0and thus the .text, not the heap data, gets written out t= o disk by
> =C2=A0 =C2=A0the dumper.
>
> Emacs provides its own malloc replacement and tries to use it by
> default, but this has to be disabled with musl, since replacing malloc=
> in dynamic programs doesn't work (and static binaries don't wo= rk right
> at all with emacs' dumper because libc state would get included in= the
> dump -- state which is "intentionally lost" when it resides = in a
> shared library whose state isn't dumped).
>
> The right solution: As I discussed on the emacs-devel list nearly a > year ago, the right solution is to get rid of the non-portable code in=
> emacs, dumping the lisp heap and its data (rather than the whole
> program) to a file and either mmapping it at runtime (and possibly
> relocating pointers in it, if the new location it's loaded at diff= ers)
> or converting it to a C source file that's compiled and linked and= for
> which the (static or dynamic) linker can perform relocations at
> link/load time. This solution also solves a number of other serious > issues related to the dumper, including its incompatibility with PIE > binaries.
>
> Unfortunately, the right solution requires a significant overhaul by > someone with expertise in emacs internals, and it's not practical = in
> the short term. Meanwhile, we have users wanting emacs on musl-based > distros (myself included).
>
> So, here's an alternate solution.

I think there was a guy on the toybox list working on editor= s and might have a emacs for you. Onefang comes to mind.

--089e013a011ad6e33e050e2e1ca6--