From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/6943 Path: news.gmane.org!not-for-mail From: Rich Felker Newsgroups: gmane.linux.lib.musl.general Subject: GNU Emacs LD_PRELOAD build hack Date: Mon, 2 Feb 2015 22:54:07 -0500 Message-ID: <20150203035407.GA14795@brightrain.aerifal.cx> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="jRHKVT23PllUwdXP" X-Trace: ger.gmane.org 1422935666 5063 80.91.229.3 (3 Feb 2015 03:54:26 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Tue, 3 Feb 2015 03:54:26 +0000 (UTC) To: musl@lists.openwall.com Original-X-From: musl-return-6956-gllmg-musl=m.gmane.org@lists.openwall.com Tue Feb 03 04:54:25 2015 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by plane.gmane.org with smtp (Exim 4.69) (envelope-from ) id 1YIUZD-00015M-Kp for gllmg-musl@m.gmane.org; Tue, 03 Feb 2015 04:54:23 +0100 Original-Received: (qmail 23728 invoked by uid 550); 3 Feb 2015 03:54:21 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: Original-Received: (qmail 23711 invoked from network); 3 Feb 2015 03:54:20 -0000 Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) Original-Sender: Rich Felker Xref: news.gmane.org gmane.linux.lib.musl.general:6943 Archived-At: --jRHKVT23PllUwdXP Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Background: GNU Emacs' build process depends on the ability of the build-stage binary (temacs) to "dump" itself to a new executable file containing preloaded lisp objects/state in its .data segment. This process is highly non-portable even in principle; in practice, the big issue is where malloc allocations end up. They need to all be contiguous just above the .data/.bss in the original binary so that they can become part of the .data mapping. Against musl's malloc, this has two major ways it can fail: 1. musl uses mmap for large allocations (roughly, > 128-256k) and has no mechanism for obtaining such large objects from the main brk-based heap or even requesting such (whereas glibc has mallopt and/or an environment variable to control the mmap threshold, and emacs cheats and uses that to control glibc). 2. musl reclaims the gaps around the edges of writable mappings in the main program and shared libraries and uses them for malloc. If these are in shared libraries, they won't be dumped at all, and if they're in the main program, they actually overlap with .text on disk (the same page is mapped twice; this is the cause of the gaps) and thus the .text, not the heap data, gets written out to disk by the dumper. Emacs provides its own malloc replacement and tries to use it by default, but this has to be disabled with musl, since replacing malloc in dynamic programs doesn't work (and static binaries don't work right at all with emacs' dumper because libc state would get included in the dump -- state which is "intentionally lost" when it resides in a shared library whose state isn't dumped). The right solution: As I discussed on the emacs-devel list nearly a year ago, the right solution is to get rid of the non-portable code in emacs, dumping the lisp heap and its data (rather than the whole program) to a file and either mmapping it at runtime (and possibly relocating pointers in it, if the new location it's loaded at differs) or converting it to a C source file that's compiled and linked and for which the (static or dynamic) linker can perform relocations at link/load time. This solution also solves a number of other serious issues related to the dumper, including its incompatibility with PIE binaries. Unfortunately, the right solution requires a significant overhaul by someone with expertise in emacs internals, and it's not practical in the short term. Meanwhile, we have users wanting emacs on musl-based distros (myself included). So, here's an alternate solution. The hack: The basic trick is that we need to satisfy emacs assumptions about malloc, but only at build (dumping) time, not permanently. My first thought was to build emacs in the presence of a modified musl libc.so whose malloc never uses mmap (issue 1) and never reclaims gaps at the edge of writable mappings (issue 2), but then I realized we could achieve the same thing without having to build a custom libc.so at package-build time by exploiting LD_PRELOAD. The attached file is my current draft of the LD_PRELOAD module to be loaded when running temacs to dump. In short, what it does is: - Throws away (wastes/leaks) and retries whenever it gets a result from malloc that's not between the initial value of the brk and the current (after malloc) value of the brk, i.e. anything not on the "main heap" that's contiguous with .bss/.data. - For large allocations that would be serviced by mmap, and for which musl's malloc won't/can't allocate from the "main heap", allocate 64k at a time, many times, from the heap, and exploit knowledge of the malloc chunk header/footer structures to paste them together to make one large chunk. (If the wrapper can't get contiguous chunks for this, then malloc will just fail and report failure.) The first part of the hack is simple and clean. The second part is hideously ugly, but the key point to realize is that it's only making an assumption about the library implementation used at build time, not when the emacs binary is later run. The dumped emacs does not include any code from the LD_PRELOAD hack and it does not depend on the assumptions made in the hack still being valid for the libc.so that's used at runtime. If these assumptions do become invalidated (unlikely, but possible), then all that's needed to get emacs building again is updating the hack (or just building with an outdated libc.so). With any luck, the non-portable dumping in emacs will be fixed long before this is needed, anyway. Over the next few days I hope to be working with people doing Alpine Linux (and/or other dists) packaging to get this turned into a clean, reproducible build procedure for GNU-Emacs-on-musl. In the mean time, the source for the hack is attached in case anyone wants to start hacking on it. Rich --jRHKVT23PllUwdXP Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="preload.c" #include #include #include #include #include #include static void *(*real_malloc)(size_t); static void *initial_brk; static pthread_once_t once_control[1]; static void once_func() { real_malloc = dlsym(RTLD_NEXT, "malloc"); initial_brk = sbrk(0); } static int cmp(const void *a, const void *b) { void *aa = *(void **)a, *bb = *(void **)b; return aa < bb ? -1 : aa > bb ? 1 : 0; } void *malloc(size_t n) { size_t i, j, k; pthread_once(once_control, once_func); if (n < 100000 || n > (size_t)-1/2) { void *p; do p = real_malloc(n); while (p > sbrk(0) || (p && p < initial_brk)); return p; } size_t cnt = n/16384; void **list = real_malloc(sizeof *list * cnt); if (!list) return 0; for (i=0; i 65536) { base = j+1; continue; } if (z-p < n+64) { continue; } for (k=0; k