Re: "non-float" malloc (was: incompatibility between libtheora/mmx and musl)

[u-uy74@aetey.se]

On Sun, Oct 02, 2016 at 12:59:50PM +0200, u-uy74@aetey.se wrote:
> Would you mind adding the brk()/stack overlap checking to this variant
> of the code?

Something like (shamelessly cut-n-pasted from expand_heap.c)

--- malloc.c.ori        2016-10-02 13:06:34.407671803 +0200
+++ malloc.c    2016-10-02 13:13:06.787149613 +0200
@@ -60,6 +60,27 @@
 
 #define IS_MMAPPED(c) !((c)->csize & (C_INUSE))
 
+/* This function returns true if the interval [old,new]
+ * intersects the 'len'-sized interval below &libc.auxv
+ * (interpreted as the main-thread stack) or below &b
+ * (the current stack). It is used to defend against
+ * buggy brk implementations that can cross the stack. */
+
+static int traverses_stack_p(uintptr_t old, uintptr_t new)
+{
+        const uintptr_t len = 8<<20;
+        uintptr_t a, b;
+
+        b = (uintptr_t)libc.auxv;
+        a = b > len ? b-len : 0;
+        if (new>a && old<b) return 1;
+
+        b = (uintptr_t)&b;
+        a = b > len ? b-len : 0;
+        if (new>a && old<b) return 1;
+
+        return 0;
+}
 
 /* Synchronization tools */
 
@@ -204,7 +225,8 @@
        new = mal.brk + n + SIZE_ALIGN + PAGE_SIZE - 1 & -PAGE_SIZE;
        n = new - mal.brk;
 
-       if (__brk(new) != new) {
+       if (traverses_stack_p(mal.brk, new) ||
+           __brk(new) != new) {
                size_t min = (size_t)PAGE_SIZE << mal.mmap_step/2;
                n += -n & PAGE_SIZE-1;
                if (n < min) n = min;

Does it look correct?

Regards,
Rune