* NULL deref SEGV in malloc.c:unbin() @ 2013-12-27 18:35 David Wuertele 2013-12-27 19:05 ` Rich Felker 0 siblings, 1 reply; 12+ messages in thread From: David Wuertele @ 2013-12-27 18:35 UTC (permalink / raw) To: musl I wonder if anyone has hit this before? In unbin(), c->next->prev is set, but c->next is NULL. It happens repeatedly, and here's what gdb says: (gdb) b fopen Breakpoint 9 at 0x90f78: file src/stdio/fopen.c, line 13. (gdb) c Continuing. Breakpoint 9, fopen (filename=0xaabe4 "/etc/hosts", mode=0xaabf0 "r") at src/stdio/fopen.c:13 13 src/stdio/fopen.c: No such file or directory. in src/stdio/fopen.c (gdb) b unbin Breakpoint 10 at 0x8bc44: file src/malloc/malloc.c, line 239. (gdb) c Continuing. Breakpoint 10, unbin (c=0x21408b8, i=40) at src/malloc/malloc.c:239 239 src/malloc/malloc.c: No such file or directory. in src/malloc/malloc.c (gdb) print *c $6 = {psize = 2096, csize = 2097, next = 0x2140088, prev = 0x0} (gdb) s 241 in src/malloc/malloc.c (gdb) Program received signal SIGSEGV, Segmentation fault. 0x0008bcc0 in unbin (c=0x21408b8, i=40) at src/malloc/malloc.c:241 241 in src/malloc/malloc.c (gdb) The root cause was not obvious on scanning the source. Is this perhaps something that's already been fixed? Dave ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: NULL deref SEGV in malloc.c:unbin() 2013-12-27 18:35 NULL deref SEGV in malloc.c:unbin() David Wuertele @ 2013-12-27 19:05 ` Rich Felker 2013-12-27 19:44 ` David Wuertele 0 siblings, 1 reply; 12+ messages in thread From: Rich Felker @ 2013-12-27 19:05 UTC (permalink / raw) To: musl On Fri, Dec 27, 2013 at 06:35:00PM +0000, David Wuertele wrote: > I wonder if anyone has hit this before? In unbin(), c->next->prev is set, but > c->next is NULL. It happens repeatedly, and here's what gdb says: It's almost surely a case of memory corruption by the calling program, most likely using memory after it's already been freed. Rich ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: NULL deref SEGV in malloc.c:unbin() 2013-12-27 19:05 ` Rich Felker @ 2013-12-27 19:44 ` David Wuertele 2013-12-27 22:13 ` Rich Felker 0 siblings, 1 reply; 12+ messages in thread From: David Wuertele @ 2013-12-27 19:44 UTC (permalink / raw) To: musl Rich Felker <dalias <at> aerifal.cx> writes: > On Fri, Dec 27, 2013 at 06:35:00PM +0000, David Wuertele wrote: > > I wonder if anyone has hit this before? In unbin(), c->next->prev is set, > > but c->next is NULL. It happens repeatedly, and here's what gdb says: > > It's almost surely a case of memory corruption by the calling program, > most likely using memory after it's already been freed. Hmm, my program calls malloc() once and never calls free(). Oh, I guess it does call free indirectly when it uses closedir() and fclose(). I will try to use gdb/watch to catch someone red-handed. Dave ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Re: NULL deref SEGV in malloc.c:unbin() 2013-12-27 19:44 ` David Wuertele @ 2013-12-27 22:13 ` Rich Felker 2013-12-28 0:25 ` David Wuertele 0 siblings, 1 reply; 12+ messages in thread From: Rich Felker @ 2013-12-27 22:13 UTC (permalink / raw) To: musl On Fri, Dec 27, 2013 at 07:44:23PM +0000, David Wuertele wrote: > Rich Felker <dalias <at> aerifal.cx> writes: > > On Fri, Dec 27, 2013 at 06:35:00PM +0000, David Wuertele wrote: > > > I wonder if anyone has hit this before? In unbin(), c->next->prev is set, > > > but c->next is NULL. It happens repeatedly, and here's what gdb says: > > > > It's almost surely a case of memory corruption by the calling program, > > most likely using memory after it's already been freed. > > Hmm, my program calls malloc() once and never calls free(). And this crash happens on the very first call to malloc? Or did you mean it only called it once successfully? > Oh, I guess it does call free indirectly when it uses closedir() and fclose(). > I will try to use gdb/watch to catch someone red-handed. It's also possible you write past the end of the buffer obtained by malloc. Rich ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: NULL deref SEGV in malloc.c:unbin() 2013-12-27 22:13 ` Rich Felker @ 2013-12-28 0:25 ` David Wuertele 2013-12-28 1:28 ` David Wuertele 0 siblings, 1 reply; 12+ messages in thread From: David Wuertele @ 2013-12-28 0:25 UTC (permalink / raw) To: musl Rich Felker <dalias <at> aerifal.cx> writes: > > On Fri, Dec 27, 2013 at 07:44:23PM +0000, David Wuertele wrote: > > Rich Felker <dalias <at> aerifal.cx> writes: > > > On Fri, Dec 27, 2013 at 06:35:00PM +0000, David Wuertele wrote: > > > > I wonder if anyone has hit this before? In unbin(), > > > > c->next->prev is set, but c->next is NULL. It happens > > > > repeatedly, and here's what gdb says: > > > > > > > > > > It's almost surely a case of memory corruption by the calling > > > program, most likely using memory after it's already been > > > freed. > > > > Hmm, my program calls malloc() once and never calls free(). > > And this crash happens on the very first call to malloc? Or did you > mean it only called it once successfully? > I only call malloc directly once, and it succeeds. I use the allocation only for a circular buffer. I have an extremely high confidence that the circular buffer does not write outside of its memory allocation. I see that malloc is called many times when I run opendir()/closedir() and other musl library functions. It is during one of the closedir() that I see the SEGV. The SEGV happens when mal.bins[40].head.next is dereferenced but mal.bins[40].head.next is NULL. So I am running my program under gdb while watching mal.bins[40].head. I see that nobody except for the musl lib writes to mal.bins[40], but I have not seen any of those writes result in a NULL value in *->next. Now I am watching all memory locations that mal.bins[40].head points to in its history, but the going is very slow. Dave ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: NULL deref SEGV in malloc.c:unbin() 2013-12-28 0:25 ` David Wuertele @ 2013-12-28 1:28 ` David Wuertele 2013-12-28 3:03 ` Rich Felker 2013-12-29 0:01 ` Szabolcs Nagy 0 siblings, 2 replies; 12+ messages in thread From: David Wuertele @ 2013-12-28 1:28 UTC (permalink / raw) To: musl I wrote: > Rich Felker <dalias <at> aerifal.cx> writes: > > On Fri, Dec 27, 2013 at 07:44:23PM +0000, David Wuertele wrote: > > > Rich Felker <dalias <at> aerifal.cx> writes: > > > > On Fri, Dec 27, 2013 at 06:35:00PM +0000, David Wuertele wrote: > > > > > I wonder if anyone has hit this before? In unbin(), > > > > > c->next->prev is set, but c->next is NULL. It happens > > > > > repeatedly, and here's what gdb says: > > > > > > > > > > > > > It's almost surely a case of memory corruption by the calling > > > > program, most likely using memory after it's already been > > > > freed. > > > > > > Hmm, my program calls malloc() once and never calls free(). > > > > And this crash happens on the very first call to malloc? Or did you > > mean it only called it once successfully? > > > > Now I am watching all memory locations that mal.bins[40].head points > to in its history, but the going is very slow. I removed all my calls to malloc(), now it is only musl that is calling it. I'm watching everything that I think might be related. It looks like calloc(), opendir(), or free() are the culprit. (gdb) info watch Num Type Disp Enb Address What 2 watchpoint keep y mal.bins[40].head breakpoint already hit 2 times print *mal.bins[40].head bt cont 3 watchpoint keep y mal.bins[40].head.next breakpoint already hit 47 times print *mal.bins[40].head bt cont 3.1 y mal.bins[40].head.next 3.2 y mal.bins[40].head.next 4 watchpoint keep y mal.bins[40].head.prev breakpoint already hit 32 times print *mal.bins[40].head bt cont 4.1 y mal.bins[40].head.prev 4.2 y mal.bins[40].head.prev (gdb) c Watchpoint 4: mal.bins[40].head.prev Old value = (struct chunk *) 0xc9690 New value = (struct chunk *) 0x0 0x0008b268 in calloc (m=519, n=2080) at src/malloc/calloc.c:20 20 in src/malloc/calloc.c $78 = {psize = 2097, csize = 2097, next = 0x0, prev = 0x0} #0 0x0008b268 in calloc (m=519, n=2080) at src/malloc/calloc.c:20 #1 0x00089f30 in opendir (name=0xaa8d4 ".") at src/dirent/opendir.c:19 #2 0x0000fb50 in mkchdir (path=0xbe83f89b "") at prog.c:1845 Watchpoint 3: mal.bins[40].head.next Old value = (struct chunk *) 0x0 New value = (struct chunk *) 0x10 opendir (name=0xaa8d4 ".") at src/dirent/opendir.c:24 24 src/dirent/opendir.c: No such file or directory. in src/dirent/opendir.c $79 = {psize = 2097, csize = 2097, next = 0x10, prev = 0x0} #0 opendir (name=0xaa8d4 ".") at src/dirent/opendir.c:24 #1 0x0000fb50 in mkchdir (path=0xbe83f89b "") at prog.c:1845 Watchpoint 3: mal.bins[40].head.next Old value = (struct chunk *) 0x10 New value = (struct chunk *) 0x2139018 free (p=0x2139020) at src/malloc/malloc.c:530 530 src/malloc/malloc.c: No such file or directory. in src/malloc/malloc.c $80 = {psize = 2096, csize = 2097, next = 0x2139018, prev = 0x0} #0 free (p=0x2139020) at src/malloc/malloc.c:530 #1 0x00089e84 in closedir (dir=0x2139020) at src/dirent/closedir.c:9 #2 0x0000fc80 in mkchdir (path=0xd22bc "/var/spool") at prog.c:1873 Program received signal SIGSEGV, Segmentation fault. 0x0008be70 in unbin (c=0x2139848, i=40) at src/malloc/malloc.c:241 241 in src/malloc/malloc.c (gdb) Dave ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Re: NULL deref SEGV in malloc.c:unbin() 2013-12-28 1:28 ` David Wuertele @ 2013-12-28 3:03 ` Rich Felker 2013-12-29 0:01 ` Szabolcs Nagy 1 sibling, 0 replies; 12+ messages in thread From: Rich Felker @ 2013-12-28 3:03 UTC (permalink / raw) To: musl On Sat, Dec 28, 2013 at 01:28:42AM +0000, David Wuertele wrote: > I removed all my calls to malloc(), now it is only musl that is > calling it. I'm watching everything that I think might be related. > It looks like calloc(), opendir(), or free() are the culprit. OK, so it seems like we should look for a possible bug in the dirent.h functions. Are you aware of anything potentially unusual about the directories you're reading? Would it be possible to provide an strace log or reduce this to a minimal self-contained failing test case that we could debug? Rich ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Re: NULL deref SEGV in malloc.c:unbin() 2013-12-28 1:28 ` David Wuertele 2013-12-28 3:03 ` Rich Felker @ 2013-12-29 0:01 ` Szabolcs Nagy 2013-12-29 0:05 ` Szabolcs Nagy 2013-12-30 19:17 ` David Wuertele 1 sibling, 2 replies; 12+ messages in thread From: Szabolcs Nagy @ 2013-12-29 0:01 UTC (permalink / raw) To: musl * David Wuertele <dave+gmane@wuertele.com> [2013-12-28 01:28:42 +0000]: > I removed all my calls to malloc(), now it is only musl that is > calling it. I'm watching everything that I think might be related. > It looks like calloc(), opendir(), or free() are the culprit. > > (gdb) info watch > Num Type Disp Enb Address What > 2 watchpoint keep y mal.bins[40].head > breakpoint already hit 2 times > print *mal.bins[40].head > bt > cont > 3 watchpoint keep y mal.bins[40].head.next > breakpoint already hit 47 times > print *mal.bins[40].head > bt > cont > 3.1 y mal.bins[40].head.next > 3.2 y mal.bins[40].head.next > 4 watchpoint keep y mal.bins[40].head.prev > breakpoint already hit 32 times > print *mal.bins[40].head > bt > cont > 4.1 y mal.bins[40].head.prev > 4.2 y mal.bins[40].head.prev > (gdb) c > > Watchpoint 4: mal.bins[40].head.prev > > Old value = (struct chunk *) 0xc9690 > New value = (struct chunk *) 0x0 > 0x0008b268 in calloc (m=519, n=2080) at src/malloc/calloc.c:20 > 20 in src/malloc/calloc.c > $78 = {psize = 2097, csize = 2097, next = 0x0, prev = 0x0} > #0 0x0008b268 in calloc (m=519, n=2080) at src/malloc/calloc.c:20 > #1 0x00089f30 in opendir (name=0xaa8d4 ".") at src/dirent/opendir.c:19 > #2 0x0000fb50 in mkchdir (path=0xbe83f89b "") at prog.c:1845 if i understand the code correctly the state is already corrupted here the printed head chunk is returned from malloc (calloc zeros out next/prev) but it is still in the free list (bin[40].head) after the last free chunk is used up in a bin i think head == tail == BIN_TO_CHUNK(i) should hold (and mal.binmap & (1ULL<<i) should be 0) > Watchpoint 3: mal.bins[40].head.next > > Old value = (struct chunk *) 0x0 > New value = (struct chunk *) 0x10 > opendir (name=0xaa8d4 ".") at src/dirent/opendir.c:24 > 24 src/dirent/opendir.c: No such file or directory. > in src/dirent/opendir.c > $79 = {psize = 2097, csize = 2097, next = 0x10, prev = 0x0} > #0 opendir (name=0xaa8d4 ".") at src/dirent/opendir.c:24 > #1 0x0000fb50 in mkchdir (path=0xbe83f89b "") at prog.c:1845 this is just opendir setting dir->fd (dir == &head->next) > Watchpoint 3: mal.bins[40].head.next > > Old value = (struct chunk *) 0x10 > New value = (struct chunk *) 0x2139018 > free (p=0x2139020) at src/malloc/malloc.c:530 > 530 src/malloc/malloc.c: No such file or directory. > in src/malloc/malloc.c > $80 = {psize = 2096, csize = 2097, next = 0x2139018, prev = 0x0} > #0 free (p=0x2139020) at src/malloc/malloc.c:530 > #1 0x00089e84 in closedir (dir=0x2139020) at src/dirent/closedir.c:9 > #2 0x0000fc80 in mkchdir (path=0xd22bc "/var/spool") at prog.c:1873 a chunk is freed here (0x2139018) which we haven't seen yet but is put on the same free list that is already broken self->prev->next = self; triggers the watchpoint (where &self->prev->next == &bin[40].head) so the free-list has two chunks now: head is the broken chunk returned in opendir and the new one freed here > > Program received signal SIGSEGV, Segmentation fault. > 0x0008be70 in unbin (c=0x2139848, i=40) at src/malloc/malloc.c:241 > 241 in src/malloc/malloc.c > (gdb) > i assume 0x2139848 is the broken chunk and unbin stumbles on the 0 prev pointer so it seems the corruption starts before opendir it would be nice to see where 0x2139018 comes from and why mal.binmap and mal.bin[40] aren't managed properly ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Re: NULL deref SEGV in malloc.c:unbin() 2013-12-29 0:01 ` Szabolcs Nagy @ 2013-12-29 0:05 ` Szabolcs Nagy 2013-12-29 1:34 ` Rich Felker 2013-12-30 19:17 ` David Wuertele 1 sibling, 1 reply; 12+ messages in thread From: Szabolcs Nagy @ 2013-12-29 0:05 UTC (permalink / raw) To: musl * Szabolcs Nagy <nsz@port70.net> [2013-12-29 01:01:12 +0100]: > > so it seems the corruption starts before opendir > > it would be nice to see where 0x2139018 comes from and why > mal.binmap and mal.bin[40] aren't managed properly probably unrelated but i dont understand split = (void *)((char *)self + n); in pretrim and trim why is the n enough between the start of self and split chunks? (and not n + overhead) ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Re: NULL deref SEGV in malloc.c:unbin() 2013-12-29 0:05 ` Szabolcs Nagy @ 2013-12-29 1:34 ` Rich Felker 0 siblings, 0 replies; 12+ messages in thread From: Rich Felker @ 2013-12-29 1:34 UTC (permalink / raw) To: musl On Sun, Dec 29, 2013 at 01:05:15AM +0100, Szabolcs Nagy wrote: > * Szabolcs Nagy <nsz@port70.net> [2013-12-29 01:01:12 +0100]: > > > > so it seems the corruption starts before opendir > > > > it would be nice to see where 0x2139018 comes from and why > > mal.binmap and mal.bin[40] aren't managed properly > > probably unrelated but i dont understand > > split = (void *)((char *)self + n); > > in pretrim and trim > > why is the n enough between the start of self and split > chunks? (and not n + overhead) The first line of malloc() calls adjust_size(&n). After that, n is always in terms of total chunk size needed, not caller-usable size. Rich ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: NULL deref SEGV in malloc.c:unbin() 2013-12-29 0:01 ` Szabolcs Nagy 2013-12-29 0:05 ` Szabolcs Nagy @ 2013-12-30 19:17 ` David Wuertele 2013-12-30 21:25 ` Rich Felker 1 sibling, 1 reply; 12+ messages in thread From: David Wuertele @ 2013-12-30 19:17 UTC (permalink / raw) To: musl I found the root cause of the SEGV, I was calling closedir() on the same dir pointer twice (quite some time before the SEGV). I assume that the behavior of closedir() is undefined when used this way, so my program now makes sure not to do that. But it seems a poor implementation that a double call to closedir should result in memory corruption, and it seems a bug in malloc() that a closedir/opendir sequence can cause it to SEGV. I tried to reduce my program to just this behavior so that I could give you a test case, but the SEGV did not occur with just the opendir/closedir sequence my program calls. Dave ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: Re: NULL deref SEGV in malloc.c:unbin() 2013-12-30 19:17 ` David Wuertele @ 2013-12-30 21:25 ` Rich Felker 0 siblings, 0 replies; 12+ messages in thread From: Rich Felker @ 2013-12-30 21:25 UTC (permalink / raw) To: musl On Mon, Dec 30, 2013 at 07:17:49PM +0000, David Wuertele wrote: > I found the root cause of the SEGV, I was calling closedir() on the > same dir pointer twice (quite some time before the SEGV). > > I assume that the behavior of closedir() is undefined when used this > way, so my program now makes sure not to do that. > > But it seems a poor implementation that a double call to closedir > should result in memory corruption, and it seems a bug in malloc() > that a closedir/opendir sequence can cause it to SEGV. No, there is fundamentally no way to make double-free errors safe; once a resource has been freed, the (numerically) same identifier may be used for new resources, and there is no way in general to distinguish between valid access via the new resource and invalid access to the already-freed old resource. In multi-threaded situations especially, this makes double-free errors (of any sort, even plain file descriptors via close()) one of the most dangerous programming errors possible. These types of errors have been responsible for many real-world remote privilege elevation vulnerabilities. musl attempts to catch and intentionally crash on those double-free errors that *can* be caught (a subset of all possible ones) so it's a bit odd that you're experiencing free-list corruption rather than a direct crash. I suspect you're actually using the already-closed DIR handle in other ways before closing it again, perhaps calling readdir on it, which would access and modify the contents of this memory (which now belongs to the allocator) as if it were a DIR handle. Rich ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2013-12-30 21:25 UTC | newest] Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2013-12-27 18:35 NULL deref SEGV in malloc.c:unbin() David Wuertele 2013-12-27 19:05 ` Rich Felker 2013-12-27 19:44 ` David Wuertele 2013-12-27 22:13 ` Rich Felker 2013-12-28 0:25 ` David Wuertele 2013-12-28 1:28 ` David Wuertele 2013-12-28 3:03 ` Rich Felker 2013-12-29 0:01 ` Szabolcs Nagy 2013-12-29 0:05 ` Szabolcs Nagy 2013-12-29 1:34 ` Rich Felker 2013-12-30 19:17 ` David Wuertele 2013-12-30 21:25 ` Rich Felker
Code repositories for project(s) associated with this public inbox https://git.vuxu.org/mirror/musl/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).