From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/9152 Path: news.gmane.org!not-for-mail From: Ward Willats Newsgroups: gmane.linux.lib.musl.general Subject: MUSCL + MIPS + threads + debugging/stack traces Date: Mon, 18 Jan 2016 17:34:27 -0800 Message-ID: Reply-To: musl@lists.openwall.com NNTP-Posting-Host: plane.gmane.org Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3117\)) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Trace: ger.gmane.org 1453167293 29752 80.91.229.3 (19 Jan 2016 01:34:53 GMT) X-Complaints-To: usenet@ger.gmane.org NNTP-Posting-Date: Tue, 19 Jan 2016 01:34:53 +0000 (UTC) To: musl@lists.openwall.com Original-X-From: musl-return-9165-gllmg-musl=m.gmane.org@lists.openwall.com Tue Jan 19 02:34:53 2016 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by plane.gmane.org with smtp (Exim 4.69) (envelope-from ) id 1aLLC7-0001sK-N5 for gllmg-musl@m.gmane.org; Tue, 19 Jan 2016 02:34:51 +0100 Original-Received: (qmail 32162 invoked by uid 550); 19 Jan 2016 01:34:49 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Original-Received: (qmail 32093 invoked from network); 19 Jan 2016 01:34:40 -0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=wardco.com; s=wardco01; h=Mime-Version:To:Date:Message-Id:Subject: Content-Transfer-Encoding:Content-Type:From; bh=xQJ4sgeLFdtPZc5aB+zWU24OjCU+XVloWfYjFzxHHyY=; b=IyF2eALHn42PW0KRPEJ1BTOMEN K3i+XTEd+oRlHwQax9Y4aFtYXAD/uwFzcx1oX3KgYw0kyQlOn8WZ4n1QQLkC7qwAUVjs4Z3cVwY5B 39WiL4fwcovZjIXXnnapmhKGvd0ieKczCDzM1onuEDw9/J9wYAAZ5oA/z3V2ZG1uAmjg=; X-Mailer: Apple Mail (2.3117) X-Spam-Score: -2.9 X-SA-Exim-Connect-IP: 70.36.190.2 X-SA-Exim-Mail-From: musl@wardco.com X-SA-Exim-Scanned: No (on mail.wardco.com); SAEximRunCond expanded to false Xref: news.gmane.org gmane.linux.lib.musl.general:9152 Archived-At: Hello. We're working on a consumer product using an OpenWRT-based MIPS Linux = platform and linking our C++11 std::thread (really pthread) app to MUSL. When we try to get a backtrace with the toolchain gdb/gdbserver (or the = latest one we cross compiled ourselves from source) we get these kind of = pathetic, truncated stack traces.=20 Here, for example, is a sample: (gdb) thread apply all backtrace Thread 4 (Thread 8009.9567): #0 0x77fcb904 in __cp_end () from = /Volumes/OpenWrt/fsp-openwrt/staging_dir/target-mipsel_24kec+dsp_musl-1.1.= 11/root-UberFSP/lib/ld-musl-mipsel-sf.so.1 #1 0x77fc6f38 in __syscall_cp_c () from = /Volumes/OpenWrt/fsp-openwrt/staging_dir/target-mipsel_24kec+dsp_musl-1.1.= 11/root-UberFSP/lib/ld-musl-mipsel-sf.so.1 #2 0x00000000 in ?? () Backtrace stopped: frame did not save the PC Thread 3 (Thread 8009.9566): #0 0x77fcb904 in __cp_end () from = /Volumes/OpenWrt/fsp-openwrt/staging_dir/target-mipsel_24kec+dsp_musl-1.1.= 11/root-UberFSP/lib/ld-musl-mipsel-sf.so.1 #1 0x77fc6f38 in __syscall_cp_c () from = /Volumes/OpenWrt/fsp-openwrt/staging_dir/target-mipsel_24kec+dsp_musl-1.1.= 11/root-UberFSP/lib/ld-musl-mipsel-sf.so.1 #2 0x00000000 in ?? () Backtrace stopped: frame did not save the PC Thread 2 (Thread 8009.9565): #0 0x77fcb904 in __cp_end () from = /Volumes/OpenWrt/fsp-openwrt/staging_dir/target-mipsel_24kec+dsp_musl-1.1.= 11/root-UberFSP/lib/ld-musl-mipsel-sf.so.1 #1 0x77fc6f38 in __syscall_cp_c () from = /Volumes/OpenWrt/fsp-openwrt/staging_dir/target-mipsel_24kec+dsp_musl-1.1.= 11/root-UberFSP/lib/ld-musl-mipsel-sf.so.1 warning: GDB can't find the start of the function at 0x77b8da46. GDB is unable to find the start of the function at 0x77b8da46 and thus can't determine the size of that function's stack frame. This means that GDB may be unable to access that stack frame, or the frames below it. This problem is most likely caused by an invalid program counter or stack pointer. However, if you think GDB should simply search farther back from 0x77b8da46 for code which looks like the beginning of a function, you can increase the range of the search using the `set heuristic-fence-post' command. #2 0x77b8da48 in ?? () Thread 1 (Thread 8009.8009): #0 0x005c4be0 in sqlite3_open_v2@plt () #1 0x0055606c in SqliteDatabase::open (this=3D0x9d69c0) at = /Users/wardw/fp/fsp/fsp/src/portable/FPActivityStore/DB/SqliteDatabase.cpp= :183 #2 0x0053e658 in Nova::ValueStore::ValueStoreDatabase::open = (this=3D0x9d69c0) at /Users/wardw/fp/fsp/fsp/src/util/ValueStore.cpp:146 #3 0x0053e494 in Nova::ValueStore::VolatileDBOpener::openOrCreateDb = (this=3D0x7ffc16d8, strName=3D..., strThreadName=3D...) at = /Users/wardw/fp/fsp/fsp/src/util/ValueStore.cpp:114 ... etc ... this is the tread with the breakpoint and the trace was good = all the way to main() The other threads (4,3,2) are blocked in select() calls and what-not, = but you'd never know it from this. We also can't decode core dumps, and = attempts to do stack traces in code with libbacktrace have yielded mixed = to poor results. SO....today we went and linked our app to uLibc just to see what would = happen, and lo, and behold! It works great! (Note: Similar/redundant = thread info below removed and indicated by ellipses) gdb) thread apply all bt Thread 11 (Thread 1278.1506): #0 0x77bb542c in __pthread_cond_wait (cond=3D0x7f954a68, = mutex=3D0x7f954a98) at libpthread/nptl/sysdeps/pthread/pthread_cond_wait.c:152 #1 0x77cde4fc in = std::condition_variable::wait(std::unique_lock&) () from = /Volumes/uClibFSP/scripts/../staging_dir/target-mipsel_24kec+dsp_uClibc-0.= 9.33.2/root-UberFSP/usr/lib/libstdc++.so.6 #2 0x0042b440 in wait = (__p=3D..., __lock=3D..., this=3D0x7f954a68) at = /Volumes/uClibFSP/staging_dir/toolchain-mipsel_24kec+dsp_gcc-4.8-linaro_uC= libc-0.9.33.2/mipsel-openwrt-linux-uclibc/include/c++/4.8.3/condition_vari= able:93 #3 Nova::Mailbox::awaitMail (this=3Dthis@entry=3D0x7f954a40, = timeout_ms=3Dtimeout_ms@entry=3D0) at = /Users/xxxxx/Desktop/RontundaTrunk/Foundation_0.1.42-20160116/fsp/src/fram= ework/Mailbox.cpp:140 #4 0x0042cdb8 in awaitMail (timeout_ms=3D0, this=3D0x7f954a20) at = /Users/xxxxx/Desktop/RontundaTrunk/Foundation_0.1.42-20160116/fsp/src/fram= ework/Task.cpp:220 #5 Nova::Task::run (this=3D0x7f954a20) at = /Users/xxxxx/Desktop/RontundaTrunk/Foundation_0.1.42-20160116/fsp/src/fram= ework/Task.cpp:66 #6 0x0042ca34 in Nova::Task::_run (this=3D0x7f954a20) at = /Users/xxxxx/Desktop/RontundaTrunk/Foundation_0.1.42-20160116/fsp/src/fram= ework/Task.cpp:87 #7 0x77ce2f38 in execute_native_thread_routine () from = /Volumes/uClibFSP/scripts/../staging_dir/target-mipsel_24kec+dsp_uClibc-0.= 9.33.2/root-UberFSP/usr/lib/libstdc++.so.6 #8 0x77bc0ce0 in start_thread (arg=3D0x76a6b520) at = libpthread/nptl/pthread_create.c:297 #9 0x77bad154 in __thread_start () at = ./libc/sysdeps/linux/mips/clone.S:146 Backtrace stopped: frame did not save the PC ... Thread 8 (Thread 1278.1459): #0 0x77af1b0c in __syscall_select (n=3D26, readfds=3D0x76e6adec, = writefds=3D0x0, exceptfds=3D0x0, timeout=3D0x76e6ae80) at libc/sysdeps/linux/common/select.c:74 #1 0x77af1c64 in __libc_select (n=3D26, readfds=3D0x76e6adec, = writefds=3D0x0, exceptfds=3D0x0, timeout=3D0x76e6ae80) at libc/sysdeps/linux/common/select.c:85 #2 0x00443ef0 in Nova::FlanInterfaceTask::run (this=3D0x7f955eb8) at = /Users/tylerrider/Desktop/RontundaTrunk/Foundation_0.1.42-20160116/fsp/src= /tasks/FlanInterfaceTask.cpp:157 #3 0x0042ca34 in Nova::Task::_run (this=3D0x7f955eb8) at = /Users/tylerrider/Desktop/RontundaTrunk/Foundation_0.1.42-20160116/fsp/src= /framework/Task.cpp:87 #4 0x77ce2f38 in execute_native_thread_routine () from = /Volumes/uClibFSP/scripts/../staging_dir/target-mipsel_24kec+dsp_uClibc-0.= 9.33.2/root-UberFSP/usr/lib/libstdc++.so.6 #5 0x77bc0ce0 in start_thread (arg=3D0x76e6b520) at = libpthread/nptl/pthread_create.c:297 #6 0x77bad154 in __thread_start () at = ./libc/sysdeps/linux/mips/clone.S:146 Backtrace stopped: frame did not save the PC ...=20 Thread 2 (Thread 1278.1453): #0 0x77bb4790 in __pthread_cond_timedwait (cond=3D0xe852d8, = mutex=3D0xe852bc, abstime=3D0x77a6ae90) at libpthread/nptl/sysdeps/pthread/pthread_cond_timedwait.c:158 #1 0x0042e07c in __gthread_cond_timedwait (__abs_timeout=3D0x77a6ae90, = __mutex=3D, __cond=3D0xe852d8) at = /Volumes/uClibFSP/staging_dir/toolchain-mipsel_24kec+dsp_gcc-4.8-linaro_uC= libc-0.9.33.2/mipsel-openwrt-linux-uclibc/include/c++/4.8.3/mipsel-openwrt= -linux-uclibc/bits/gthr-default.h:871 #2 __wait_until_impl > > ( __atime=3D, __lock=3D..., this=3D0xe852d8) at = /Volumes/uClibFSP/staging_dir/toolchain-mipsel_24kec+dsp_gcc-4.8-linaro_uC= libc-0.9.33.2/mipsel-openwrt-linux-uclibc/include/c++/4.8.3/condition_vari= able:160 #3 wait_until > > ( __atime=3D, __lock=3D..., this=3D0xe852d8) at = /Volumes/uClibFSP/staging_dir/toolchain-mipsel_24kec+dsp_gcc-4.8-linaro_uC= libc-0.9.33.2/mipsel-openwrt-linux-uclibc/include/c++/4.8.3/condition_vari= able:113 #4 Nova::Timer::run (this=3D0xe85278) at = /Users/xxxxx/Desktop/RontundaTrunk/Foundation_0.1.42-20160116/fsp/src/fram= ework/Timer.cpp:110 #5 0x77ce2f38 in execute_native_thread_routine () from = /Volumes/uClibFSP/scripts/../staging_dir/target-mipsel_24kec+dsp_uClibc-0.= 9.33.2/root-UberFSP/usr/lib/libstdc++.so.6 #6 0x77bc0ce0 in start_thread (arg=3D0x77a6b520) at = libpthread/nptl/pthread_create.c:297 #7 0x77bad154 in __thread_start () at = ./libc/sysdeps/linux/mips/clone.S:146 Backtrace stopped: frame did not save the PC Thread 1 (Thread 1278.1278): #0 0x77aef294 in __syscall_nanosleep (req=3D0x7f920aa0, rem=3D0x0) at = libc/sysdeps/linux/common/nanosleep.c:21 #1 0x77aef3b8 in __libc_nanosleep (req=3D0x7f920aa0, rem=3D0x0) at = libc/sysdeps/linux/common/nanosleep.c:33 #2 0x77b85e38 in usleep (usec=3D100000) at libc/unistd/usleep.c:21 #3 0x004287cc in main (argc=3D, argv=3D) at = /Users/xxxxx/Desktop/RontundaTrunk/Foundation_0.1.42-20160116/fsp/src/main= .cpp:500 Now, I realize MIPS is a bit of a ghetto, and backtracing on the = architecture is a heuristic-filled nightmare, and also that backtracing = is not part of the "core" features you provide since "most people just = use GDB" -- but gee-whiz, it sure doesn't work on MIPS and we can't be = the first people to run into this. Hopefully there is something simple = and stupid we are doing wrong and you folks can set us straight. =20 We'd like to keep using MUSL since it is modern, clean, and "preferred" = -- but with customer deadlines and money on the line we can't keep = working blind like this and struggling with "debugging by printf()" -- = so we're going to have to switch to uLiibC real soon unless there is = some kind of way to make MUSL and GDB be more helpful. Thanks! -- Ward