* pthreads broken (freeradius testcase) @ 2015-01-15 22:52 Sebastian Gottschall 2015-01-15 23:10 ` Rich Felker 2015-01-16 7:20 ` Natanael Copa 0 siblings, 2 replies; 14+ messages in thread From: Sebastian Gottschall @ 2015-01-15 22:52 UTC (permalink / raw) To: musl following test case configure freeradius with --with-threads (which is on by default) if you start radiusd with your radius configuration you will see that radius does not listen on any ports. it will hang in the listener thread which creates the socket. if you configure it as --without-threads, it works tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10 Sebastian ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase) 2015-01-15 22:52 pthreads broken (freeradius testcase) Sebastian Gottschall @ 2015-01-15 23:10 ` Rich Felker 2015-01-15 23:42 ` Rich Felker 2015-01-16 7:20 ` Natanael Copa 1 sibling, 1 reply; 14+ messages in thread From: Rich Felker @ 2015-01-15 23:10 UTC (permalink / raw) To: musl On Thu, Jan 15, 2015 at 11:52:36PM +0100, Sebastian Gottschall wrote: > following test case > > configure freeradius with --with-threads (which is on by default) > if you start radiusd with your radius configuration you will see > that radius does not listen on any ports. it will hang in the > listener thread which creates the socket. > if you configure it as --without-threads, it works > > > tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10 Thanks for the report. I'm not terribly familiar with freeradius. Can you tell where it's hung, e.g. get a backtrace in gdb, and possibly strace -f leading up to the hang? Rich ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase) 2015-01-15 23:10 ` Rich Felker @ 2015-01-15 23:42 ` Rich Felker 2015-01-15 23:53 ` Sebastian Gottschall 2015-01-16 0:11 ` Sebastian Gottschall 0 siblings, 2 replies; 14+ messages in thread From: Rich Felker @ 2015-01-15 23:42 UTC (permalink / raw) To: musl On Thu, Jan 15, 2015 at 06:10:37PM -0500, Rich Felker wrote: > On Thu, Jan 15, 2015 at 11:52:36PM +0100, Sebastian Gottschall wrote: > > following test case > > > > configure freeradius with --with-threads (which is on by default) > > if you start radiusd with your radius configuration you will see > > that radius does not listen on any ports. it will hang in the > > listener thread which creates the socket. > > if you configure it as --without-threads, it works > > > > > > tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10 > > Thanks for the report. I'm not terribly familiar with freeradius. Can > you tell where it's hung, e.g. get a backtrace in gdb, and possibly > strace -f leading up to the hang? Short of that, a minimal procedure to reproduce the bug (build configuration, runtime config files, which distro (if any) you're using, and anything else needed in the environment, etc.) would be helpful. Rich ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase) 2015-01-15 23:42 ` Rich Felker @ 2015-01-15 23:53 ` Sebastian Gottschall 2015-01-16 0:11 ` Sebastian Gottschall 1 sibling, 0 replies; 14+ messages in thread From: Sebastian Gottschall @ 2015-01-15 23:53 UTC (permalink / raw) To: musl [-- Attachment #1: Type: text/plain, Size: 3215 bytes --] cd freeradius && \ sys_lib_dlsearch_path_spec="mips-uclibc" \ sys_lib_search_path_spec="mips-uclibc" \ MYSQL_CONFIG="no" \ ac_cv_lib_readline=no \ ac_cv_lib_ssl_SSL_new=yes \ ac_cv_lib_crypto_DH_new=yes \ ac_cv_func_strncasecmp=yes \ ac_cv_func_strcasecmp=yes \ ac_cv_lib_crypt_crypt=yes \ ac_cv_func_gettimeofday=yes \ ac_cv_func_getnameinfo=yes \ ac_cv_func_getaddrinfo=yes \ ac_cv_func_setlinebuf=yes \ ac_cv_host=mips-uclibc-linux \ ./configure --target=mips-linux --host=mips CFLAGS="-Os -pipe -mips32r2 -mtune=mips32r2 -msoft-float -fno-caller-saves -mno-branch-likely -minterlink-mips16 -mips16 -I/home/seg/DEV/pb42/src/router/openssl/include " LDFLAGS="-Os -pipe -mips32r2 -mtune=mips32r2 -msoft-float -fno-caller-saves -mno-branch-likely -minterlink-mips16 -mips16 -L/home/seg/DEV/pb42/src/router/openssl" --enable-shared \ --program-prefix="" \ --program-suffix="" \ --prefix=/usr \ --exec-prefix=/usr \ --bindir=/usr/bin \ --datadir=/usr/share \ --includedir=/usr/include \ --infodir=/usr/share/info \ --libdir=/usr/lib \ --libexecdir=/usr/lib \ --with-raddbdir=/etc/freeradius2 \ --with-radacctdir=/var/db/radacct \ --with-logdir=/var/log \ --localstatedir=/var \ --mandir=/usr/share/man \ --sbindir=/usr/sbin \ --sysconfdir=/etc \ --enable-shared \ --disable-static \ --disable-developer \ --with-threads \ --with-ltdl-include="/home/seg/DEV/pb42/src/router/freeradius/libltdl/.libs" \ --with-ltdl-lib="/home/seg/DEV/pb42/src/router/freeradius/libltdl" \ --with-openssl-includes="/home/seg/DEV/pb42/src/router/openssl/include" \ --with-openssl-libraries="/home/seg/DEV/pb42/src/router/openssl" \ --enable-strict-dependencies \ --with-raddbdir=/etc/freeradius \ --without-edir \ --without-snmp \ --with-experimental-modules \ --without-rlm_attr-rewrite \ --without-rlm_checkval \ --without-rlm_counter \ --without-rlm_dbm \ --without-rlm_ldap \ --without-edir \ --without-snmp \ --with-rlm_expr \ --with-rlm_eap \ --without-rlm_eap_sim \ --without-rlm_example \ --without-rlm_ippool \ --without-rlm_krb5 \ --without-rlm_pam \ --without-rlm_perl \ --without-rlm_python \ --without-rlm_smb \ --with-rlm_sql \ --with-rlm_sqlcounter \ --without-rlm_sql_db2 \ --without-rlm_sql_freetds \ --without-rlm_sql_iodbc \ --without-rlm_sql_oracle \ --without-rlm_sql_sybase \ --without-rlm_sql_unixodbc \ --without-rlm_x99-token \ --without-rlm_eap_ikev2 \ --without-rlm_eap_tnc \ --without-rlm_opendirectory \ --without-rlm_wimax \ --with-rlm_eap_peap \ --with-rlm_eap_tls \ --with-rlm_eap_ttls \ --with-rlm_expiration \ --with-rlm_logintime \ --with-rlm_attr-rewrite \ --without-rlm_otp \ --without-rlm_smsotp \ --without-rlm_sqlhpwippool \ --without-rlm_sqlippool \ --without-rlm_sql_db2 \ --without-rlm_sql_firebird \ --without-rlm_sql_freetds \ --without-rlm_sql_iodbc \ --without-rlm_sql_oracle \ --without-rlm_sql_sybase \ --without-rlm_sql_unixodbc \ --without-rlm_sql_log \ --without-rlm_sql_sqlite \ --without-rlm_caching \ --without-rlm_redis \ --without-rlm_rediswho \ --without-rlm_eap_tnc \ --without-rlm_eap_ikev2 \ --without-rlm_opendirectory \ --without-rlm_wimax \ --without-rlm_ruby \ --without-rlm_sql_mysql \ --without-rlm_sql_postgresql c [-- Attachment #2: freeradius.tar.xz --] [-- Type: application/octet-stream, Size: 133540 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase) 2015-01-15 23:42 ` Rich Felker 2015-01-15 23:53 ` Sebastian Gottschall @ 2015-01-16 0:11 ` Sebastian Gottschall 2015-01-16 0:23 ` Rich Felker 1 sibling, 1 reply; 14+ messages in thread From: Sebastian Gottschall @ 2015-01-16 0:11 UTC (permalink / raw) To: musl [-- Attachment #1: Type: text/plain, Size: 5197 bytes --] sorry i forgot the full content. of the message. the first attemt failed due a too big attachment i can help you more. because i found the deadlocking function. its setresuid which hangs forever. (i know that you submitted a patch recently which is related to that function, but i dont know if this will help here) debugging is more complicated with gdb here. its a embedded system without any remote debugging support its exact the same problem which is described here, but without any further solution or problem cause http://freeradius.1045715.n5.nabble.com/Hack-way-to-compile-freeradius-causing-freeradius-to-hang-under-multithread-mode-td2794760.html distribution is simply dd-wrt (router os). so maybe hard for you to work with. i can provide you the full configs used which can be directly executed using radiusd -d /directory to config the full configure line for freeradius, just taken out of my buildsystem (you likelly need to adjust some paths) cd freeradius && \ sys_lib_dlsearch_path_spec="mips-uclibc" \ sys_lib_search_path_spec="mips-uclibc" \ MYSQL_CONFIG="no" \ ac_cv_lib_readline=no \ ac_cv_lib_ssl_SSL_new=yes \ ac_cv_lib_crypto_DH_new=yes \ ac_cv_func_strncasecmp=yes \ ac_cv_func_strcasecmp=yes \ ac_cv_lib_crypt_crypt=yes \ ac_cv_func_gettimeofday=yes \ ac_cv_func_getnameinfo=yes \ ac_cv_func_getaddrinfo=yes \ ac_cv_func_setlinebuf=yes \ ac_cv_host=mips-uclibc-linux \ ./configure --target=mips-linux --host=mips CFLAGS="-Os -pipe -mips32r2 -mtune=mips32r2 -msoft-float -fno-caller-saves -mno-branch-likely -minterlink-mips16 -mips16 -I/home/seg/DEV/pb42/src/router/openssl/include " LDFLAGS="-Os -pipe -mips32r2 -mtune=mips32r2 -msoft-float -fno-caller-saves -mno-branch-likely -minterlink-mips16 -mips16 -L/home/seg/DEV/pb42/src/router/openssl" --enable-shared \ --program-prefix="" \ --program-suffix="" \ --prefix=/usr \ --exec-prefix=/usr \ --bindir=/usr/bin \ --datadir=/usr/share \ --includedir=/usr/include \ --infodir=/usr/share/info \ --libdir=/usr/lib \ --libexecdir=/usr/lib \ --with-raddbdir=/etc/freeradius2 \ --with-radacctdir=/var/db/radacct \ --with-logdir=/var/log \ --localstatedir=/var \ --mandir=/usr/share/man \ --sbindir=/usr/sbin \ --sysconfdir=/etc \ --enable-shared \ --disable-static \ --disable-developer \ --with-threads \ --with-ltdl-include="/home/seg/DEV/pb42/src/router/freeradius/libltdl/.libs" \ --with-ltdl-lib="/home/seg/DEV/pb42/src/router/freeradius/libltdl" \ --with-openssl-includes="/home/seg/DEV/pb42/src/router/openssl/include" \ --with-openssl-libraries="/home/seg/DEV/pb42/src/router/openssl" \ --enable-strict-dependencies \ --with-raddbdir=/etc/freeradius \ --without-edir \ --without-snmp \ --with-experimental-modules \ --without-rlm_attr-rewrite \ --without-rlm_checkval \ --without-rlm_counter \ --without-rlm_dbm \ --without-rlm_ldap \ --without-edir \ --without-snmp \ --with-rlm_expr \ --with-rlm_eap \ --without-rlm_eap_sim \ --without-rlm_example \ --without-rlm_ippool \ --without-rlm_krb5 \ --without-rlm_pam \ --without-rlm_perl \ --without-rlm_python \ --without-rlm_smb \ --with-rlm_sql \ --with-rlm_sqlcounter \ --without-rlm_sql_db2 \ --without-rlm_sql_freetds \ --without-rlm_sql_iodbc \ --without-rlm_sql_oracle \ --without-rlm_sql_sybase \ --without-rlm_sql_unixodbc \ --without-rlm_x99-token \ --without-rlm_eap_ikev2 \ --without-rlm_eap_tnc \ --without-rlm_opendirectory \ --without-rlm_wimax \ --with-rlm_eap_peap \ --with-rlm_eap_tls \ --with-rlm_eap_ttls \ --with-rlm_expiration \ --with-rlm_logintime \ --with-rlm_attr-rewrite \ --without-rlm_otp \ --without-rlm_smsotp \ --without-rlm_sqlhpwippool \ --without-rlm_sqlippool \ --without-rlm_sql_db2 \ --without-rlm_sql_firebird \ --without-rlm_sql_freetds \ --without-rlm_sql_iodbc \ --without-rlm_sql_oracle \ --without-rlm_sql_sybase \ --without-rlm_sql_unixodbc \ --without-rlm_sql_log \ --without-rlm_sql_sqlite \ --without-rlm_caching \ --without-rlm_redis \ --without-rlm_rediswho \ --without-rlm_eap_tnc \ --without-rlm_eap_ikev2 \ --without-rlm_opendirectory \ --without-rlm_wimax \ --without-rlm_ruby \ --without-rlm_sql_mysql \ --without-rlm_sql_postgresql c Am 16.01.2015 um 00:42 schrieb Rich Felker: > On Thu, Jan 15, 2015 at 06:10:37PM -0500, Rich Felker wrote: >> On Thu, Jan 15, 2015 at 11:52:36PM +0100, Sebastian Gottschall wrote: >>> following test case >>> >>> configure freeradius with --with-threads (which is on by default) >>> if you start radiusd with your radius configuration you will see >>> that radius does not listen on any ports. it will hang in the >>> listener thread which creates the socket. >>> if you configure it as --without-threads, it works >>> >>> >>> tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10 >> Thanks for the report. I'm not terribly familiar with freeradius. Can >> you tell where it's hung, e.g. get a backtrace in gdb, and possibly >> strace -f leading up to the hang? > Short of that, a minimal procedure to reproduce the bug (build > configuration, runtime config files, which distro (if any) you're > using, and anything else needed in the environment, etc.) would be > helpful. > > Rich > [-- Attachment #2: Type: text/html, Size: 8693 bytes --] ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase) 2015-01-16 0:11 ` Sebastian Gottschall @ 2015-01-16 0:23 ` Rich Felker 2015-01-16 1:53 ` Sebastian Gottschall 0 siblings, 1 reply; 14+ messages in thread From: Rich Felker @ 2015-01-16 0:23 UTC (permalink / raw) To: musl On Fri, Jan 16, 2015 at 01:11:36AM +0100, Sebastian Gottschall wrote: > sorry i forgot the full content. of the message. the first attemt > failed due a too big attachment > > > i can help you more. because i found the deadlocking function. > its setresuid which hangs forever. (i know that you submitted a > patch recently which is related to that function, but i dont know if > this will help here) > debugging is more complicated with gdb here. its a embedded system > without any remote debugging support > > its exact the same problem which is described here, but without any > further solution or problem cause > > http://freeradius.1045715.n5.nabble.com/Hack-way-to-compile-freeradius-causing-freeradius-to-hang-under-multithread-mode-td2794760.html Interesting. If the deadlock happens in setresuid, this may actually be a very timely bug report. The mailing list threads about multithreaded set*id/synccall, and my new blog post based on them, http://ewontfix.com/17, are about fixing longstanding problems in this area. I'll look into how it's being used in freeradius and see if I can tell what's going on. I'm also going to be committing new code for this soon (probably in the next 24 hrs) so you could give it a try with that too and see if the problem goes away. Rich ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase) 2015-01-16 0:23 ` Rich Felker @ 2015-01-16 1:53 ` Sebastian Gottschall 0 siblings, 0 replies; 14+ messages in thread From: Sebastian Gottschall @ 2015-01-16 1:53 UTC (permalink / raw) To: musl Am 16.01.2015 um 01:23 schrieb Rich Felker: > On Fri, Jan 16, 2015 at 01:11:36AM +0100, Sebastian Gottschall wrote: >> sorry i forgot the full content. of the message. the first attemt >> failed due a too big attachment >> >> >> i can help you more. because i found the deadlocking function. >> its setresuid which hangs forever. (i know that you submitted a >> patch recently which is related to that function, but i dont know if >> this will help here) >> debugging is more complicated with gdb here. its a embedded system >> without any remote debugging support >> >> its exact the same problem which is described here, but without any >> further solution or problem cause >> >> http://freeradius.1045715.n5.nabble.com/Hack-way-to-compile-freeradius-causing-freeradius-to-hang-under-multithread-mode-td2794760.html > Interesting. If the deadlock happens in setresuid, this may actually > be a very timely bug report. The mailing list threads about > multithreaded set*id/synccall, and my new blog post based on them, > http://ewontfix.com/17, are about fixing longstanding problems in this > area. I'll look into how it's being used in freeradius and see if I > can tell what's going on. I'm also going to be committing new code for > this soon (probably in the next 24 hrs) so you could give it a try > with that too and see if the problem goes away. i will immediatly test it as soon as you commit it and give response thank you for your help Sebastian > Rich > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase) 2015-01-15 22:52 pthreads broken (freeradius testcase) Sebastian Gottschall 2015-01-15 23:10 ` Rich Felker @ 2015-01-16 7:20 ` Natanael Copa 2015-01-16 8:07 ` Sebastian Gottschall 2015-01-16 11:44 ` Sebastian Gottschall 1 sibling, 2 replies; 14+ messages in thread From: Natanael Copa @ 2015-01-16 7:20 UTC (permalink / raw) To: Sebastian Gottschall; +Cc: musl On Thu, 15 Jan 2015 23:52:36 +0100 Sebastian Gottschall <s.gottschall@dd-wrt.com> wrote: > following test case > > configure freeradius with --with-threads (which is on by default) > if you start radiusd with your radius configuration you will see that > radius does not listen on any ports. it will hang in the listener thread > which creates the socket. > if you configure it as --without-threads, it works > > > tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10 > > Sebastian What version of freeradius is it? I have had some interesting threading issues with freeradius 2.2.x. Some modules are marked as non-thread safe but will still run in a separate thread. It runs main thread + a single non-thread-safe thread. They used getgrnam and getpwnam in both main thread and in the non-thread-safe module so memory got corrupted. (IMHO this should get a CVE but upstream disagrees because it only happens on a non-recommended config) They fixed that in 3.x.x but AFAIK they didn't fix it in 2.x.x. Patches: http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-Use-threadsafe-wrapper-for-getpwnam-getgrnam.patch http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-use-threadsafe-rad_getgrnam.patch (upstream patched it differently in 3.x.x branch) When backporting the fix to 2.x.x I also found that the TLS configure test is completely broke in 2.x.x branch too. IIRC it will say "TLS found" but behind the scenes it will still disable TLS support. patch: http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/fix-tls-test.patch This is probably not the related the issue you have have at hand, but I'm would not be surprised if musl libc has unmasked another bug in freeradius. -nc ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase) 2015-01-16 7:20 ` Natanael Copa @ 2015-01-16 8:07 ` Sebastian Gottschall 2015-01-16 11:44 ` Sebastian Gottschall 1 sibling, 0 replies; 14+ messages in thread From: Sebastian Gottschall @ 2015-01-16 8:07 UTC (permalink / raw) To: Natanael Copa; +Cc: musl Am 16.01.2015 um 08:20 schrieb Natanael Copa: > On Thu, 15 Jan 2015 23:52:36 +0100 > Sebastian Gottschall <s.gottschall@dd-wrt.com> wrote: > >> following test case >> >> configure freeradius with --with-threads (which is on by default) >> if you start radiusd with your radius configuration you will see that >> radius does not listen on any ports. it will hang in the listener thread >> which creates the socket. >> if you configure it as --without-threads, it works >> >> >> tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10 >> >> Sebastian > What version of freeradius is it? latest 2.2.6 if you disable setguid support it will also work again. i tracked it down to the function > > I have had some interesting threading issues with freeradius 2.2.x. > Some modules are marked as non-thread safe but will still run in a > separate thread. It runs main thread + a single non-thread-safe thread. it already hangs in the main application while initializing the udp listeners. this can be also seen if you start it with -xx and write the debug output to a log file. > > They used getgrnam and getpwnam in both main thread and in the > non-thread-safe module so memory got corrupted. (IMHO this should get a > CVE but upstream disagrees because it only happens on a non-recommended > config) > > They fixed that in 3.x.x but AFAIK they didn't fix it in 2.x.x. > > Patches: > http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-Use-threadsafe-wrapper-for-getpwnam-getgrnam.patch > http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-use-threadsafe-rad_getgrnam.patch > > (upstream patched it differently in 3.x.x branch) > > When backporting the fix to 2.x.x I also found that the TLS configure > test is completely broke in 2.x.x branch too. IIRC it will say "TLS > found" but behind the scenes it will still disable TLS support. ouch. thanks. i stumble over that issue too. a user reported that bug recently to me. > > patch: > http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/fix-tls-test.patch > > This is probably not the related the issue you have have at hand, but > I'm would not be surprised if musl libc has unmasked another bug in > freeradius. it has. and you helped me alot just right now thanks Sebastian > > > -nc > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase) 2015-01-16 7:20 ` Natanael Copa 2015-01-16 8:07 ` Sebastian Gottschall @ 2015-01-16 11:44 ` Sebastian Gottschall 2015-01-16 16:25 ` Rich Felker 1 sibling, 1 reply; 14+ messages in thread From: Sebastian Gottschall @ 2015-01-16 11:44 UTC (permalink / raw) To: Natanael Copa; +Cc: musl Am 16.01.2015 um 08:20 schrieb Natanael Copa: > On Thu, 15 Jan 2015 23:52:36 +0100 > Sebastian Gottschall <s.gottschall@dd-wrt.com> wrote: > >> following test case >> >> configure freeradius with --with-threads (which is on by default) >> if you start radiusd with your radius configuration you will see that >> radius does not listen on any ports. it will hang in the listener thread >> which creates the socket. >> if you configure it as --without-threads, it works >> >> >> tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10 >> >> Sebastian > What version of freeradius is it? > > I have had some interesting threading issues with freeradius 2.2.x. > Some modules are marked as non-thread safe but will still run in a > separate thread. It runs main thread + a single non-thread-safe thread. > > They used getgrnam and getpwnam in both main thread and in the > non-thread-safe module so memory got corrupted. (IMHO this should get a > CVE but upstream disagrees because it only happens on a non-recommended > config) > > They fixed that in 3.x.x but AFAIK they didn't fix it in 2.x.x. > > Patches: > http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-Use-threadsafe-wrapper-for-getpwnam-getgrnam.patch > http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-use-threadsafe-rad_getgrnam.patch > > (upstream patched it differently in 3.x.x branch) > > When backporting the fix to 2.x.x I also found that the TLS configure > test is completely broke in 2.x.x branch too. IIRC it will say "TLS > found" but behind the scenes it will still disable TLS support. > > patch: > http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/fix-tls-test.patch > > This is probably not the related the issue you have have at hand, but > I'm would not be surprised if musl libc has unmasked another bug in > freeradius. i applied all these fixes. the first thread for port 1812 works, the second thread for internal tunnel 18120 doesnt work an hangs again. even if setuid support is disabled > > > -nc > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase) 2015-01-16 11:44 ` Sebastian Gottschall @ 2015-01-16 16:25 ` Rich Felker 2015-01-16 17:57 ` Sebastian Gottschall 0 siblings, 1 reply; 14+ messages in thread From: Rich Felker @ 2015-01-16 16:25 UTC (permalink / raw) To: musl On Fri, Jan 16, 2015 at 12:44:51PM +0100, Sebastian Gottschall wrote: > Am 16.01.2015 um 08:20 schrieb Natanael Copa: > >On Thu, 15 Jan 2015 23:52:36 +0100 > >Sebastian Gottschall <s.gottschall@dd-wrt.com> wrote: > > > >>following test case > >> > >>configure freeradius with --with-threads (which is on by default) > >>if you start radiusd with your radius configuration you will see that > >>radius does not listen on any ports. it will hang in the listener thread > >>which creates the socket. > >>if you configure it as --without-threads, it works > >> > >> > >>tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10 > >> > >>Sebastian > >What version of freeradius is it? > > > >I have had some interesting threading issues with freeradius 2.2.x. > >Some modules are marked as non-thread safe but will still run in a > >separate thread. It runs main thread + a single non-thread-safe thread. > > > >They used getgrnam and getpwnam in both main thread and in the > >non-thread-safe module so memory got corrupted. (IMHO this should get a > >CVE but upstream disagrees because it only happens on a non-recommended > >config) > > > >They fixed that in 3.x.x but AFAIK they didn't fix it in 2.x.x. > > > >Patches: > >http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-Use-threadsafe-wrapper-for-getpwnam-getgrnam.patch > >http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-use-threadsafe-rad_getgrnam.patch > > > >(upstream patched it differently in 3.x.x branch) > > > >When backporting the fix to 2.x.x I also found that the TLS configure > >test is completely broke in 2.x.x branch too. IIRC it will say "TLS > >found" but behind the scenes it will still disable TLS support. > > > >patch: > >http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/fix-tls-test.patch > > > >This is probably not the related the issue you have have at hand, but > >I'm would not be surprised if musl libc has unmasked another bug in > >freeradius. > i applied all these fixes. the first thread for port 1812 works, the > second thread for internal tunnel 18120 doesnt work > an hangs again. even if setuid support is disabled Could you report where the hang is occurring (using gdb backtrace) now with the setuid support disabled? Rich ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase) 2015-01-16 16:25 ` Rich Felker @ 2015-01-16 17:57 ` Sebastian Gottschall 2015-01-16 18:09 ` Rich Felker 0 siblings, 1 reply; 14+ messages in thread From: Sebastian Gottschall @ 2015-01-16 17:57 UTC (permalink / raw) To: musl Am 16.01.2015 um 17:25 schrieb Rich Felker: > On Fri, Jan 16, 2015 at 12:44:51PM +0100, Sebastian Gottschall wrote: >> Am 16.01.2015 um 08:20 schrieb Natanael Copa: >>> On Thu, 15 Jan 2015 23:52:36 +0100 >>> Sebastian Gottschall <s.gottschall@dd-wrt.com> wrote: >>> >>>> following test case >>>> >>>> configure freeradius with --with-threads (which is on by default) >>>> if you start radiusd with your radius configuration you will see that >>>> radius does not listen on any ports. it will hang in the listener thread >>>> which creates the socket. >>>> if you configure it as --without-threads, it works >>>> >>>> >>>> tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10 >>>> >>>> Sebastian >>> What version of freeradius is it? >>> >>> I have had some interesting threading issues with freeradius 2.2.x. >>> Some modules are marked as non-thread safe but will still run in a >>> separate thread. It runs main thread + a single non-thread-safe thread. >>> >>> They used getgrnam and getpwnam in both main thread and in the >>> non-thread-safe module so memory got corrupted. (IMHO this should get a >>> CVE but upstream disagrees because it only happens on a non-recommended >>> config) >>> >>> They fixed that in 3.x.x but AFAIK they didn't fix it in 2.x.x. >>> >>> Patches: >>> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-Use-threadsafe-wrapper-for-getpwnam-getgrnam.patch >>> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-use-threadsafe-rad_getgrnam.patch >>> >>> (upstream patched it differently in 3.x.x branch) >>> >>> When backporting the fix to 2.x.x I also found that the TLS configure >>> test is completely broke in 2.x.x branch too. IIRC it will say "TLS >>> found" but behind the scenes it will still disable TLS support. >>> >>> patch: >>> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/fix-tls-test.patch >>> >>> This is probably not the related the issue you have have at hand, but >>> I'm would not be surprised if musl libc has unmasked another bug in >>> freeradius. >> i applied all these fixes. the first thread for port 1812 works, the >> second thread for internal tunnel 18120 doesnt work >> an hangs again. even if setuid support is disabled > Could you report where the hang is occurring (using gdb backtrace) now > with the setuid support disabled? i will try my best to find it out. gdb is hard to run on embedded shrinked to death the firmware. :-) > > Rich > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase) 2015-01-16 17:57 ` Sebastian Gottschall @ 2015-01-16 18:09 ` Rich Felker 2015-01-16 21:59 ` Sebastian Gottschall 0 siblings, 1 reply; 14+ messages in thread From: Rich Felker @ 2015-01-16 18:09 UTC (permalink / raw) To: musl On Fri, Jan 16, 2015 at 06:57:45PM +0100, Sebastian Gottschall wrote: > Am 16.01.2015 um 17:25 schrieb Rich Felker: > >On Fri, Jan 16, 2015 at 12:44:51PM +0100, Sebastian Gottschall wrote: > >>Am 16.01.2015 um 08:20 schrieb Natanael Copa: > >>>On Thu, 15 Jan 2015 23:52:36 +0100 > >>>Sebastian Gottschall <s.gottschall@dd-wrt.com> wrote: > >>> > >>>>following test case > >>>> > >>>>configure freeradius with --with-threads (which is on by default) > >>>>if you start radiusd with your radius configuration you will see that > >>>>radius does not listen on any ports. it will hang in the listener thread > >>>>which creates the socket. > >>>>if you configure it as --without-threads, it works > >>>> > >>>> > >>>>tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10 > >>>> > >>>>Sebastian > >>>What version of freeradius is it? > >>> > >>>I have had some interesting threading issues with freeradius 2.2.x. > >>>Some modules are marked as non-thread safe but will still run in a > >>>separate thread. It runs main thread + a single non-thread-safe thread. > >>> > >>>They used getgrnam and getpwnam in both main thread and in the > >>>non-thread-safe module so memory got corrupted. (IMHO this should get a > >>>CVE but upstream disagrees because it only happens on a non-recommended > >>>config) > >>> > >>>They fixed that in 3.x.x but AFAIK they didn't fix it in 2.x.x. > >>> > >>>Patches: > >>>http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-Use-threadsafe-wrapper-for-getpwnam-getgrnam.patch > >>>http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-use-threadsafe-rad_getgrnam.patch > >>> > >>>(upstream patched it differently in 3.x.x branch) > >>> > >>>When backporting the fix to 2.x.x I also found that the TLS configure > >>>test is completely broke in 2.x.x branch too. IIRC it will say "TLS > >>>found" but behind the scenes it will still disable TLS support. > >>> > >>>patch: > >>>http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/fix-tls-test.patch > >>> > >>>This is probably not the related the issue you have have at hand, but > >>>I'm would not be surprised if musl libc has unmasked another bug in > >>>freeradius. > >>i applied all these fixes. the first thread for port 1812 works, the > >>second thread for internal tunnel 18120 doesnt work > >>an hangs again. even if setuid support is disabled > >Could you report where the hang is occurring (using gdb backtrace) now > >with the setuid support disabled? > i will try my best to find it out. gdb is hard to run on embedded > shrinked to death the firmware. :-) If you're familiar with debugging, it might be possible to determine where the threads are hung without gdb. /proc/$pid/task/$tid/stat should contain as field 30 the last-observed program-counter (aka instruction pointer) for the thread, which can be translated into a function address using your binaries and possibly the contents of /proc/self/maps. This won't give a full backtrace, but combined with the output of the strace utility it might give a good idea what code was running when it hung. If the above all sounds like [insert language you can't speak] to you, though, you're probably best sticking to getting gdb running on it. Rich ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase) 2015-01-16 18:09 ` Rich Felker @ 2015-01-16 21:59 ` Sebastian Gottschall 0 siblings, 0 replies; 14+ messages in thread From: Sebastian Gottschall @ 2015-01-16 21:59 UTC (permalink / raw) To: musl Am 16.01.2015 um 19:09 schrieb Rich Felker: > On Fri, Jan 16, 2015 at 06:57:45PM +0100, Sebastian Gottschall wrote: >> Am 16.01.2015 um 17:25 schrieb Rich Felker: >>> On Fri, Jan 16, 2015 at 12:44:51PM +0100, Sebastian Gottschall wrote: >>>> Am 16.01.2015 um 08:20 schrieb Natanael Copa: >>>>> On Thu, 15 Jan 2015 23:52:36 +0100 >>>>> Sebastian Gottschall <s.gottschall@dd-wrt.com> wrote: >>>>> >>>>>> following test case >>>>>> >>>>>> configure freeradius with --with-threads (which is on by default) >>>>>> if you start radiusd with your radius configuration you will see that >>>>>> radius does not listen on any ports. it will hang in the listener thread >>>>>> which creates the socket. >>>>>> if you configure it as --without-threads, it works >>>>>> >>>>>> >>>>>> tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10 >>>>>> >>>>>> Sebastian >>>>> What version of freeradius is it? >>>>> >>>>> I have had some interesting threading issues with freeradius 2.2.x. >>>>> Some modules are marked as non-thread safe but will still run in a >>>>> separate thread. It runs main thread + a single non-thread-safe thread. >>>>> >>>>> They used getgrnam and getpwnam in both main thread and in the >>>>> non-thread-safe module so memory got corrupted. (IMHO this should get a >>>>> CVE but upstream disagrees because it only happens on a non-recommended >>>>> config) >>>>> >>>>> They fixed that in 3.x.x but AFAIK they didn't fix it in 2.x.x. >>>>> >>>>> Patches: >>>>> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-Use-threadsafe-wrapper-for-getpwnam-getgrnam.patch >>>>> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-use-threadsafe-rad_getgrnam.patch >>>>> >>>>> (upstream patched it differently in 3.x.x branch) >>>>> >>>>> When backporting the fix to 2.x.x I also found that the TLS configure >>>>> test is completely broke in 2.x.x branch too. IIRC it will say "TLS >>>>> found" but behind the scenes it will still disable TLS support. >>>>> >>>>> patch: >>>>> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/fix-tls-test.patch >>>>> >>>>> This is probably not the related the issue you have have at hand, but >>>>> I'm would not be surprised if musl libc has unmasked another bug in >>>>> freeradius. >>>> i applied all these fixes. the first thread for port 1812 works, the >>>> second thread for internal tunnel 18120 doesnt work >>>> an hangs again. even if setuid support is disabled >>> Could you report where the hang is occurring (using gdb backtrace) now >>> with the setuid support disabled? >> i will try my best to find it out. gdb is hard to run on embedded >> shrinked to death the firmware. :-) > If you're familiar with debugging, it might be possible to determine > where the threads are hung without gdb. /proc/$pid/task/$tid/stat > should contain as field 30 the last-observed program-counter (aka > instruction pointer) for the thread, which can be translated into a > function address using your binaries and possibly the contents of > /proc/self/maps. This won't give a full backtrace, but combined with > the output of the strace utility it might give a good idea what code > was running when it hung. > > If the above all sounds like [insert language you can't speak] to you, > though, you're probably best sticking to getting gdb running on it. i'm pretty good in finding out such issues, even with the old printf way. will try it with the stats way, even if it will likelly point to a musl libc function. so a stack backtrace would be better to see the caller location > > Rich > ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2015-01-16 21:59 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2015-01-15 22:52 pthreads broken (freeradius testcase) Sebastian Gottschall 2015-01-15 23:10 ` Rich Felker 2015-01-15 23:42 ` Rich Felker 2015-01-15 23:53 ` Sebastian Gottschall 2015-01-16 0:11 ` Sebastian Gottschall 2015-01-16 0:23 ` Rich Felker 2015-01-16 1:53 ` Sebastian Gottschall 2015-01-16 7:20 ` Natanael Copa 2015-01-16 8:07 ` Sebastian Gottschall 2015-01-16 11:44 ` Sebastian Gottschall 2015-01-16 16:25 ` Rich Felker 2015-01-16 17:57 ` Sebastian Gottschall 2015-01-16 18:09 ` Rich Felker 2015-01-16 21:59 ` Sebastian Gottschall
Code repositories for project(s) associated with this public inbox https://git.vuxu.org/mirror/musl/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).