* pthreads broken (freeradius testcase)
@ 2015-01-15 22:52 Sebastian Gottschall
2015-01-15 23:10 ` Rich Felker
2015-01-16 7:20 ` Natanael Copa
0 siblings, 2 replies; 14+ messages in thread
From: Sebastian Gottschall @ 2015-01-15 22:52 UTC (permalink / raw)
To: musl
following test case
configure freeradius with --with-threads (which is on by default)
if you start radiusd with your radius configuration you will see that
radius does not listen on any ports. it will hang in the listener thread
which creates the socket.
if you configure it as --without-threads, it works
tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10
Sebastian
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase)
2015-01-15 22:52 pthreads broken (freeradius testcase) Sebastian Gottschall
@ 2015-01-15 23:10 ` Rich Felker
2015-01-15 23:42 ` Rich Felker
2015-01-16 7:20 ` Natanael Copa
1 sibling, 1 reply; 14+ messages in thread
From: Rich Felker @ 2015-01-15 23:10 UTC (permalink / raw)
To: musl
On Thu, Jan 15, 2015 at 11:52:36PM +0100, Sebastian Gottschall wrote:
> following test case
>
> configure freeradius with --with-threads (which is on by default)
> if you start radiusd with your radius configuration you will see
> that radius does not listen on any ports. it will hang in the
> listener thread which creates the socket.
> if you configure it as --without-threads, it works
>
>
> tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10
Thanks for the report. I'm not terribly familiar with freeradius. Can
you tell where it's hung, e.g. get a backtrace in gdb, and possibly
strace -f leading up to the hang?
Rich
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase)
2015-01-15 23:10 ` Rich Felker
@ 2015-01-15 23:42 ` Rich Felker
2015-01-15 23:53 ` Sebastian Gottschall
2015-01-16 0:11 ` Sebastian Gottschall
0 siblings, 2 replies; 14+ messages in thread
From: Rich Felker @ 2015-01-15 23:42 UTC (permalink / raw)
To: musl
On Thu, Jan 15, 2015 at 06:10:37PM -0500, Rich Felker wrote:
> On Thu, Jan 15, 2015 at 11:52:36PM +0100, Sebastian Gottschall wrote:
> > following test case
> >
> > configure freeradius with --with-threads (which is on by default)
> > if you start radiusd with your radius configuration you will see
> > that radius does not listen on any ports. it will hang in the
> > listener thread which creates the socket.
> > if you configure it as --without-threads, it works
> >
> >
> > tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10
>
> Thanks for the report. I'm not terribly familiar with freeradius. Can
> you tell where it's hung, e.g. get a backtrace in gdb, and possibly
> strace -f leading up to the hang?
Short of that, a minimal procedure to reproduce the bug (build
configuration, runtime config files, which distro (if any) you're
using, and anything else needed in the environment, etc.) would be
helpful.
Rich
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase)
2015-01-15 23:42 ` Rich Felker
@ 2015-01-15 23:53 ` Sebastian Gottschall
2015-01-16 0:11 ` Sebastian Gottschall
1 sibling, 0 replies; 14+ messages in thread
From: Sebastian Gottschall @ 2015-01-15 23:53 UTC (permalink / raw)
To: musl
[-- Attachment #1: Type: text/plain, Size: 3215 bytes --]
cd freeradius && \
sys_lib_dlsearch_path_spec="mips-uclibc" \
sys_lib_search_path_spec="mips-uclibc" \
MYSQL_CONFIG="no" \
ac_cv_lib_readline=no \
ac_cv_lib_ssl_SSL_new=yes \
ac_cv_lib_crypto_DH_new=yes \
ac_cv_func_strncasecmp=yes \
ac_cv_func_strcasecmp=yes \
ac_cv_lib_crypt_crypt=yes \
ac_cv_func_gettimeofday=yes \
ac_cv_func_getnameinfo=yes \
ac_cv_func_getaddrinfo=yes \
ac_cv_func_setlinebuf=yes \
ac_cv_host=mips-uclibc-linux \
./configure --target=mips-linux --host=mips CFLAGS="-Os -pipe -mips32r2
-mtune=mips32r2 -msoft-float -fno-caller-saves -mno-branch-likely
-minterlink-mips16 -mips16
-I/home/seg/DEV/pb42/src/router/openssl/include " LDFLAGS="-Os -pipe
-mips32r2 -mtune=mips32r2 -msoft-float -fno-caller-saves
-mno-branch-likely -minterlink-mips16 -mips16
-L/home/seg/DEV/pb42/src/router/openssl" --enable-shared \
--program-prefix="" \
--program-suffix="" \
--prefix=/usr \
--exec-prefix=/usr \
--bindir=/usr/bin \
--datadir=/usr/share \
--includedir=/usr/include \
--infodir=/usr/share/info \
--libdir=/usr/lib \
--libexecdir=/usr/lib \
--with-raddbdir=/etc/freeradius2 \
--with-radacctdir=/var/db/radacct \
--with-logdir=/var/log \
--localstatedir=/var \
--mandir=/usr/share/man \
--sbindir=/usr/sbin \
--sysconfdir=/etc \
--enable-shared \
--disable-static \
--disable-developer \
--with-threads \
--with-ltdl-include="/home/seg/DEV/pb42/src/router/freeradius/libltdl/.libs"
\
--with-ltdl-lib="/home/seg/DEV/pb42/src/router/freeradius/libltdl" \
--with-openssl-includes="/home/seg/DEV/pb42/src/router/openssl/include" \
--with-openssl-libraries="/home/seg/DEV/pb42/src/router/openssl" \
--enable-strict-dependencies \
--with-raddbdir=/etc/freeradius \
--without-edir \
--without-snmp \
--with-experimental-modules \
--without-rlm_attr-rewrite \
--without-rlm_checkval \
--without-rlm_counter \
--without-rlm_dbm \
--without-rlm_ldap \
--without-edir \
--without-snmp \
--with-rlm_expr \
--with-rlm_eap \
--without-rlm_eap_sim \
--without-rlm_example \
--without-rlm_ippool \
--without-rlm_krb5 \
--without-rlm_pam \
--without-rlm_perl \
--without-rlm_python \
--without-rlm_smb \
--with-rlm_sql \
--with-rlm_sqlcounter \
--without-rlm_sql_db2 \
--without-rlm_sql_freetds \
--without-rlm_sql_iodbc \
--without-rlm_sql_oracle \
--without-rlm_sql_sybase \
--without-rlm_sql_unixodbc \
--without-rlm_x99-token \
--without-rlm_eap_ikev2 \
--without-rlm_eap_tnc \
--without-rlm_opendirectory \
--without-rlm_wimax \
--with-rlm_eap_peap \
--with-rlm_eap_tls \
--with-rlm_eap_ttls \
--with-rlm_expiration \
--with-rlm_logintime \
--with-rlm_attr-rewrite \
--without-rlm_otp \
--without-rlm_smsotp \
--without-rlm_sqlhpwippool \
--without-rlm_sqlippool \
--without-rlm_sql_db2 \
--without-rlm_sql_firebird \
--without-rlm_sql_freetds \
--without-rlm_sql_iodbc \
--without-rlm_sql_oracle \
--without-rlm_sql_sybase \
--without-rlm_sql_unixodbc \
--without-rlm_sql_log \
--without-rlm_sql_sqlite \
--without-rlm_caching \
--without-rlm_redis \
--without-rlm_rediswho \
--without-rlm_eap_tnc \
--without-rlm_eap_ikev2 \
--without-rlm_opendirectory \
--without-rlm_wimax \
--without-rlm_ruby \
--without-rlm_sql_mysql \
--without-rlm_sql_postgresql
c
[-- Attachment #2: freeradius.tar.xz --]
[-- Type: application/octet-stream, Size: 133540 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase)
2015-01-15 23:42 ` Rich Felker
2015-01-15 23:53 ` Sebastian Gottschall
@ 2015-01-16 0:11 ` Sebastian Gottschall
2015-01-16 0:23 ` Rich Felker
1 sibling, 1 reply; 14+ messages in thread
From: Sebastian Gottschall @ 2015-01-16 0:11 UTC (permalink / raw)
To: musl
[-- Attachment #1: Type: text/plain, Size: 5197 bytes --]
sorry i forgot the full content. of the message. the first attemt failed
due a too big attachment
i can help you more. because i found the deadlocking function.
its setresuid which hangs forever. (i know that you submitted a patch
recently which is related to that function, but i dont know if this will
help here)
debugging is more complicated with gdb here. its a embedded system
without any remote debugging support
its exact the same problem which is described here, but without any
further solution or problem cause
http://freeradius.1045715.n5.nabble.com/Hack-way-to-compile-freeradius-causing-freeradius-to-hang-under-multithread-mode-td2794760.html
distribution is simply dd-wrt (router os). so maybe hard for you to work
with.
i can provide you the full configs used which can be directly executed
using radiusd -d /directory to config
the full configure line for freeradius, just taken out of my buildsystem
(you likelly need to adjust some paths)
cd freeradius && \
sys_lib_dlsearch_path_spec="mips-uclibc" \
sys_lib_search_path_spec="mips-uclibc" \
MYSQL_CONFIG="no" \
ac_cv_lib_readline=no \
ac_cv_lib_ssl_SSL_new=yes \
ac_cv_lib_crypto_DH_new=yes \
ac_cv_func_strncasecmp=yes \
ac_cv_func_strcasecmp=yes \
ac_cv_lib_crypt_crypt=yes \
ac_cv_func_gettimeofday=yes \
ac_cv_func_getnameinfo=yes \
ac_cv_func_getaddrinfo=yes \
ac_cv_func_setlinebuf=yes \
ac_cv_host=mips-uclibc-linux \
./configure --target=mips-linux --host=mips CFLAGS="-Os -pipe -mips32r2
-mtune=mips32r2 -msoft-float -fno-caller-saves -mno-branch-likely
-minterlink-mips16 -mips16
-I/home/seg/DEV/pb42/src/router/openssl/include " LDFLAGS="-Os -pipe
-mips32r2 -mtune=mips32r2 -msoft-float -fno-caller-saves
-mno-branch-likely -minterlink-mips16 -mips16
-L/home/seg/DEV/pb42/src/router/openssl" --enable-shared \
--program-prefix="" \
--program-suffix="" \
--prefix=/usr \
--exec-prefix=/usr \
--bindir=/usr/bin \
--datadir=/usr/share \
--includedir=/usr/include \
--infodir=/usr/share/info \
--libdir=/usr/lib \
--libexecdir=/usr/lib \
--with-raddbdir=/etc/freeradius2 \
--with-radacctdir=/var/db/radacct \
--with-logdir=/var/log \
--localstatedir=/var \
--mandir=/usr/share/man \
--sbindir=/usr/sbin \
--sysconfdir=/etc \
--enable-shared \
--disable-static \
--disable-developer \
--with-threads \
--with-ltdl-include="/home/seg/DEV/pb42/src/router/freeradius/libltdl/.libs"
\
--with-ltdl-lib="/home/seg/DEV/pb42/src/router/freeradius/libltdl" \
--with-openssl-includes="/home/seg/DEV/pb42/src/router/openssl/include" \
--with-openssl-libraries="/home/seg/DEV/pb42/src/router/openssl" \
--enable-strict-dependencies \
--with-raddbdir=/etc/freeradius \
--without-edir \
--without-snmp \
--with-experimental-modules \
--without-rlm_attr-rewrite \
--without-rlm_checkval \
--without-rlm_counter \
--without-rlm_dbm \
--without-rlm_ldap \
--without-edir \
--without-snmp \
--with-rlm_expr \
--with-rlm_eap \
--without-rlm_eap_sim \
--without-rlm_example \
--without-rlm_ippool \
--without-rlm_krb5 \
--without-rlm_pam \
--without-rlm_perl \
--without-rlm_python \
--without-rlm_smb \
--with-rlm_sql \
--with-rlm_sqlcounter \
--without-rlm_sql_db2 \
--without-rlm_sql_freetds \
--without-rlm_sql_iodbc \
--without-rlm_sql_oracle \
--without-rlm_sql_sybase \
--without-rlm_sql_unixodbc \
--without-rlm_x99-token \
--without-rlm_eap_ikev2 \
--without-rlm_eap_tnc \
--without-rlm_opendirectory \
--without-rlm_wimax \
--with-rlm_eap_peap \
--with-rlm_eap_tls \
--with-rlm_eap_ttls \
--with-rlm_expiration \
--with-rlm_logintime \
--with-rlm_attr-rewrite \
--without-rlm_otp \
--without-rlm_smsotp \
--without-rlm_sqlhpwippool \
--without-rlm_sqlippool \
--without-rlm_sql_db2 \
--without-rlm_sql_firebird \
--without-rlm_sql_freetds \
--without-rlm_sql_iodbc \
--without-rlm_sql_oracle \
--without-rlm_sql_sybase \
--without-rlm_sql_unixodbc \
--without-rlm_sql_log \
--without-rlm_sql_sqlite \
--without-rlm_caching \
--without-rlm_redis \
--without-rlm_rediswho \
--without-rlm_eap_tnc \
--without-rlm_eap_ikev2 \
--without-rlm_opendirectory \
--without-rlm_wimax \
--without-rlm_ruby \
--without-rlm_sql_mysql \
--without-rlm_sql_postgresql
c
Am 16.01.2015 um 00:42 schrieb Rich Felker:
> On Thu, Jan 15, 2015 at 06:10:37PM -0500, Rich Felker wrote:
>> On Thu, Jan 15, 2015 at 11:52:36PM +0100, Sebastian Gottschall wrote:
>>> following test case
>>>
>>> configure freeradius with --with-threads (which is on by default)
>>> if you start radiusd with your radius configuration you will see
>>> that radius does not listen on any ports. it will hang in the
>>> listener thread which creates the socket.
>>> if you configure it as --without-threads, it works
>>>
>>>
>>> tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10
>> Thanks for the report. I'm not terribly familiar with freeradius. Can
>> you tell where it's hung, e.g. get a backtrace in gdb, and possibly
>> strace -f leading up to the hang?
> Short of that, a minimal procedure to reproduce the bug (build
> configuration, runtime config files, which distro (if any) you're
> using, and anything else needed in the environment, etc.) would be
> helpful.
>
> Rich
>
[-- Attachment #2: Type: text/html, Size: 8693 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase)
2015-01-16 0:11 ` Sebastian Gottschall
@ 2015-01-16 0:23 ` Rich Felker
2015-01-16 1:53 ` Sebastian Gottschall
0 siblings, 1 reply; 14+ messages in thread
From: Rich Felker @ 2015-01-16 0:23 UTC (permalink / raw)
To: musl
On Fri, Jan 16, 2015 at 01:11:36AM +0100, Sebastian Gottschall wrote:
> sorry i forgot the full content. of the message. the first attemt
> failed due a too big attachment
>
>
> i can help you more. because i found the deadlocking function.
> its setresuid which hangs forever. (i know that you submitted a
> patch recently which is related to that function, but i dont know if
> this will help here)
> debugging is more complicated with gdb here. its a embedded system
> without any remote debugging support
>
> its exact the same problem which is described here, but without any
> further solution or problem cause
>
> http://freeradius.1045715.n5.nabble.com/Hack-way-to-compile-freeradius-causing-freeradius-to-hang-under-multithread-mode-td2794760.html
Interesting. If the deadlock happens in setresuid, this may actually
be a very timely bug report. The mailing list threads about
multithreaded set*id/synccall, and my new blog post based on them,
http://ewontfix.com/17, are about fixing longstanding problems in this
area. I'll look into how it's being used in freeradius and see if I
can tell what's going on. I'm also going to be committing new code for
this soon (probably in the next 24 hrs) so you could give it a try
with that too and see if the problem goes away.
Rich
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase)
2015-01-16 0:23 ` Rich Felker
@ 2015-01-16 1:53 ` Sebastian Gottschall
0 siblings, 0 replies; 14+ messages in thread
From: Sebastian Gottschall @ 2015-01-16 1:53 UTC (permalink / raw)
To: musl
Am 16.01.2015 um 01:23 schrieb Rich Felker:
> On Fri, Jan 16, 2015 at 01:11:36AM +0100, Sebastian Gottschall wrote:
>> sorry i forgot the full content. of the message. the first attemt
>> failed due a too big attachment
>>
>>
>> i can help you more. because i found the deadlocking function.
>> its setresuid which hangs forever. (i know that you submitted a
>> patch recently which is related to that function, but i dont know if
>> this will help here)
>> debugging is more complicated with gdb here. its a embedded system
>> without any remote debugging support
>>
>> its exact the same problem which is described here, but without any
>> further solution or problem cause
>>
>> http://freeradius.1045715.n5.nabble.com/Hack-way-to-compile-freeradius-causing-freeradius-to-hang-under-multithread-mode-td2794760.html
> Interesting. If the deadlock happens in setresuid, this may actually
> be a very timely bug report. The mailing list threads about
> multithreaded set*id/synccall, and my new blog post based on them,
> http://ewontfix.com/17, are about fixing longstanding problems in this
> area. I'll look into how it's being used in freeradius and see if I
> can tell what's going on. I'm also going to be committing new code for
> this soon (probably in the next 24 hrs) so you could give it a try
> with that too and see if the problem goes away.
i will immediatly test it as soon as you commit it and give response
thank you for your help
Sebastian
> Rich
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase)
2015-01-15 22:52 pthreads broken (freeradius testcase) Sebastian Gottschall
2015-01-15 23:10 ` Rich Felker
@ 2015-01-16 7:20 ` Natanael Copa
2015-01-16 8:07 ` Sebastian Gottschall
2015-01-16 11:44 ` Sebastian Gottschall
1 sibling, 2 replies; 14+ messages in thread
From: Natanael Copa @ 2015-01-16 7:20 UTC (permalink / raw)
To: Sebastian Gottschall; +Cc: musl
On Thu, 15 Jan 2015 23:52:36 +0100
Sebastian Gottschall <s.gottschall@dd-wrt.com> wrote:
> following test case
>
> configure freeradius with --with-threads (which is on by default)
> if you start radiusd with your radius configuration you will see that
> radius does not listen on any ports. it will hang in the listener thread
> which creates the socket.
> if you configure it as --without-threads, it works
>
>
> tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10
>
> Sebastian
What version of freeradius is it?
I have had some interesting threading issues with freeradius 2.2.x.
Some modules are marked as non-thread safe but will still run in a
separate thread. It runs main thread + a single non-thread-safe thread.
They used getgrnam and getpwnam in both main thread and in the
non-thread-safe module so memory got corrupted. (IMHO this should get a
CVE but upstream disagrees because it only happens on a non-recommended
config)
They fixed that in 3.x.x but AFAIK they didn't fix it in 2.x.x.
Patches:
http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-Use-threadsafe-wrapper-for-getpwnam-getgrnam.patch
http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-use-threadsafe-rad_getgrnam.patch
(upstream patched it differently in 3.x.x branch)
When backporting the fix to 2.x.x I also found that the TLS configure
test is completely broke in 2.x.x branch too. IIRC it will say "TLS
found" but behind the scenes it will still disable TLS support.
patch:
http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/fix-tls-test.patch
This is probably not the related the issue you have have at hand, but
I'm would not be surprised if musl libc has unmasked another bug in
freeradius.
-nc
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase)
2015-01-16 7:20 ` Natanael Copa
@ 2015-01-16 8:07 ` Sebastian Gottschall
2015-01-16 11:44 ` Sebastian Gottschall
1 sibling, 0 replies; 14+ messages in thread
From: Sebastian Gottschall @ 2015-01-16 8:07 UTC (permalink / raw)
To: Natanael Copa; +Cc: musl
Am 16.01.2015 um 08:20 schrieb Natanael Copa:
> On Thu, 15 Jan 2015 23:52:36 +0100
> Sebastian Gottschall <s.gottschall@dd-wrt.com> wrote:
>
>> following test case
>>
>> configure freeradius with --with-threads (which is on by default)
>> if you start radiusd with your radius configuration you will see that
>> radius does not listen on any ports. it will hang in the listener thread
>> which creates the socket.
>> if you configure it as --without-threads, it works
>>
>>
>> tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10
>>
>> Sebastian
> What version of freeradius is it?
latest 2.2.6
if you disable setguid support it will also work again. i tracked it
down to the function
>
> I have had some interesting threading issues with freeradius 2.2.x.
> Some modules are marked as non-thread safe but will still run in a
> separate thread. It runs main thread + a single non-thread-safe thread.
it already hangs in the main application while initializing the udp
listeners. this can be also seen if you start it with -xx and write the
debug output to a log file.
>
> They used getgrnam and getpwnam in both main thread and in the
> non-thread-safe module so memory got corrupted. (IMHO this should get a
> CVE but upstream disagrees because it only happens on a non-recommended
> config)
>
> They fixed that in 3.x.x but AFAIK they didn't fix it in 2.x.x.
>
> Patches:
> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-Use-threadsafe-wrapper-for-getpwnam-getgrnam.patch
> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-use-threadsafe-rad_getgrnam.patch
>
> (upstream patched it differently in 3.x.x branch)
>
> When backporting the fix to 2.x.x I also found that the TLS configure
> test is completely broke in 2.x.x branch too. IIRC it will say "TLS
> found" but behind the scenes it will still disable TLS support.
ouch. thanks. i stumble over that issue too. a user reported that bug
recently to me.
>
> patch:
> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/fix-tls-test.patch
>
> This is probably not the related the issue you have have at hand, but
> I'm would not be surprised if musl libc has unmasked another bug in
> freeradius.
it has. and you helped me alot just right now
thanks
Sebastian
>
>
> -nc
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase)
2015-01-16 7:20 ` Natanael Copa
2015-01-16 8:07 ` Sebastian Gottschall
@ 2015-01-16 11:44 ` Sebastian Gottschall
2015-01-16 16:25 ` Rich Felker
1 sibling, 1 reply; 14+ messages in thread
From: Sebastian Gottschall @ 2015-01-16 11:44 UTC (permalink / raw)
To: Natanael Copa; +Cc: musl
Am 16.01.2015 um 08:20 schrieb Natanael Copa:
> On Thu, 15 Jan 2015 23:52:36 +0100
> Sebastian Gottschall <s.gottschall@dd-wrt.com> wrote:
>
>> following test case
>>
>> configure freeradius with --with-threads (which is on by default)
>> if you start radiusd with your radius configuration you will see that
>> radius does not listen on any ports. it will hang in the listener thread
>> which creates the socket.
>> if you configure it as --without-threads, it works
>>
>>
>> tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10
>>
>> Sebastian
> What version of freeradius is it?
>
> I have had some interesting threading issues with freeradius 2.2.x.
> Some modules are marked as non-thread safe but will still run in a
> separate thread. It runs main thread + a single non-thread-safe thread.
>
> They used getgrnam and getpwnam in both main thread and in the
> non-thread-safe module so memory got corrupted. (IMHO this should get a
> CVE but upstream disagrees because it only happens on a non-recommended
> config)
>
> They fixed that in 3.x.x but AFAIK they didn't fix it in 2.x.x.
>
> Patches:
> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-Use-threadsafe-wrapper-for-getpwnam-getgrnam.patch
> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-use-threadsafe-rad_getgrnam.patch
>
> (upstream patched it differently in 3.x.x branch)
>
> When backporting the fix to 2.x.x I also found that the TLS configure
> test is completely broke in 2.x.x branch too. IIRC it will say "TLS
> found" but behind the scenes it will still disable TLS support.
>
> patch:
> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/fix-tls-test.patch
>
> This is probably not the related the issue you have have at hand, but
> I'm would not be surprised if musl libc has unmasked another bug in
> freeradius.
i applied all these fixes. the first thread for port 1812 works, the
second thread for internal tunnel 18120 doesnt work
an hangs again. even if setuid support is disabled
>
>
> -nc
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase)
2015-01-16 11:44 ` Sebastian Gottschall
@ 2015-01-16 16:25 ` Rich Felker
2015-01-16 17:57 ` Sebastian Gottschall
0 siblings, 1 reply; 14+ messages in thread
From: Rich Felker @ 2015-01-16 16:25 UTC (permalink / raw)
To: musl
On Fri, Jan 16, 2015 at 12:44:51PM +0100, Sebastian Gottschall wrote:
> Am 16.01.2015 um 08:20 schrieb Natanael Copa:
> >On Thu, 15 Jan 2015 23:52:36 +0100
> >Sebastian Gottschall <s.gottschall@dd-wrt.com> wrote:
> >
> >>following test case
> >>
> >>configure freeradius with --with-threads (which is on by default)
> >>if you start radiusd with your radius configuration you will see that
> >>radius does not listen on any ports. it will hang in the listener thread
> >>which creates the socket.
> >>if you configure it as --without-threads, it works
> >>
> >>
> >>tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10
> >>
> >>Sebastian
> >What version of freeradius is it?
> >
> >I have had some interesting threading issues with freeradius 2.2.x.
> >Some modules are marked as non-thread safe but will still run in a
> >separate thread. It runs main thread + a single non-thread-safe thread.
> >
> >They used getgrnam and getpwnam in both main thread and in the
> >non-thread-safe module so memory got corrupted. (IMHO this should get a
> >CVE but upstream disagrees because it only happens on a non-recommended
> >config)
> >
> >They fixed that in 3.x.x but AFAIK they didn't fix it in 2.x.x.
> >
> >Patches:
> >http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-Use-threadsafe-wrapper-for-getpwnam-getgrnam.patch
> >http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-use-threadsafe-rad_getgrnam.patch
> >
> >(upstream patched it differently in 3.x.x branch)
> >
> >When backporting the fix to 2.x.x I also found that the TLS configure
> >test is completely broke in 2.x.x branch too. IIRC it will say "TLS
> >found" but behind the scenes it will still disable TLS support.
> >
> >patch:
> >http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/fix-tls-test.patch
> >
> >This is probably not the related the issue you have have at hand, but
> >I'm would not be surprised if musl libc has unmasked another bug in
> >freeradius.
> i applied all these fixes. the first thread for port 1812 works, the
> second thread for internal tunnel 18120 doesnt work
> an hangs again. even if setuid support is disabled
Could you report where the hang is occurring (using gdb backtrace) now
with the setuid support disabled?
Rich
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase)
2015-01-16 16:25 ` Rich Felker
@ 2015-01-16 17:57 ` Sebastian Gottschall
2015-01-16 18:09 ` Rich Felker
0 siblings, 1 reply; 14+ messages in thread
From: Sebastian Gottschall @ 2015-01-16 17:57 UTC (permalink / raw)
To: musl
Am 16.01.2015 um 17:25 schrieb Rich Felker:
> On Fri, Jan 16, 2015 at 12:44:51PM +0100, Sebastian Gottschall wrote:
>> Am 16.01.2015 um 08:20 schrieb Natanael Copa:
>>> On Thu, 15 Jan 2015 23:52:36 +0100
>>> Sebastian Gottschall <s.gottschall@dd-wrt.com> wrote:
>>>
>>>> following test case
>>>>
>>>> configure freeradius with --with-threads (which is on by default)
>>>> if you start radiusd with your radius configuration you will see that
>>>> radius does not listen on any ports. it will hang in the listener thread
>>>> which creates the socket.
>>>> if you configure it as --without-threads, it works
>>>>
>>>>
>>>> tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10
>>>>
>>>> Sebastian
>>> What version of freeradius is it?
>>>
>>> I have had some interesting threading issues with freeradius 2.2.x.
>>> Some modules are marked as non-thread safe but will still run in a
>>> separate thread. It runs main thread + a single non-thread-safe thread.
>>>
>>> They used getgrnam and getpwnam in both main thread and in the
>>> non-thread-safe module so memory got corrupted. (IMHO this should get a
>>> CVE but upstream disagrees because it only happens on a non-recommended
>>> config)
>>>
>>> They fixed that in 3.x.x but AFAIK they didn't fix it in 2.x.x.
>>>
>>> Patches:
>>> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-Use-threadsafe-wrapper-for-getpwnam-getgrnam.patch
>>> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-use-threadsafe-rad_getgrnam.patch
>>>
>>> (upstream patched it differently in 3.x.x branch)
>>>
>>> When backporting the fix to 2.x.x I also found that the TLS configure
>>> test is completely broke in 2.x.x branch too. IIRC it will say "TLS
>>> found" but behind the scenes it will still disable TLS support.
>>>
>>> patch:
>>> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/fix-tls-test.patch
>>>
>>> This is probably not the related the issue you have have at hand, but
>>> I'm would not be surprised if musl libc has unmasked another bug in
>>> freeradius.
>> i applied all these fixes. the first thread for port 1812 works, the
>> second thread for internal tunnel 18120 doesnt work
>> an hangs again. even if setuid support is disabled
> Could you report where the hang is occurring (using gdb backtrace) now
> with the setuid support disabled?
i will try my best to find it out. gdb is hard to run on embedded
shrinked to death the firmware. :-)
>
> Rich
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase)
2015-01-16 17:57 ` Sebastian Gottschall
@ 2015-01-16 18:09 ` Rich Felker
2015-01-16 21:59 ` Sebastian Gottschall
0 siblings, 1 reply; 14+ messages in thread
From: Rich Felker @ 2015-01-16 18:09 UTC (permalink / raw)
To: musl
On Fri, Jan 16, 2015 at 06:57:45PM +0100, Sebastian Gottschall wrote:
> Am 16.01.2015 um 17:25 schrieb Rich Felker:
> >On Fri, Jan 16, 2015 at 12:44:51PM +0100, Sebastian Gottschall wrote:
> >>Am 16.01.2015 um 08:20 schrieb Natanael Copa:
> >>>On Thu, 15 Jan 2015 23:52:36 +0100
> >>>Sebastian Gottschall <s.gottschall@dd-wrt.com> wrote:
> >>>
> >>>>following test case
> >>>>
> >>>>configure freeradius with --with-threads (which is on by default)
> >>>>if you start radiusd with your radius configuration you will see that
> >>>>radius does not listen on any ports. it will hang in the listener thread
> >>>>which creates the socket.
> >>>>if you configure it as --without-threads, it works
> >>>>
> >>>>
> >>>>tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10
> >>>>
> >>>>Sebastian
> >>>What version of freeradius is it?
> >>>
> >>>I have had some interesting threading issues with freeradius 2.2.x.
> >>>Some modules are marked as non-thread safe but will still run in a
> >>>separate thread. It runs main thread + a single non-thread-safe thread.
> >>>
> >>>They used getgrnam and getpwnam in both main thread and in the
> >>>non-thread-safe module so memory got corrupted. (IMHO this should get a
> >>>CVE but upstream disagrees because it only happens on a non-recommended
> >>>config)
> >>>
> >>>They fixed that in 3.x.x but AFAIK they didn't fix it in 2.x.x.
> >>>
> >>>Patches:
> >>>http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-Use-threadsafe-wrapper-for-getpwnam-getgrnam.patch
> >>>http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-use-threadsafe-rad_getgrnam.patch
> >>>
> >>>(upstream patched it differently in 3.x.x branch)
> >>>
> >>>When backporting the fix to 2.x.x I also found that the TLS configure
> >>>test is completely broke in 2.x.x branch too. IIRC it will say "TLS
> >>>found" but behind the scenes it will still disable TLS support.
> >>>
> >>>patch:
> >>>http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/fix-tls-test.patch
> >>>
> >>>This is probably not the related the issue you have have at hand, but
> >>>I'm would not be surprised if musl libc has unmasked another bug in
> >>>freeradius.
> >>i applied all these fixes. the first thread for port 1812 works, the
> >>second thread for internal tunnel 18120 doesnt work
> >>an hangs again. even if setuid support is disabled
> >Could you report where the hang is occurring (using gdb backtrace) now
> >with the setuid support disabled?
> i will try my best to find it out. gdb is hard to run on embedded
> shrinked to death the firmware. :-)
If you're familiar with debugging, it might be possible to determine
where the threads are hung without gdb. /proc/$pid/task/$tid/stat
should contain as field 30 the last-observed program-counter (aka
instruction pointer) for the thread, which can be translated into a
function address using your binaries and possibly the contents of
/proc/self/maps. This won't give a full backtrace, but combined with
the output of the strace utility it might give a good idea what code
was running when it hung.
If the above all sounds like [insert language you can't speak] to you,
though, you're probably best sticking to getting gdb running on it.
Rich
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: pthreads broken (freeradius testcase)
2015-01-16 18:09 ` Rich Felker
@ 2015-01-16 21:59 ` Sebastian Gottschall
0 siblings, 0 replies; 14+ messages in thread
From: Sebastian Gottschall @ 2015-01-16 21:59 UTC (permalink / raw)
To: musl
Am 16.01.2015 um 19:09 schrieb Rich Felker:
> On Fri, Jan 16, 2015 at 06:57:45PM +0100, Sebastian Gottschall wrote:
>> Am 16.01.2015 um 17:25 schrieb Rich Felker:
>>> On Fri, Jan 16, 2015 at 12:44:51PM +0100, Sebastian Gottschall wrote:
>>>> Am 16.01.2015 um 08:20 schrieb Natanael Copa:
>>>>> On Thu, 15 Jan 2015 23:52:36 +0100
>>>>> Sebastian Gottschall <s.gottschall@dd-wrt.com> wrote:
>>>>>
>>>>>> following test case
>>>>>>
>>>>>> configure freeradius with --with-threads (which is on by default)
>>>>>> if you start radiusd with your radius configuration you will see that
>>>>>> radius does not listen on any ports. it will hang in the listener thread
>>>>>> which creates the socket.
>>>>>> if you configure it as --without-threads, it works
>>>>>>
>>>>>>
>>>>>> tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10
>>>>>>
>>>>>> Sebastian
>>>>> What version of freeradius is it?
>>>>>
>>>>> I have had some interesting threading issues with freeradius 2.2.x.
>>>>> Some modules are marked as non-thread safe but will still run in a
>>>>> separate thread. It runs main thread + a single non-thread-safe thread.
>>>>>
>>>>> They used getgrnam and getpwnam in both main thread and in the
>>>>> non-thread-safe module so memory got corrupted. (IMHO this should get a
>>>>> CVE but upstream disagrees because it only happens on a non-recommended
>>>>> config)
>>>>>
>>>>> They fixed that in 3.x.x but AFAIK they didn't fix it in 2.x.x.
>>>>>
>>>>> Patches:
>>>>> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-Use-threadsafe-wrapper-for-getpwnam-getgrnam.patch
>>>>> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-use-threadsafe-rad_getgrnam.patch
>>>>>
>>>>> (upstream patched it differently in 3.x.x branch)
>>>>>
>>>>> When backporting the fix to 2.x.x I also found that the TLS configure
>>>>> test is completely broke in 2.x.x branch too. IIRC it will say "TLS
>>>>> found" but behind the scenes it will still disable TLS support.
>>>>>
>>>>> patch:
>>>>> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/fix-tls-test.patch
>>>>>
>>>>> This is probably not the related the issue you have have at hand, but
>>>>> I'm would not be surprised if musl libc has unmasked another bug in
>>>>> freeradius.
>>>> i applied all these fixes. the first thread for port 1812 works, the
>>>> second thread for internal tunnel 18120 doesnt work
>>>> an hangs again. even if setuid support is disabled
>>> Could you report where the hang is occurring (using gdb backtrace) now
>>> with the setuid support disabled?
>> i will try my best to find it out. gdb is hard to run on embedded
>> shrinked to death the firmware. :-)
> If you're familiar with debugging, it might be possible to determine
> where the threads are hung without gdb. /proc/$pid/task/$tid/stat
> should contain as field 30 the last-observed program-counter (aka
> instruction pointer) for the thread, which can be translated into a
> function address using your binaries and possibly the contents of
> /proc/self/maps. This won't give a full backtrace, but combined with
> the output of the strace utility it might give a good idea what code
> was running when it hung.
>
> If the above all sounds like [insert language you can't speak] to you,
> though, you're probably best sticking to getting gdb running on it.
i'm pretty good in finding out such issues, even with the old printf way.
will try it with the stats way, even if it will likelly point to a musl
libc function. so a stack backtrace would be better to see the caller
location
>
> Rich
>
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2015-01-16 21:59 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-01-15 22:52 pthreads broken (freeradius testcase) Sebastian Gottschall
2015-01-15 23:10 ` Rich Felker
2015-01-15 23:42 ` Rich Felker
2015-01-15 23:53 ` Sebastian Gottschall
2015-01-16 0:11 ` Sebastian Gottschall
2015-01-16 0:23 ` Rich Felker
2015-01-16 1:53 ` Sebastian Gottschall
2015-01-16 7:20 ` Natanael Copa
2015-01-16 8:07 ` Sebastian Gottschall
2015-01-16 11:44 ` Sebastian Gottschall
2015-01-16 16:25 ` Rich Felker
2015-01-16 17:57 ` Sebastian Gottschall
2015-01-16 18:09 ` Rich Felker
2015-01-16 21:59 ` Sebastian Gottschall
Code repositories for project(s) associated with this public inbox
https://git.vuxu.org/mirror/musl/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).