pthreads broken (freeradius testcase)

pthreads broken (freeradius testcase)

[Sebastian Gottschall]

following test case

configure freeradius with --with-threads (which is on by default)
if you start radiusd with your radius configuration you will see that 
radius does not listen on any ports. it will hang in the listener thread 
which creates the socket.
if you configure it as --without-threads, it works


tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10

Sebastian

Re: pthreads broken (freeradius testcase)

[Rich Felker]

On Thu, Jan 15, 2015 at 11:52:36PM +0100, Sebastian Gottschall wrote:
> following test case
> 
> configure freeradius with --with-threads (which is on by default)
> if you start radiusd with your radius configuration you will see
> that radius does not listen on any ports. it will hang in the
> listener thread which creates the socket.
> if you configure it as --without-threads, it works
> 
> 
> tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10

Thanks for the report. I'm not terribly familiar with freeradius. Can
you tell where it's hung, e.g. get a backtrace in gdb, and possibly
strace -f leading up to the hang?

Rich

Re: pthreads broken (freeradius testcase)

[Rich Felker]

On Thu, Jan 15, 2015 at 06:10:37PM -0500, Rich Felker wrote:
> On Thu, Jan 15, 2015 at 11:52:36PM +0100, Sebastian Gottschall wrote:
> > following test case
> > 
> > configure freeradius with --with-threads (which is on by default)
> > if you start radiusd with your radius configuration you will see
> > that radius does not listen on any ports. it will hang in the
> > listener thread which creates the socket.
> > if you configure it as --without-threads, it works
> > 
> > 
> > tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10
> 
> Thanks for the report. I'm not terribly familiar with freeradius. Can
> you tell where it's hung, e.g. get a backtrace in gdb, and possibly
> strace -f leading up to the hang?

Short of that, a minimal procedure to reproduce the bug (build
configuration, runtime config files, which distro (if any) you're
using, and anything else needed in the environment, etc.) would be
helpful.

Rich

Re: pthreads broken (freeradius testcase)

[Sebastian Gottschall]

[-- Attachment #1: Type: text/plain, Size: 3215 bytes --]

cd freeradius && \
sys_lib_dlsearch_path_spec="mips-uclibc" \
sys_lib_search_path_spec="mips-uclibc" \
MYSQL_CONFIG="no" \
ac_cv_lib_readline=no \
ac_cv_lib_ssl_SSL_new=yes \
ac_cv_lib_crypto_DH_new=yes \
ac_cv_func_strncasecmp=yes \
ac_cv_func_strcasecmp=yes \
ac_cv_lib_crypt_crypt=yes \
ac_cv_func_gettimeofday=yes \
ac_cv_func_getnameinfo=yes \
ac_cv_func_getaddrinfo=yes \
ac_cv_func_setlinebuf=yes \
ac_cv_host=mips-uclibc-linux \
./configure  --target=mips-linux --host=mips CFLAGS="-Os -pipe -mips32r2 
-mtune=mips32r2 -msoft-float  -fno-caller-saves -mno-branch-likely 
-minterlink-mips16 -mips16 
-I/home/seg/DEV/pb42/src/router/openssl/include " LDFLAGS="-Os -pipe 
-mips32r2 -mtune=mips32r2 -msoft-float  -fno-caller-saves 
-mno-branch-likely -minterlink-mips16 -mips16 
-L/home/seg/DEV/pb42/src/router/openssl" --enable-shared \
--program-prefix="" \
--program-suffix="" \
--prefix=/usr \
--exec-prefix=/usr \
--bindir=/usr/bin \
--datadir=/usr/share \
--includedir=/usr/include \
--infodir=/usr/share/info \
--libdir=/usr/lib \
--libexecdir=/usr/lib \
--with-raddbdir=/etc/freeradius2 \
--with-radacctdir=/var/db/radacct \
--with-logdir=/var/log \
--localstatedir=/var \
--mandir=/usr/share/man \
--sbindir=/usr/sbin \
--sysconfdir=/etc \
--enable-shared \
--disable-static \
--disable-developer \
--with-threads \
--with-ltdl-include="/home/seg/DEV/pb42/src/router/freeradius/libltdl/.libs" 
\
--with-ltdl-lib="/home/seg/DEV/pb42/src/router/freeradius/libltdl" \
--with-openssl-includes="/home/seg/DEV/pb42/src/router/openssl/include" \
--with-openssl-libraries="/home/seg/DEV/pb42/src/router/openssl" \
--enable-strict-dependencies \
--with-raddbdir=/etc/freeradius \
--without-edir \
--without-snmp \
--with-experimental-modules \
--without-rlm_attr-rewrite \
--without-rlm_checkval \
--without-rlm_counter \
--without-rlm_dbm \
--without-rlm_ldap \
--without-edir \
--without-snmp \
--with-rlm_expr \
--with-rlm_eap \
--without-rlm_eap_sim \
--without-rlm_example \
--without-rlm_ippool \
--without-rlm_krb5 \
--without-rlm_pam \
--without-rlm_perl \
--without-rlm_python \
--without-rlm_smb \
--with-rlm_sql \
--with-rlm_sqlcounter \
--without-rlm_sql_db2 \
--without-rlm_sql_freetds \
--without-rlm_sql_iodbc \
--without-rlm_sql_oracle \
--without-rlm_sql_sybase \
--without-rlm_sql_unixodbc \
--without-rlm_x99-token \
         --without-rlm_eap_ikev2 \
         --without-rlm_eap_tnc \
--without-rlm_opendirectory \
--without-rlm_wimax \
         --with-rlm_eap_peap \
         --with-rlm_eap_tls \
         --with-rlm_eap_ttls \
--with-rlm_expiration \
--with-rlm_logintime \
--with-rlm_attr-rewrite \
--without-rlm_otp \
--without-rlm_smsotp \
--without-rlm_sqlhpwippool \
--without-rlm_sqlippool \
--without-rlm_sql_db2 \
--without-rlm_sql_firebird \
--without-rlm_sql_freetds \
--without-rlm_sql_iodbc \
--without-rlm_sql_oracle \
--without-rlm_sql_sybase \
--without-rlm_sql_unixodbc \
--without-rlm_sql_log \
--without-rlm_sql_sqlite \
--without-rlm_caching \
--without-rlm_redis \
--without-rlm_rediswho \
--without-rlm_eap_tnc \
--without-rlm_eap_ikev2 \
--without-rlm_opendirectory \
--without-rlm_wimax \
--without-rlm_ruby \
--without-rlm_sql_mysql \
--without-rlm_sql_postgresql
c

[-- Attachment #2: freeradius.tar.xz --]
[-- Type: application/octet-stream, Size: 133540 bytes --]

Re: pthreads broken (freeradius testcase)

[Sebastian Gottschall]

[-- Attachment #1: Type: text/plain, Size: 5197 bytes --]

sorry i forgot the full content. of the message. the first attemt failed 
due a too big attachment


i can help you more. because i found the deadlocking function.
its setresuid which hangs forever.  (i know that you submitted a patch 
recently which is related to that function, but i dont know if this will 
help here)
debugging is more complicated with gdb here. its a embedded system 
without any remote debugging support

its exact the same problem which is described here, but without any 
further solution or problem cause

http://freeradius.1045715.n5.nabble.com/Hack-way-to-compile-freeradius-causing-freeradius-to-hang-under-multithread-mode-td2794760.html 


distribution is simply dd-wrt (router os). so maybe hard for you to work 
with.
i can provide you the full configs used which can be directly executed 
using radiusd -d /directory to config


the full configure line for freeradius, just taken out of my buildsystem 
(you likelly need to adjust some paths)

cd freeradius && \
sys_lib_dlsearch_path_spec="mips-uclibc" \
sys_lib_search_path_spec="mips-uclibc" \
MYSQL_CONFIG="no" \
ac_cv_lib_readline=no \
ac_cv_lib_ssl_SSL_new=yes \
ac_cv_lib_crypto_DH_new=yes \
ac_cv_func_strncasecmp=yes \
ac_cv_func_strcasecmp=yes \
ac_cv_lib_crypt_crypt=yes \
ac_cv_func_gettimeofday=yes \
ac_cv_func_getnameinfo=yes \
ac_cv_func_getaddrinfo=yes \
ac_cv_func_setlinebuf=yes \
ac_cv_host=mips-uclibc-linux \
./configure  --target=mips-linux --host=mips CFLAGS="-Os -pipe -mips32r2 
-mtune=mips32r2 -msoft-float  -fno-caller-saves -mno-branch-likely 
-minterlink-mips16 -mips16 
-I/home/seg/DEV/pb42/src/router/openssl/include " LDFLAGS="-Os -pipe 
-mips32r2 -mtune=mips32r2 -msoft-float  -fno-caller-saves 
-mno-branch-likely -minterlink-mips16 -mips16 
-L/home/seg/DEV/pb42/src/router/openssl" --enable-shared \
--program-prefix="" \
--program-suffix="" \
--prefix=/usr \
--exec-prefix=/usr \
--bindir=/usr/bin \
--datadir=/usr/share \
--includedir=/usr/include \
--infodir=/usr/share/info \
--libdir=/usr/lib \
--libexecdir=/usr/lib \
--with-raddbdir=/etc/freeradius2 \
--with-radacctdir=/var/db/radacct \
--with-logdir=/var/log \
--localstatedir=/var \
--mandir=/usr/share/man \
--sbindir=/usr/sbin \
--sysconfdir=/etc \
--enable-shared \
--disable-static \
--disable-developer \
--with-threads \
--with-ltdl-include="/home/seg/DEV/pb42/src/router/freeradius/libltdl/.libs" 
\
--with-ltdl-lib="/home/seg/DEV/pb42/src/router/freeradius/libltdl" \
--with-openssl-includes="/home/seg/DEV/pb42/src/router/openssl/include" \
--with-openssl-libraries="/home/seg/DEV/pb42/src/router/openssl" \
--enable-strict-dependencies \
--with-raddbdir=/etc/freeradius \
--without-edir \
--without-snmp \
--with-experimental-modules \
--without-rlm_attr-rewrite \
--without-rlm_checkval \
--without-rlm_counter \
--without-rlm_dbm \
--without-rlm_ldap \
--without-edir \
--without-snmp \
--with-rlm_expr \
--with-rlm_eap \
--without-rlm_eap_sim \
--without-rlm_example \
--without-rlm_ippool \
--without-rlm_krb5 \
--without-rlm_pam \
--without-rlm_perl \
--without-rlm_python \
--without-rlm_smb \
--with-rlm_sql \
--with-rlm_sqlcounter \
--without-rlm_sql_db2 \
--without-rlm_sql_freetds \
--without-rlm_sql_iodbc \
--without-rlm_sql_oracle \
--without-rlm_sql_sybase \
--without-rlm_sql_unixodbc \
--without-rlm_x99-token \
         --without-rlm_eap_ikev2 \
         --without-rlm_eap_tnc \
--without-rlm_opendirectory \
--without-rlm_wimax \
         --with-rlm_eap_peap \
         --with-rlm_eap_tls \
         --with-rlm_eap_ttls \
--with-rlm_expiration \
--with-rlm_logintime \
--with-rlm_attr-rewrite \
--without-rlm_otp \
--without-rlm_smsotp \
--without-rlm_sqlhpwippool \
--without-rlm_sqlippool \
--without-rlm_sql_db2 \
--without-rlm_sql_firebird \
--without-rlm_sql_freetds \
--without-rlm_sql_iodbc \
--without-rlm_sql_oracle \
--without-rlm_sql_sybase \
--without-rlm_sql_unixodbc \
--without-rlm_sql_log \
--without-rlm_sql_sqlite \
--without-rlm_caching \
--without-rlm_redis \
--without-rlm_rediswho \
--without-rlm_eap_tnc \
--without-rlm_eap_ikev2 \
--without-rlm_opendirectory \
--without-rlm_wimax \
--without-rlm_ruby \
--without-rlm_sql_mysql \
--without-rlm_sql_postgresql
c


Am 16.01.2015 um 00:42 schrieb Rich Felker:
> On Thu, Jan 15, 2015 at 06:10:37PM -0500, Rich Felker wrote:
>> On Thu, Jan 15, 2015 at 11:52:36PM +0100, Sebastian Gottschall wrote:
>>> following test case
>>>
>>> configure freeradius with --with-threads (which is on by default)
>>> if you start radiusd with your radius configuration you will see
>>> that radius does not listen on any ports. it will hang in the
>>> listener thread which creates the socket.
>>> if you configure it as --without-threads, it works
>>>
>>>
>>> tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10
>> Thanks for the report. I'm not terribly familiar with freeradius. Can
>> you tell where it's hung, e.g. get a backtrace in gdb, and possibly
>> strace -f leading up to the hang?
> Short of that, a minimal procedure to reproduce the bug (build
> configuration, runtime config files, which distro (if any) you're
> using, and anything else needed in the environment, etc.) would be
> helpful.
>
> Rich
>


[-- Attachment #2: Type: text/html, Size: 8693 bytes --]

Re: pthreads broken (freeradius testcase)

[Rich Felker]

On Fri, Jan 16, 2015 at 01:11:36AM +0100, Sebastian Gottschall wrote:
> sorry i forgot the full content. of the message. the first attemt
> failed due a too big attachment
> 
> 
> i can help you more. because i found the deadlocking function.
> its setresuid which hangs forever.  (i know that you submitted a
> patch recently which is related to that function, but i dont know if
> this will help here)
> debugging is more complicated with gdb here. its a embedded system
> without any remote debugging support
> 
> its exact the same problem which is described here, but without any
> further solution or problem cause
> 
> http://freeradius.1045715.n5.nabble.com/Hack-way-to-compile-freeradius-causing-freeradius-to-hang-under-multithread-mode-td2794760.html

Interesting. If the deadlock happens in setresuid, this may actually
be a very timely bug report. The mailing list threads about
multithreaded set*id/synccall, and my new blog post based on them,
http://ewontfix.com/17, are about fixing longstanding problems in this
area. I'll look into how it's being used in freeradius and see if I
can tell what's going on. I'm also going to be committing new code for
this soon (probably in the next 24 hrs) so you could give it a try
with that too and see if the problem goes away.

Rich

Re: pthreads broken (freeradius testcase)

[Sebastian Gottschall]

Am 16.01.2015 um 01:23 schrieb Rich Felker:
> On Fri, Jan 16, 2015 at 01:11:36AM +0100, Sebastian Gottschall wrote:
>> sorry i forgot the full content. of the message. the first attemt
>> failed due a too big attachment
>>
>>
>> i can help you more. because i found the deadlocking function.
>> its setresuid which hangs forever.  (i know that you submitted a
>> patch recently which is related to that function, but i dont know if
>> this will help here)
>> debugging is more complicated with gdb here. its a embedded system
>> without any remote debugging support
>>
>> its exact the same problem which is described here, but without any
>> further solution or problem cause
>>
>> http://freeradius.1045715.n5.nabble.com/Hack-way-to-compile-freeradius-causing-freeradius-to-hang-under-multithread-mode-td2794760.html
> Interesting. If the deadlock happens in setresuid, this may actually
> be a very timely bug report. The mailing list threads about
> multithreaded set*id/synccall, and my new blog post based on them,
> http://ewontfix.com/17, are about fixing longstanding problems in this
> area. I'll look into how it's being used in freeradius and see if I
> can tell what's going on. I'm also going to be committing new code for
> this soon (probably in the next 24 hrs) so you could give it a try
> with that too and see if the problem goes away.
i will immediatly test it as soon as you commit it and give response
thank you for your help

Sebastian
> Rich
>

Re: pthreads broken (freeradius testcase)

[Natanael Copa]

On Thu, 15 Jan 2015 23:52:36 +0100
Sebastian Gottschall <s.gottschall@dd-wrt.com> wrote:

> following test case
> 
> configure freeradius with --with-threads (which is on by default)
> if you start radiusd with your radius configuration you will see that 
> radius does not listen on any ports. it will hang in the listener thread 
> which creates the socket.
> if you configure it as --without-threads, it works
> 
> 
> tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10
> 
> Sebastian

What version of freeradius is it?

I have had some interesting threading issues with freeradius 2.2.x.
Some modules are marked as non-thread safe but will still run in a
separate thread. It runs main thread + a single non-thread-safe thread.

They used getgrnam and getpwnam in both main thread and in the
non-thread-safe module so memory got corrupted. (IMHO this should get a
CVE but upstream disagrees because it only happens on a non-recommended
config)

They fixed that in 3.x.x but AFAIK they didn't fix it in 2.x.x.

Patches:
http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-Use-threadsafe-wrapper-for-getpwnam-getgrnam.patch
http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-use-threadsafe-rad_getgrnam.patch

(upstream patched it differently in 3.x.x branch)

When backporting the fix to 2.x.x I also found that the TLS configure
test is completely broke in 2.x.x branch too. IIRC it will say "TLS
found" but behind the scenes it will still disable TLS support.

patch:
http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/fix-tls-test.patch

This is probably not the related the issue you have have at hand, but
I'm would not be surprised if musl libc has unmasked another bug in
freeradius.


-nc

Re: pthreads broken (freeradius testcase)

[Sebastian Gottschall]

Am 16.01.2015 um 08:20 schrieb Natanael Copa:
> On Thu, 15 Jan 2015 23:52:36 +0100
> Sebastian Gottschall <s.gottschall@dd-wrt.com> wrote:
>
>> following test case
>>
>> configure freeradius with --with-threads (which is on by default)
>> if you start radiusd with your radius configuration you will see that
>> radius does not listen on any ports. it will hang in the listener thread
>> which creates the socket.
>> if you configure it as --without-threads, it works
>>
>>
>> tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10
>>
>> Sebastian
> What version of freeradius is it?
latest 2.2.6
if you disable setguid support it will also work again. i tracked it 
down to the function
>
> I have had some interesting threading issues with freeradius 2.2.x.
> Some modules are marked as non-thread safe but will still run in a
> separate thread. It runs main thread + a single non-thread-safe thread.
it already hangs in the main application while initializing the udp 
listeners. this can be also seen if you start it with -xx and write the 
debug output to a log file.

>
> They used getgrnam and getpwnam in both main thread and in the
> non-thread-safe module so memory got corrupted. (IMHO this should get a
> CVE but upstream disagrees because it only happens on a non-recommended
> config)
>
> They fixed that in 3.x.x but AFAIK they didn't fix it in 2.x.x.
>
> Patches:
> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-Use-threadsafe-wrapper-for-getpwnam-getgrnam.patch
> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-use-threadsafe-rad_getgrnam.patch
>
> (upstream patched it differently in 3.x.x branch)
>
> When backporting the fix to 2.x.x I also found that the TLS configure
> test is completely broke in 2.x.x branch too. IIRC it will say "TLS
> found" but behind the scenes it will still disable TLS support.
ouch. thanks. i stumble over that issue too. a user reported that bug 
recently to me.
>
> patch:
> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/fix-tls-test.patch
>
> This is probably not the related the issue you have have at hand, but
> I'm would not be surprised if musl libc has unmasked another bug in
> freeradius.
it has. and you helped me alot just right now
thanks

Sebastian
>
>
> -nc
>

Re: pthreads broken (freeradius testcase)

[Sebastian Gottschall]

Am 16.01.2015 um 08:20 schrieb Natanael Copa:
> On Thu, 15 Jan 2015 23:52:36 +0100
> Sebastian Gottschall <s.gottschall@dd-wrt.com> wrote:
>
>> following test case
>>
>> configure freeradius with --with-threads (which is on by default)
>> if you start radiusd with your radius configuration you will see that
>> radius does not listen on any ports. it will hang in the listener thread
>> which creates the socket.
>> if you configure it as --without-threads, it works
>>
>>
>> tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10
>>
>> Sebastian
> What version of freeradius is it?
>
> I have had some interesting threading issues with freeradius 2.2.x.
> Some modules are marked as non-thread safe but will still run in a
> separate thread. It runs main thread + a single non-thread-safe thread.
>
> They used getgrnam and getpwnam in both main thread and in the
> non-thread-safe module so memory got corrupted. (IMHO this should get a
> CVE but upstream disagrees because it only happens on a non-recommended
> config)
>
> They fixed that in 3.x.x but AFAIK they didn't fix it in 2.x.x.
>
> Patches:
> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-Use-threadsafe-wrapper-for-getpwnam-getgrnam.patch
> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-use-threadsafe-rad_getgrnam.patch
>
> (upstream patched it differently in 3.x.x branch)
>
> When backporting the fix to 2.x.x I also found that the TLS configure
> test is completely broke in 2.x.x branch too. IIRC it will say "TLS
> found" but behind the scenes it will still disable TLS support.
>
> patch:
> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/fix-tls-test.patch
>
> This is probably not the related the issue you have have at hand, but
> I'm would not be surprised if musl libc has unmasked another bug in
> freeradius.
i applied all these fixes. the first thread for port 1812 works, the 
second thread for internal tunnel 18120 doesnt work
an hangs again. even if setuid support is disabled
>
>
> -nc
>

Re: pthreads broken (freeradius testcase)

[Rich Felker]

On Fri, Jan 16, 2015 at 12:44:51PM +0100, Sebastian Gottschall wrote:
> Am 16.01.2015 um 08:20 schrieb Natanael Copa:
> >On Thu, 15 Jan 2015 23:52:36 +0100
> >Sebastian Gottschall <s.gottschall@dd-wrt.com> wrote:
> >
> >>following test case
> >>
> >>configure freeradius with --with-threads (which is on by default)
> >>if you start radiusd with your radius configuration you will see that
> >>radius does not listen on any ports. it will hang in the listener thread
> >>which creates the socket.
> >>if you configure it as --without-threads, it works
> >>
> >>
> >>tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10
> >>
> >>Sebastian
> >What version of freeradius is it?
> >
> >I have had some interesting threading issues with freeradius 2.2.x.
> >Some modules are marked as non-thread safe but will still run in a
> >separate thread. It runs main thread + a single non-thread-safe thread.
> >
> >They used getgrnam and getpwnam in both main thread and in the
> >non-thread-safe module so memory got corrupted. (IMHO this should get a
> >CVE but upstream disagrees because it only happens on a non-recommended
> >config)
> >
> >They fixed that in 3.x.x but AFAIK they didn't fix it in 2.x.x.
> >
> >Patches:
> >http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-Use-threadsafe-wrapper-for-getpwnam-getgrnam.patch
> >http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-use-threadsafe-rad_getgrnam.patch
> >
> >(upstream patched it differently in 3.x.x branch)
> >
> >When backporting the fix to 2.x.x I also found that the TLS configure
> >test is completely broke in 2.x.x branch too. IIRC it will say "TLS
> >found" but behind the scenes it will still disable TLS support.
> >
> >patch:
> >http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/fix-tls-test.patch
> >
> >This is probably not the related the issue you have have at hand, but
> >I'm would not be surprised if musl libc has unmasked another bug in
> >freeradius.
> i applied all these fixes. the first thread for port 1812 works, the
> second thread for internal tunnel 18120 doesnt work
> an hangs again. even if setuid support is disabled

Could you report where the hang is occurring (using gdb backtrace) now
with the setuid support disabled?

Rich

Re: pthreads broken (freeradius testcase)

[Sebastian Gottschall]

Am 16.01.2015 um 17:25 schrieb Rich Felker:
> On Fri, Jan 16, 2015 at 12:44:51PM +0100, Sebastian Gottschall wrote:
>> Am 16.01.2015 um 08:20 schrieb Natanael Copa:
>>> On Thu, 15 Jan 2015 23:52:36 +0100
>>> Sebastian Gottschall <s.gottschall@dd-wrt.com> wrote:
>>>
>>>> following test case
>>>>
>>>> configure freeradius with --with-threads (which is on by default)
>>>> if you start radiusd with your radius configuration you will see that
>>>> radius does not listen on any ports. it will hang in the listener thread
>>>> which creates the socket.
>>>> if you configure it as --without-threads, it works
>>>>
>>>>
>>>> tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10
>>>>
>>>> Sebastian
>>> What version of freeradius is it?
>>>
>>> I have had some interesting threading issues with freeradius 2.2.x.
>>> Some modules are marked as non-thread safe but will still run in a
>>> separate thread. It runs main thread + a single non-thread-safe thread.
>>>
>>> They used getgrnam and getpwnam in both main thread and in the
>>> non-thread-safe module so memory got corrupted. (IMHO this should get a
>>> CVE but upstream disagrees because it only happens on a non-recommended
>>> config)
>>>
>>> They fixed that in 3.x.x but AFAIK they didn't fix it in 2.x.x.
>>>
>>> Patches:
>>> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-Use-threadsafe-wrapper-for-getpwnam-getgrnam.patch
>>> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-use-threadsafe-rad_getgrnam.patch
>>>
>>> (upstream patched it differently in 3.x.x branch)
>>>
>>> When backporting the fix to 2.x.x I also found that the TLS configure
>>> test is completely broke in 2.x.x branch too. IIRC it will say "TLS
>>> found" but behind the scenes it will still disable TLS support.
>>>
>>> patch:
>>> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/fix-tls-test.patch
>>>
>>> This is probably not the related the issue you have have at hand, but
>>> I'm would not be surprised if musl libc has unmasked another bug in
>>> freeradius.
>> i applied all these fixes. the first thread for port 1812 works, the
>> second thread for internal tunnel 18120 doesnt work
>> an hangs again. even if setuid support is disabled
> Could you report where the hang is occurring (using gdb backtrace) now
> with the setuid support disabled?
i will try my best to find it out. gdb is hard to run on embedded 
shrinked to death the firmware.  :-)
>
> Rich
>

Re: pthreads broken (freeradius testcase)

[Rich Felker]

On Fri, Jan 16, 2015 at 06:57:45PM +0100, Sebastian Gottschall wrote:
> Am 16.01.2015 um 17:25 schrieb Rich Felker:
> >On Fri, Jan 16, 2015 at 12:44:51PM +0100, Sebastian Gottschall wrote:
> >>Am 16.01.2015 um 08:20 schrieb Natanael Copa:
> >>>On Thu, 15 Jan 2015 23:52:36 +0100
> >>>Sebastian Gottschall <s.gottschall@dd-wrt.com> wrote:
> >>>
> >>>>following test case
> >>>>
> >>>>configure freeradius with --with-threads (which is on by default)
> >>>>if you start radiusd with your radius configuration you will see that
> >>>>radius does not listen on any ports. it will hang in the listener thread
> >>>>which creates the socket.
> >>>>if you configure it as --without-threads, it works
> >>>>
> >>>>
> >>>>tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10
> >>>>
> >>>>Sebastian
> >>>What version of freeradius is it?
> >>>
> >>>I have had some interesting threading issues with freeradius 2.2.x.
> >>>Some modules are marked as non-thread safe but will still run in a
> >>>separate thread. It runs main thread + a single non-thread-safe thread.
> >>>
> >>>They used getgrnam and getpwnam in both main thread and in the
> >>>non-thread-safe module so memory got corrupted. (IMHO this should get a
> >>>CVE but upstream disagrees because it only happens on a non-recommended
> >>>config)
> >>>
> >>>They fixed that in 3.x.x but AFAIK they didn't fix it in 2.x.x.
> >>>
> >>>Patches:
> >>>http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-Use-threadsafe-wrapper-for-getpwnam-getgrnam.patch
> >>>http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-use-threadsafe-rad_getgrnam.patch
> >>>
> >>>(upstream patched it differently in 3.x.x branch)
> >>>
> >>>When backporting the fix to 2.x.x I also found that the TLS configure
> >>>test is completely broke in 2.x.x branch too. IIRC it will say "TLS
> >>>found" but behind the scenes it will still disable TLS support.
> >>>
> >>>patch:
> >>>http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/fix-tls-test.patch
> >>>
> >>>This is probably not the related the issue you have have at hand, but
> >>>I'm would not be surprised if musl libc has unmasked another bug in
> >>>freeradius.
> >>i applied all these fixes. the first thread for port 1812 works, the
> >>second thread for internal tunnel 18120 doesnt work
> >>an hangs again. even if setuid support is disabled
> >Could you report where the hang is occurring (using gdb backtrace) now
> >with the setuid support disabled?
> i will try my best to find it out. gdb is hard to run on embedded
> shrinked to death the firmware.  :-)

If you're familiar with debugging, it might be possible to determine
where the threads are hung without gdb. /proc/$pid/task/$tid/stat
should contain as field 30 the last-observed program-counter (aka
instruction pointer) for the thread, which can be translated into a
function address using your binaries and possibly the contents of
/proc/self/maps. This won't give a full backtrace, but combined with
the output of the strace utility it might give a good idea what code
was running when it hung.

If the above all sounds like [insert language you can't speak] to you,
though, you're probably best sticking to getting gdb running on it.

Rich

Re: pthreads broken (freeradius testcase)

[Sebastian Gottschall]

Am 16.01.2015 um 19:09 schrieb Rich Felker:
> On Fri, Jan 16, 2015 at 06:57:45PM +0100, Sebastian Gottschall wrote:
>> Am 16.01.2015 um 17:25 schrieb Rich Felker:
>>> On Fri, Jan 16, 2015 at 12:44:51PM +0100, Sebastian Gottschall wrote:
>>>> Am 16.01.2015 um 08:20 schrieb Natanael Copa:
>>>>> On Thu, 15 Jan 2015 23:52:36 +0100
>>>>> Sebastian Gottschall <s.gottschall@dd-wrt.com> wrote:
>>>>>
>>>>>> following test case
>>>>>>
>>>>>> configure freeradius with --with-threads (which is on by default)
>>>>>> if you start radiusd with your radius configuration you will see that
>>>>>> radius does not listen on any ports. it will hang in the listener thread
>>>>>> which creates the socket.
>>>>>> if you configure it as --without-threads, it works
>>>>>>
>>>>>>
>>>>>> tested with musl 1.1.6 on a mips (big endian) system using kernel 3.10
>>>>>>
>>>>>> Sebastian
>>>>> What version of freeradius is it?
>>>>>
>>>>> I have had some interesting threading issues with freeradius 2.2.x.
>>>>> Some modules are marked as non-thread safe but will still run in a
>>>>> separate thread. It runs main thread + a single non-thread-safe thread.
>>>>>
>>>>> They used getgrnam and getpwnam in both main thread and in the
>>>>> non-thread-safe module so memory got corrupted. (IMHO this should get a
>>>>> CVE but upstream disagrees because it only happens on a non-recommended
>>>>> config)
>>>>>
>>>>> They fixed that in 3.x.x but AFAIK they didn't fix it in 2.x.x.
>>>>>
>>>>> Patches:
>>>>> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-Use-threadsafe-wrapper-for-getpwnam-getgrnam.patch
>>>>> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/0001-use-threadsafe-rad_getgrnam.patch
>>>>>
>>>>> (upstream patched it differently in 3.x.x branch)
>>>>>
>>>>> When backporting the fix to 2.x.x I also found that the TLS configure
>>>>> test is completely broke in 2.x.x branch too. IIRC it will say "TLS
>>>>> found" but behind the scenes it will still disable TLS support.
>>>>>
>>>>> patch:
>>>>> http://git.alpinelinux.org/cgit/aports/tree/main/freeradius/fix-tls-test.patch
>>>>>
>>>>> This is probably not the related the issue you have have at hand, but
>>>>> I'm would not be surprised if musl libc has unmasked another bug in
>>>>> freeradius.
>>>> i applied all these fixes. the first thread for port 1812 works, the
>>>> second thread for internal tunnel 18120 doesnt work
>>>> an hangs again. even if setuid support is disabled
>>> Could you report where the hang is occurring (using gdb backtrace) now
>>> with the setuid support disabled?
>> i will try my best to find it out. gdb is hard to run on embedded
>> shrinked to death the firmware.  :-)
> If you're familiar with debugging, it might be possible to determine
> where the threads are hung without gdb. /proc/$pid/task/$tid/stat
> should contain as field 30 the last-observed program-counter (aka
> instruction pointer) for the thread, which can be translated into a
> function address using your binaries and possibly the contents of
> /proc/self/maps. This won't give a full backtrace, but combined with
> the output of the strace utility it might give a good idea what code
> was running when it hung.
>
> If the above all sounds like [insert language you can't speak] to you,
> though, you're probably best sticking to getting gdb running on it.
i'm pretty good in finding out such issues, even with the old printf way.
will try it with the stats way, even if it will likelly point to a musl 
libc function. so a stack backtrace would be better to see the caller 
location
>
> Rich
>