[9fans] off-topic linux threading bug complaint

[9fans] off-topic linux threading bug complaint

[Russ Cox]

I'm sorry for the off-topic post, but I have no idea
where to send this and I suspect there are still Linux
experts lurking here who do.  Please feel free to forward.

	$ uname -a
	Linux t23 2.6.8-1-686 #1 Thu Oct 7 03:15:25 EDT 2004 i686 GNU/Linux
	$

This program creates a second pthread and then
lets the first pthread exit:

	$ cat x.c
	#include <stdio.h>
	#include <unistd.h>
	#include <pthread.h>

	void*
	sleeper(void *a)
	{
		for(;;)
			sleep(1000);
	}

	int
	main(void)
	{
		pthread_t id;

		pthread_create(&id, 0, sleeper, 0);
		pthread_exit(0);
	}
	$ gcc x.c -lpthread
	$ ./a.out &
	[1] 4723
	$

Because the first pthread has exited, the ps listing
makes it look like the process is a zombie, even
though it's still running:

	$ ps axwwu|grep a.out |grep -v grep
	rsc       4723  0.0  0.0     0    0 pts/2    Z    13:29   0:00
[a.out] <defunct>
	$

The -T and -L flags to ps don't help:

	$ ps -T axwwu|grep a.out |grep -v grep
	$ ps -L axwwu|grep a.out |grep -v grep
	$

It's not ps just misdisplaying information: gdb cannot attach either.

	$ gdb a.out 4723
	GNU gdb 6.1-debian
	Copyright 2004 Free Software Foundation, Inc.
	GDB is free software, covered by the GNU General Public License, and you are
	welcome to change it and/or distribute copies of it under certain conditions.
	Type "show copying" to see the conditions.
	There is absolutely no warranty for GDB.  Type "show warranty" for details.
	This GDB was configured as "i386-linux"...Using host libthread_db
library "/lib/tls/libthread_db.so.1".

	Attaching to program: /home/rsc/a.out, process 4723
	ptrace: Operation not permitted.
	/home/rsc/4723: No such file or directory.
	(gdb) quit
	$

On the other hand, if I guess the process/thread id of the
second pthread, then I can still look at that:

	$ gdb a.out 4724
	GNU gdb 6.1-debian
	Copyright 2004 Free Software Foundation, Inc.
	GDB is free software, covered by the GNU General Public License, and you are
	welcome to change it and/or distribute copies of it under certain conditions.
	Type "show copying" to see the conditions.
	There is absolutely no warranty for GDB.  Type "show warranty" for details.
	This GDB was configured as "i386-linux"...Using host libthread_db
library "/lib/tls/libthread_db.so.1".

	Attaching to program: /home/rsc/a.out, process 4724
	Reading symbols from /lib/tls/libpthread.so.0...done.
	[Thread debugging using libthread_db enabled]
	[New Thread 1083603888 (LWP 4724)]
	Loaded symbols for /lib/tls/libpthread.so.0
	Reading symbols from /lib/tls/libc.so.6...done.
	Loaded symbols for /lib/tls/libc.so.6
	Reading symbols from /lib/ld-linux.so.2...done.
	Loaded symbols for /lib/ld-linux.so.2
	Reading symbols from /lib/libgcc_s.so.1...done.
	Loaded symbols for /lib/libgcc_s.so.1
	0x400dc5dc in nanosleep () from /lib/tls/libc.so.6
	(gdb) quit
	The program is running.  Quit anyway (and detach it)? (y or n) y
	Detaching from program: /home/rsc/a.out, process 4724
	$

But that's stupid.  The process known as 4723 has not
exited, and it should be debuggable, even though the
pthread known as 4723 _has_ exited.

Russ

Re: [9fans] off-topic linux threading bug complaint

[boyd, rounin]

> On the other hand, if I guess the process/thread id of the
> second pthread, then I can still look at that:
>
> $ gdb a.out 4724

ahh, when i was doing some driver hacking for some
PCI based Xilinx FPGA driver i had to learn a buncha
stuff about the process structure and all sorts of horrible
mire.

from what you describe, it is consistant with the way linux
does threads.  lunix threads are 'processes' and the thread
is therefore a new 'process'.

my other guess is that their tools are not in sync with the kernel.

does linux have crash(8)?  that might shed some more light.
--
MGRS 31U DQ 52572 12604

Re: [9fans] off-topic linux threading bug complaint

[bs]

Works for me...
[bs@rebox temp]$ uname -a
Linux rebox 2.4.20-31.9 #1 Tue Apr 13 17:41:45 EDT 2004 i686 i686 i386
GNU/Linux
[bs@rebox temp]$ cat /etc/redhat-release
Red Hat Linux release 9 (Shrike)
[bs@rebox temp]$ gcc x.c -lpthread
[bs@rebox temp]$ ./a.out &
[1] 3909
[bs@rebox temp]$ ps -ef|grep a.out
bs        3909  2249  0 14:13 pts/0    00:00:00 ./a.out
bs        3928  2281  0 14:14 pts/1    00:00:00 grep a.out
[bs@rebox temp]$ gdb a.out 3909
GNU gdb Red Hat Linux (5.3post-0.20021129.18rh)
Copyright 2003 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...
Attaching to program: /home/bs/temp/a.out, process 3909
Reading symbols from /lib/libpthread.so.0...done.
[New Thread 32769 (LWP 3911)]
[New Thread 16386 (LWP 3913)]
Loaded symbols for /lib/libpthread.so.0
Reading symbols from /lib/libc.so.6...done.
Loaded symbols for /lib/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
0x4002ae0d in __pthread_sigsuspend () from /lib/libpthread.so.0
(gdb) where
#0  0x4002ae0d in __pthread_sigsuspend () from /lib/libpthread.so.0
#1  0x4002a69d in __pthread_wait_for_restart_signal ()
    from /lib/libpthread.so.0
#2  0x40027506 in __pthread_do_exit () from /lib/libpthread.so.0
#3  0x40027398 in pthread_exit () from /lib/libpthread.so.0
#4  0x0804845c in main ()
#5  0x4008862d in __libc_start_main () from /lib/libc.so.6
(gdb) q
The program is running.  Quit anyway (and detach it)? (y or n) y
Detaching from program: /home/bs/temp/a.out, process 3909
[bs@rebox temp]$

> I'm sorry for the off-topic post, but I have no idea
> where to send this and I suspect there are still Linux
> experts lurking here who do.  Please feel free to forward.
>
> 	$ uname -a
> 	Linux t23 2.6.8-1-686 #1 Thu Oct 7 03:15:25 EDT 2004 i686 GNU/Linux
> 	$
>
> This program creates a second pthread and then
> lets the first pthread exit:
>
> 	$ cat x.c
> 	#include <stdio.h>
> 	#include <unistd.h>
> 	#include <pthread.h>
>
> 	void*
> 	sleeper(void *a)
> 	{
> 		for(;;)
> 			sleep(1000);
> 	}
>
> 	int
> 	main(void)
> 	{
> 		pthread_t id;
>
> 		pthread_create(&id, 0, sleeper, 0);
> 		pthread_exit(0);
> 	}
> 	$ gcc x.c -lpthread
> 	$ ./a.out &
> 	[1] 4723
> 	$
>
> Because the first pthread has exited, the ps listing
> makes it look like the process is a zombie, even
> though it's still running:
>
> 	$ ps axwwu|grep a.out |grep -v grep
> 	rsc       4723  0.0  0.0     0    0 pts/2    Z    13:29   0:00
> [a.out] <defunct>
> 	$
>
> The -T and -L flags to ps don't help:
>
> 	$ ps -T axwwu|grep a.out |grep -v grep
> 	$ ps -L axwwu|grep a.out |grep -v grep
> 	$
>
> It's not ps just misdisplaying information: gdb cannot attach either.
>
> 	$ gdb a.out 4723
> 	GNU gdb 6.1-debian
> 	Copyright 2004 Free Software Foundation, Inc.
> 	GDB is free software, covered by the GNU General Public License, and you are
> 	welcome to change it and/or distribute copies of it under certain conditions.
> 	Type "show copying" to see the conditions.
> 	There is absolutely no warranty for GDB.  Type "show warranty" for details.
> 	This GDB was configured as "i386-linux"...Using host libthread_db
> library "/lib/tls/libthread_db.so.1".
>
> 	Attaching to program: /home/rsc/a.out, process 4723
> 	ptrace: Operation not permitted.
> 	/home/rsc/4723: No such file or directory.
> 	(gdb) quit
> 	$
>
> On the other hand, if I guess the process/thread id of the
> second pthread, then I can still look at that:
>
> 	$ gdb a.out 4724
> 	GNU gdb 6.1-debian
> 	Copyright 2004 Free Software Foundation, Inc.
> 	GDB is free software, covered by the GNU General Public License, and you are
> 	welcome to change it and/or distribute copies of it under certain conditions.
> 	Type "show copying" to see the conditions.
> 	There is absolutely no warranty for GDB.  Type "show warranty" for details.
> 	This GDB was configured as "i386-linux"...Using host libthread_db
> library "/lib/tls/libthread_db.so.1".
>
> 	Attaching to program: /home/rsc/a.out, process 4724
> 	Reading symbols from /lib/tls/libpthread.so.0...done.
> 	[Thread debugging using libthread_db enabled]
> 	[New Thread 1083603888 (LWP 4724)]
> 	Loaded symbols for /lib/tls/libpthread.so.0
> 	Reading symbols from /lib/tls/libc.so.6...done.
> 	Loaded symbols for /lib/tls/libc.so.6
> 	Reading symbols from /lib/ld-linux.so.2...done.
> 	Loaded symbols for /lib/ld-linux.so.2
> 	Reading symbols from /lib/libgcc_s.so.1...done.
> 	Loaded symbols for /lib/libgcc_s.so.1
> 	0x400dc5dc in nanosleep () from /lib/tls/libc.so.6
> 	(gdb) quit
> 	The program is running.  Quit anyway (and detach it)? (y or n) y
> 	Detaching from program: /home/rsc/a.out, process 4724
> 	$
>
> But that's stupid.  The process known as 4723 has not
> exited, and it should be debuggable, even though the
> pthread known as 4723 _has_ exited.
>
> Russ
>

RE: [9fans] off-topic linux threading bug complaint

[Nigel Roles]

9fans-bounces+ngr=9fs.org@cse.psu.edu wrote:
> Works for me...
> [bs@rebox temp]$ uname -a
> Linux rebox 2.4.20-31.9 #1 Tue Apr 13 17:41:45 EDT 2004 i686 i686

Russ has 2.6. I would say that makes all the difference in the
world, including version of glibc.

Re: [9fans] off-topic linux threading bug complaint

[bs]

Nigel Roles wrote:
> 9fans-bounces+ngr=9fs.org@cse.psu.edu wrote:
>
>>Works for me...
>>[bs@rebox temp]$ uname -a
>>Linux rebox 2.4.20-31.9 #1 Tue Apr 13 17:41:45 EDT 2004 i686 i686
>
>
> Russ has 2.6. I would say that makes all the difference in the
> world, including version of glibc.
and I use the NPTL for RedHat Linux.

Re: [9fans] off-topic linux threading bug complaint

[Latchesar Ionkov]

The bug was reported to LKML back in October:

	http://www.ussg.iu.edu/hypermail/linux/kernel/0410.0/1141.html

	Lucho

On Thu, Feb 10, 2005 at 01:35:46PM -0500, Russ Cox said:
> I'm sorry for the off-topic post, but I have no idea
> where to send this and I suspect there are still Linux
> experts lurking here who do.  Please feel free to forward.
>
> 	$ uname -a
> 	Linux t23 2.6.8-1-686 #1 Thu Oct 7 03:15:25 EDT 2004 i686 GNU/Linux
> 	$
>
> This program creates a second pthread and then
> lets the first pthread exit:
>
> 	$ cat x.c
> 	#include <stdio.h>
> 	#include <unistd.h>
> 	#include <pthread.h>
>
> 	void*
> 	sleeper(void *a)
> 	{
> 		for(;;)
> 			sleep(1000);
> 	}
>
> 	int
> 	main(void)
> 	{
> 		pthread_t id;
>
> 		pthread_create(&id, 0, sleeper, 0);
> 		pthread_exit(0);
> 	}
> 	$ gcc x.c -lpthread
> 	$ ./a.out &
> 	[1] 4723
> 	$
>
> Because the first pthread has exited, the ps listing
> makes it look like the process is a zombie, even
> though it's still running:
>
> 	$ ps axwwu|grep a.out |grep -v grep
> 	rsc       4723  0.0  0.0     0    0 pts/2    Z    13:29   0:00
> [a.out] <defunct>
> 	$
>
> The -T and -L flags to ps don't help:
>
> 	$ ps -T axwwu|grep a.out |grep -v grep
> 	$ ps -L axwwu|grep a.out |grep -v grep
> 	$
>
> It's not ps just misdisplaying information: gdb cannot attach either.
>
> 	$ gdb a.out 4723
> 	GNU gdb 6.1-debian
> 	Copyright 2004 Free Software Foundation, Inc.
> 	GDB is free software, covered by the GNU General Public License, and you are
> 	welcome to change it and/or distribute copies of it under certain conditions.
> 	Type "show copying" to see the conditions.
> 	There is absolutely no warranty for GDB.  Type "show warranty" for details.
> 	This GDB was configured as "i386-linux"...Using host libthread_db
> library "/lib/tls/libthread_db.so.1".
>
> 	Attaching to program: /home/rsc/a.out, process 4723
> 	ptrace: Operation not permitted.
> 	/home/rsc/4723: No such file or directory.
> 	(gdb) quit
> 	$
>
> On the other hand, if I guess the process/thread id of the
> second pthread, then I can still look at that:
>
> 	$ gdb a.out 4724
> 	GNU gdb 6.1-debian
> 	Copyright 2004 Free Software Foundation, Inc.
> 	GDB is free software, covered by the GNU General Public License, and you are
> 	welcome to change it and/or distribute copies of it under certain conditions.
> 	Type "show copying" to see the conditions.
> 	There is absolutely no warranty for GDB.  Type "show warranty" for details.
> 	This GDB was configured as "i386-linux"...Using host libthread_db
> library "/lib/tls/libthread_db.so.1".
>
> 	Attaching to program: /home/rsc/a.out, process 4724
> 	Reading symbols from /lib/tls/libpthread.so.0...done.
> 	[Thread debugging using libthread_db enabled]
> 	[New Thread 1083603888 (LWP 4724)]
> 	Loaded symbols for /lib/tls/libpthread.so.0
> 	Reading symbols from /lib/tls/libc.so.6...done.
> 	Loaded symbols for /lib/tls/libc.so.6
> 	Reading symbols from /lib/ld-linux.so.2...done.
> 	Loaded symbols for /lib/ld-linux.so.2
> 	Reading symbols from /lib/libgcc_s.so.1...done.
> 	Loaded symbols for /lib/libgcc_s.so.1
> 	0x400dc5dc in nanosleep () from /lib/tls/libc.so.6
> 	(gdb) quit
> 	The program is running.  Quit anyway (and detach it)? (y or n) y
> 	Detaching from program: /home/rsc/a.out, process 4724
> 	$
>
> But that's stupid.  The process known as 4723 has not
> exited, and it should be debuggable, even though the
> pthread known as 4723 _has_ exited.
>
> Russ

Re: [9fans] off-topic linux threading bug complaint

[David Leimbach]

I tried it on solaris:


sultan% ps -p 1362 -o "lwp user pid ppid taskid pcpu pmem fname"
   LWP     USER   PID  PPID TASKID %CPU %MEM COMMAND
     1     dave  1362  1354    59    -  0.1 <defunct>
     2     dave  1362  1354    59  0.0  0.1 x


2 Light Weight processes, one of which is defunct.  Sounds similar to
the linux behavior.

Dave

On Thu, 10 Feb 2005 13:35:46 -0500, Russ Cox <russcox@gmail.com> wrote:
> I'm sorry for the off-topic post, but I have no idea
> where to send this and I suspect there are still Linux
> experts lurking here who do.  Please feel free to forward.
>
>         $ uname -a
>         Linux t23 2.6.8-1-686 #1 Thu Oct 7 03:15:25 EDT 2004 i686 GNU/Linux
>         $
>
> This program creates a second pthread and then
> lets the first pthread exit:
>
>         $ cat x.c
>         #include <stdio.h>
>         #include <unistd.h>
>         #include <pthread.h>
>
>         void*
>         sleeper(void *a)
>         {
>                 for(;;)
>                         sleep(1000);
>         }
>
>         int
>         main(void)
>         {
>                 pthread_t id;
>
>                 pthread_create(&id, 0, sleeper, 0);
>                 pthread_exit(0);
>         }
>         $ gcc x.c -lpthread
>         $ ./a.out &
>         [1] 4723
>         $
>
> Because the first pthread has exited, the ps listing
> makes it look like the process is a zombie, even
> though it's still running:
>
>         $ ps axwwu|grep a.out |grep -v grep
>         rsc       4723  0.0  0.0     0    0 pts/2    Z    13:29   0:00
> [a.out] <defunct>
>         $
>
> The -T and -L flags to ps don't help:
>
>         $ ps -T axwwu|grep a.out |grep -v grep
>         $ ps -L axwwu|grep a.out |grep -v grep
>         $
>
> It's not ps just misdisplaying information: gdb cannot attach either.
>
>         $ gdb a.out 4723
>         GNU gdb 6.1-debian
>         Copyright 2004 Free Software Foundation, Inc.
>         GDB is free software, covered by the GNU General Public License, and you are
>         welcome to change it and/or distribute copies of it under certain conditions.
>         Type "show copying" to see the conditions.
>         There is absolutely no warranty for GDB.  Type "show warranty" for details.
>         This GDB was configured as "i386-linux"...Using host libthread_db
> library "/lib/tls/libthread_db.so.1".
>
>         Attaching to program: /home/rsc/a.out, process 4723
>         ptrace: Operation not permitted.
>         /home/rsc/4723: No such file or directory.
>         (gdb) quit
>         $
>
> On the other hand, if I guess the process/thread id of the
> second pthread, then I can still look at that:
>
>         $ gdb a.out 4724
>         GNU gdb 6.1-debian
>         Copyright 2004 Free Software Foundation, Inc.
>         GDB is free software, covered by the GNU General Public License, and you are
>         welcome to change it and/or distribute copies of it under certain conditions.
>         Type "show copying" to see the conditions.
>         There is absolutely no warranty for GDB.  Type "show warranty" for details.
>         This GDB was configured as "i386-linux"...Using host libthread_db
> library "/lib/tls/libthread_db.so.1".
>
>         Attaching to program: /home/rsc/a.out, process 4724
>         Reading symbols from /lib/tls/libpthread.so.0...done.
>         [Thread debugging using libthread_db enabled]
>         [New Thread 1083603888 (LWP 4724)]
>         Loaded symbols for /lib/tls/libpthread.so.0
>         Reading symbols from /lib/tls/libc.so.6...done.
>         Loaded symbols for /lib/tls/libc.so.6
>         Reading symbols from /lib/ld-linux.so.2...done.
>         Loaded symbols for /lib/ld-linux.so.2
>         Reading symbols from /lib/libgcc_s.so.1...done.
>         Loaded symbols for /lib/libgcc_s.so.1
>         0x400dc5dc in nanosleep () from /lib/tls/libc.so.6
>         (gdb) quit
>         The program is running.  Quit anyway (and detach it)? (y or n) y
>         Detaching from program: /home/rsc/a.out, process 4724
>         $
>
> But that's stupid.  The process known as 4723 has not
> exited, and it should be debuggable, even though the
> pthread known as 4723 _has_ exited.
>
> Russ
>

Re: [9fans] off-topic linux threading bug complaint

[Russ Cox]

> I tried it on solaris:
>
> sultan% ps -p 1362 -o "lwp user pid ppid taskid pcpu pmem fname"
>    LWP     USER   PID  PPID TASKID %CPU %MEM COMMAND
>      1     dave  1362  1354    59    -  0.1 <defunct>
>      2     dave  1362  1354    59  0.0  0.1 x
>
> 2 Light Weight processes, one of which is defunct.  Sounds similar to
> the linux behavior.

The Linux behavior -- that ps is misleading and gdb is unusable unless
you have some magic guess as to some other "pid" -- is broken.

SunOS is not broken this way: ps shows a non-zombie
process, and the debugger can attach to it.

% ./a.out &
[3] 12058
% ps -aA -o 'user,pid,stime,time,rss,s,args'  | grep a.out
     rsc 12058 15:55:40        0:00  920 S ./a.out
% adb a.out
:A 12058
$c
libc.so.1`_libc_nanosleep+4(3e8, ff350234, 0, 0, 0, 0)
sleeper+0xc(0, 0, 0, 0, 0, 0)
libthread.so.1`_lwp_start(0, 0, 0, 0, 0, 0)
^D
%

Russ

Re: [9fans] off-topic linux threading bug complaint

[Taj Khattra]

i think the process' task_struct->mm may already have been deallocated
which is why ptrace() fails for gdb?

% grep State /proc/5962/status /proc/5963/status
/proc/5962/status:State:	Z (zombie)
/proc/5963/status:State:	S (sleeping)

% cat /proc/5962/maps
%

Re: [9fans] off-topic linux threading bug complaint

[Ronald G. Minnich]

The new thread is going to do a clone, which gets you a new pid.

So you'll see old pid as zombie, new pid as new proc.

That's what I thought, and I just tested this on my machine and it sure
seems to work that way.

ron

Re: [9fans] off-topic linux threading bug complaint

[Ronald G. Minnich]

On Thu, 10 Feb 2005, Latchesar Ionkov wrote:

> The bug was reported to LKML back in October:
>
> 	http://www.ussg.iu.edu/hypermail/linux/kernel/0410.0/1141.html

what's weird for me is that the process really does exit. It does not hang
around. But I'm on 2.6.10. Actually, the child doesn't hang around at all
either.

Can someone remind me: the 'thread group leader', if it exits: doesn't it
take everyone with it? Because on my box the parent does an exit_group and
that does indeed  take the child out.

ron