Re: duplicate processes

Re: duplicate processes

[Jussi Ramo]

 >What evidence do you have that the processes are started directly by 
init? Remember that a process will be inherited by init if its direct 
 >parent dies.

No evidence. Just looked at the parent process. So you suggest that the 
"runsv ndb_mgmd" dies and the ndb_mgmd is inherited by init. Then "runsv 
ndb_mgmd" is respawned by runsvdir (?) and that starts another ndb_mgmd. 
This makes sense to me but now the question is why runsv first dies once 
for certain processes.




 >> So duplicate process will be generated and system becomes unstable. 
Both of those processes (the one started by init and the one started by 
runsv) react on sv command.
 >React in what ways?

Do not know if this brings any extra information but if I have first the 
following ndb_mgmd processes (one of them badly as child of init) :

root      1950  1945  0 07:04 ?        00:00:00 runsv ndb_mgmd
ais       1963     1  0 07:04 ?        00:00:00 /opt/SGC/bin/ndb_mgmd -f 
/opt/SGC/etc/ndbconfig.ini
ais       2276  1950  2 07:05 ?        00:00:00 /opt/SGC/bin/ndb_mgmd -f 
/opt/SGC/etc/ndbconfig.ini

Then I do like:

blade_0_7:~ # /opt/SGC/bin/sv down /var/services/ndb_mgmd/

and the other "right" ndb_mgmd disappears:

root      1950  1945  0 07:04 ?        00:00:00 runsv ndb_mgmd
ais       1963     1  0 07:04 ?        00:00:00 /opt/SGC/bin/ndb_mgmd -f 
/opt/SGC/etc/ndbconfig.ini

I then kill the "wrong" ndb_mgmd

blade_0_7:~ # kill -9 1963
root      1950  1945  0 07:04 ?        00:00:00 runsv ndb_mgmd

And when ndb_mgmd is put "up" there are again those two processes:

blade_0_7:~ # /opt/SGC/bin/sv up /var/services/ndb_mgmd/

root      1950  1945  0 07:04 ?        00:00:00 runsv ndb_mgmd
ais       2805     1  0 07:08 ?        00:00:00 /opt/SGC/bin/ndb_mgmd -f 
/opt/SGC/etc/ndbconfig.ini
ais       2837  1950  4 07:08 ?        00:00:00 /opt/SGC/bin/ndb_mgmd -f 
/opt/SGC/etc/ndbconfig.ini


 >> For example below: aisexec works properly but ndb_mgmd causes problems.
 >Google tells me that ndb_mgmd is part of mysql. mysql is known to be 
badly designed wrt co-operating with runit/daemontools.

This is a good piece of information. Other 3pp product that causes similar 
troubles as mysql to me is EMANATE snmp agent, when using with runit.


 >> There are no major diffrencies in run scripts but file paths or so. 
Both of the run scrips are quite simple.
 >What are they?

#!/bin/sh
# Source environment settings
. /opt/SGC/etc/sgcenv
exec >> $LOG_FILE 2>&1
aisexec_BIN=/opt/SGC/bin/aisexec
exec $aisexec_BIN


#!/bin/sh
# Source environment settings
. /opt/SGC/etc/sgcenv
exec >> $LOG_FILE 2>&1
ndb_mgmd_BIN=/opt/SGC/bin/ndb_mgmd
exec $ndb_mgmd_BIN -f /opt/SGC/etc/ndbconfig.ini



 >> The other ndb_mgmd is restarted frequently by runsv because of the 
same process is started directly by init for some reason.
 >Again, what evidence do you have that there is any process started 
directly by init? Even if so, why would runsv restart the process it is 
managing? I expect that the process runsv is monitoring is exiting, and 
that is why runsv is starting a new process.

Right. The process runsv is monitoring is exiting because the port it 
tries to use is reserved by the extra process (now hopefully correct 
phrasing:) whose parent is init.

Thanks and regards,
Jussi

P.S. I now tried to create a new message for a mailing thread, first time 
I just replied on an existing thread changing the subject. Sorry for that!

Re: duplicate processes

[Charlie Brady]

On Tue, 27 Sep 2005, Jussi Ramo wrote:

>> What evidence do you have that the processes are started directly by 
> init? Remember that a process will be inherited by init if its direct 
>> parent dies.
>
> No evidence. Just looked at the parent process.

I'm pretty sure that's misleading.

> So you suggest that the "runsv ndb_mgmd" dies and the ndb_mgmd is 
> inherited by init. Then "runsv ndb_mgmd" is respawned by runsvdir (?) 
> and that starts another ndb_mgmd. This makes sense to me but now the 
> question is why runsv first dies once for certain processes.

No, I didn't mean to suggest that runsv dies. I expect that ndb_mgmd has 
forked, and the parent died. Perhaps it was designed to do that, to 
"daemonise" the child. You will have to find some way to prevent that from 
happening.

>>> So duplicate process will be generated and system becomes unstable. 
> Both of those processes (the one started by init and the one started by 
> runsv) react on sv command.
>> React in what ways?
>
> Do not know if this brings any extra information but if I have first the 
> following ndb_mgmd processes (one of them badly as child of init) :
>
> root      1950  1945  0 07:04 ?        00:00:00 runsv ndb_mgmd
> ais       1963     1  0 07:04 ?        00:00:00 /opt/SGC/bin/ndb_mgmd -f 
> /opt/SGC/etc/ndbconfig.ini
> ais       2276  1950  2 07:05 ?        00:00:00 /opt/SGC/bin/ndb_mgmd -f 
> /opt/SGC/etc/ndbconfig.ini
>
> Then I do like:
>
> blade_0_7:~ # /opt/SGC/bin/sv down /var/services/ndb_mgmd/
>
> and the other "right" ndb_mgmd disappears:
>
> root      1950  1945  0 07:04 ?        00:00:00 runsv ndb_mgmd
> ais       1963     1  0 07:04 ?        00:00:00 /opt/SGC/bin/ndb_mgmd -f 
> /opt/SGC/etc/ndbconfig.ini

OK.

> I then kill the "wrong" ndb_mgmd
>
> blade_0_7:~ # kill -9 1963
> root      1950  1945  0 07:04 ?        00:00:00 runsv ndb_mgmd

OK, but you shouldn't be using -9 unless there is no alternative.

> And when ndb_mgmd is put "up" there are again those two processes:
>
> blade_0_7:~ # /opt/SGC/bin/sv up /var/services/ndb_mgmd/
>
> root      1950  1945  0 07:04 ?        00:00:00 runsv ndb_mgmd
> ais       2805     1  0 07:08 ?        00:00:00 /opt/SGC/bin/ndb_mgmd -f 
> /opt/SGC/etc/ndbconfig.ini
> ais       2837  1950  4 07:08 ?        00:00:00 /opt/SGC/bin/ndb_mgmd -f 
> /opt/SGC/etc/ndbconfig.ini

OK. Assuming that you have pids allocated sequentially, look how many 
intervening processes there are between the two invocations.

>>> The other ndb_mgmd is restarted frequently by runsv because of the 
> same process is started directly by init for some reason.
>> Again, what evidence do you have that there is any process started 
> directly by init? Even if so, why would runsv restart the process it is 
> managing? I expect that the process runsv is monitoring is exiting, and that 
> is why runsv is starting a new process.
>
> Right. The process runsv is monitoring is exiting because the port it tries 
> to use is reserved by the extra process (now hopefully correct phrasing:) 
> whose parent is init.

Not quite. The process runsv is monitoring exits when told to by runsv. 
You showed us that. The other ndb_mgmd process however doesn't exit, and 
runsv will therefore not be able to start a new process which is able to 
keep running.

Does ndb_mgmd create a pid file? If so, what is its content?

Re: duplicate processes

[Jussi Ramo]

Thanks Charlie!

Charlie Brady wrote:
> 
> No, I didn't mean to suggest that runsv dies. I expect that ndb_mgmd has 
> forked, and the parent died. Perhaps it was designed to do that, to 
> "daemonise" the child. You will have to find some way to prevent that 
> from happening.
> 

That was it!! There is an option --nodeamon, and when using that 
everything works fine. (same goes for snmpdm -d)

 >mysql is known to be badly designed wrt co-operating with runit/daemontools.

Is it just because of this daemonizing issue, or is there other reasons 
not to use runit with mysql? If not, I could send those runs scripts to 
run script collection page when they are mature. I am using SUSE 9.1 and SLES.

Regards,
Jussi

Re: duplicate processes

[Charlie Brady]

On Mon, 26 Sep 2005, Jussi Ramo wrote:

> Some processes are still started first directly by init even if I try to use 
> runits supervision only.

What evidence do you have that the processes are started directly by init? 
Remember that a process will be inherited by init if its direct parent 
dies.

> So duplicate process will be generated and system 
> becomes unstable. Both of those processes (the one started by init and the 
> one started by runsv) react on sv command.

React in what ways?

> For example below: aisexec works properly but ndb_mgmd causes problems.

Google tells me that ndb_mgmd is part of mysql. mysql is known to be badly 
designed wrt co-operating with runit/daemontools.

> There are no major diffrencies in run scripts but file paths or so. Both 
> of the run scrips are quite simple.

What are they?

> The other ndb_mgmd is restarted frequently by runsv because of the same 
> process is started directly by init for some reason.

Again, what evidence do you have that there is any process started 
directly by init? Even if so, why would runsv restart the process it is 
managing? I expect that the process runsv is monitoring is exiting, and 
that is why runsv is starting a new process.

---
Charlie

duplicate processes

[Jussi Ramo]

Hello,

I am new user for runit (sv part) and it would be highly appreciated if 
someone could give me
a hint for troubleshooting the following problem:

Some processes are still started first directly by init even if I try to 
use runits supervision only. So duplicate process will be generated and 
system becomes unstable. Both of those processes (the one started by init 
and the one started by runsv) react on sv command.

For example below: aisexec works properly but ndb_mgmd causes problems. 
There are no
major diffrencies in run scripts but file paths or so. Both of the run 
scrips are quite simple.

root         1     0  1 14:40 ?        00:00:05 init [3]
root      1946     1  0 14:40 ?        00:00:00 runsvdir -P /var/services 
log: ...................
root      1950  1946  0 14:40 ?        00:00:00 runsv aisexec
root      1951  1946  0 14:40 ?        00:00:00 runsv ndb_mgmd
ais       1955  1950  0 14:40 ?        00:00:00 /opt/SGC/bin/aisexec
ais       1967     1  0 14:40 ?        00:00:00 /opt/SGC/bin/ndb_mgmd -f 
/opt/SGC/etc/ndbconfig.ini
ais       3181  1951  3 14:46 ?        00:00:00 /opt/SGC/bin/ndb_mgmd -f 
/opt/SGC/etc/ndbconfig.ini

The other ndb_mgmd is restarted frequently by runsv because of the same 
process is started directly by init for some reason.

I have double checked that:
-There is nothing more added in inittab but runsvdir
-There are no files under init.d that might start ndb_mgmd

Kind regards,

Jussi