[9fans] Quick question on stopping a process that waits for IO

[9fans] Quick question on stopping a process that waits for IO

[Roman Shaposhnik]

Guys,

when somebody tries to stop a process that is waiting for the IO the
process
doesn't get transferred to a Stopped state immediately but only when
the scheduler sees it for the first time. This leads to a process
writing to
a /proc/n/ctl being put in a Stopwait state which is a bit inconvenient.

Is there any way I can poke the target process so that it gets attention
from the scheduler an can be put in a Stopped state?

Thanks,
Roman.

Re: [9fans] Quick question on stopping a process that waits for IO

[Roman V. Shaposhnik]

On Sun, 2008-11-02 at 21:55 -0800, Roman Shaposhnik wrote:
> Is there any way I can poke the target process so that it gets attention
> from the scheduler an can be put in a Stopped state?

I know, I know we all don't like those guys who talk to themselves
on mailing lists replying to their own emails, but since there were
no takers here's what I cooked up so far:
   echo stop > /proc/n/ctl &
   echo ping > /proc/n/note
   cat /proc/n/note > /dev/null

That of course doesn't quite work either since once I resume 'n'
by issuing
    echo start > /proc/n/ctl
the process fails with:
    error reading <stdin>: interrupted

So the question remains -- what is the proper way of putting a process
that waits for an IO into a Stopped state?

Sorry that I keep bothering the list, but I can't really think
of anything here :-(

Thanks,
Roman.

Re: [9fans] Quick question on stopping a process that waits for IO

[ron minnich]

On Sun, Nov 2, 2008 at 9:55 PM, Roman Shaposhnik <rvs@sun.com> wrote:
> Guys,
>
> when somebody tries to stop a process that is waiting for the IO the process
> doesn't get transferred to a Stopped state immediately but only when
> the scheduler sees it for the first time. This leads to a process writing to
> a /proc/n/ctl being put in a Stopwait state which is a bit inconvenient.
>
> Is there any way I can poke the target process so that it gets attention
> from the scheduler an can be put in a Stopped state?
>

Hi Roman, what's the bigger picture here? I am trying to understand
... sorry if I'm missing the obvious.

ron

Re: [9fans] Quick question on stopping a process that waits for IO

[erik quanstrom]

>> Is there any way I can poke the target process so that it gets attention
>> from the scheduler an can be put in a Stopped state?
>
> I know, I know we all don't like those guys who talk to themselves
> on mailing lists replying to their own emails, but since there were
> no takers here's what I cooked up so far:
>    echo stop > /proc/n/ctl &
>    echo ping > /proc/n/note
>    cat /proc/n/note > /dev/null
>
> That of course doesn't quite work either since once I resume 'n'
> by issuing
>     echo start > /proc/n/ctl
> the process fails with:
>     error reading <stdin>: interrupted

this is an uncaught error in port/proc.c:/^sleep.  many interruptable
calls to sleep are wrapped by
	while(waserror())
		;
	sleep(...);
	poperror();

but this is a catch-22.  if the i/o resumed and didn't send a note,
it would not be possible to interrupt processes with del with rio's
current convention.

on the other hand, sending a note is the only way to get to procctl
unless one is doing syscall tracing.

oddly procctl says
/*
 *  called splhi() by notify().  See comment in notify for the
 *  reasoning.
 */
but there doesn't seem to be a comment in any version of notify
i can find.

>
> So the question remains -- what is the proper way of putting a process
> that waits for an IO into a Stopped state?

i don't think it's possible without changing the kernel.
but it's a good question, why does it work this way?
obviously one doesn't want to stop a process and kernel work in
the middle of i/o.  there may be locks involved, etc.
but one could easily check for Proc_stopme on syscall
exit.

does anyone remember the thinking that went into this?

- erik

Re: [9fans] Quick question on stopping a process that waits for IO

[Roman V. Shaposhnik]

On Mon, 2008-11-03 at 17:01 -0800, ron minnich wrote:
> On Sun, Nov 2, 2008 at 9:55 PM, Roman Shaposhnik <rvs@sun.com> wrote:
> > when somebody tries to stop a process that is waiting for the IO the process
> > doesn't get transferred to a Stopped state immediately but only when
> > the scheduler sees it for the first time. This leads to a process writing to
> > a /proc/n/ctl being put in a Stopwait state which is a bit inconvenient.
> >
> > Is there any way I can poke the target process so that it gets attention
> > from the scheduler an can be put in a Stopped state?
> >
>
> Hi Roman, what's the bigger picture here? I am trying to understand
> ... sorry if I'm missing the obvious.

I'm glad you've asked ;-) In fact, there's a bigger context and it is
around managing processes run by cpu(1) from the terminal host. I was
planning on writing an email on that subject to this list over the
weekend but I need to amass some level of intelligence in that area
first.

For now, though, the easiest way to illustrate my problem would be to
imagine a situation where you want to write some data into a Data
segment of the process that is waiting for the IO. Now, waiting for
the IO is almost as good as state as Stopped (the process is stuck in
kernel waiting to be put back on the runqueue) so one would expect the
transition between these 2 states to be almost trivial. Literally, all
I'm asking is -- "dear kernel, please don't advance this process even
if you otherwise can". All I need is a frozen state so that I can
manipulate some data. Yet, I can't figure out how to do that:
  term% ps | grep 310
  glenda          310    0:00   0:00       24K Pread  cat

  term% cat /proc/310/seg*
  Stack     0f000000 10000000    0
  Text   R  00001000 00006000    0
  Data      00006000 00007000    0
  Bss       00007000 00007000    0

  term% dd -if /dev/zero -bs 1 -count 1 -of /proc/310/mem -seek 6100
  write: bad process or channel control request
  1+0 records in
  0+0 records out

Of course, the process is Spleeping, so we need to stop it:

  term% echo stop > /proc/310/ctl

This hangs, so I'll provide some input to 'cat' so that it becomes
Stopped and I can do:
  term% dd -if /dev/zero -bs 1 -count 1 -of /proc/310/mem -seek 6100
  1+0 records in
  1+0 records out

But! I'm really stuck if I can't give that input to something
that is Pread.

How are debugging/analysis tools supposed to deal with a situation
like that? I would imagine this to be a pretty common situation
for them, right?

Thanks,
Roman.

Re: [9fans] Quick question on stopping a process that waits for IO

[erik quanstrom]

> I'm glad you've asked ;-) In fact, there's a bigger context and it is
> around managing processes run by cpu(1) from the terminal host. I was
> planning on writing an email on that subject to this list over the
> weekend but I need to amass some level of intelligence in that area
> first.
>
> For now, though, the easiest way to illustrate my problem would be to
> imagine a situation where you want to write some data into a Data
> segment of the process that is waiting for the IO. Now, waiting for
> the IO is almost as good as state as Stopped (the process is stuck in
> kernel waiting to be put back on the runqueue) so one would expect the
> transition between these 2 states to be almost trivial. Literally, all
> I'm asking is -- "dear kernel, please don't advance this process even
> if you otherwise can". All I need is a frozen state so that I can
> manipulate some data. Yet, I can't figure out how to do that:

i don't think the kernel has this level of control.

let's suppose that we have a process that gets a stop message
that's doing i/o.  let's suppose that it's doing io to a particularly
cranky device with lots of neat locks that really hates getting
interrupted.  i don't know, something esoteric — say, ata.  are you
really so sure that you can do what you want to this process
when ata io is going on?

i don't think this is safe unless all copies back to user space are
done by some kernel function that does all the proper checking.
and you have some way of unlocking safely from any part of the
i/o code.

i sure would like to hear from someone who knows more about
what went into the design of notes.  i don't think i've thought of
all the scary dark corners.

- erik

Re: [9fans] Quick question on stopping a process that waits for IO

[erik quanstrom]

> I'm asking is -- "dear kernel, please don't advance this process even
> if you otherwise can". All I need is a frozen state so that I can

not so easy on a multiprocessor.  (unless you turn all but one
processor off.)

- erik

Re: [9fans] Quick question on stopping a process that waits for IO

[Roman Shaposhnik]

On Nov 4, 2008, at 8:01 PM, erik quanstrom wrote:

>> I'm asking is -- "dear kernel, please don't advance this process even
>> if you otherwise can". All I need is a frozen state so that I can
>
> not so easy on a multiprocessor.  (unless you turn all but one
> processor off.)

Hm. May be its getting late, but I can't quite see why that would
be the case (or may be I didn't quite communicate the intent
in my plea to the kernel ;-))

The target process is *already* waiting for the IO stuck inside the
kernel. It is not on a runqueue, not it is considered to be places
there.
All of the user-level data structures are frozen and
not changing. All I'm asking is that if, and only if, the process
gets attention from the scheduler (for whatever reason) it doesn't
get placed on a runque.

I don't quite see how MP would matter in this case.

Thanks,
Roman.

Re: [9fans] Quick question on stopping a process that waits for IO

[Roman Shaposhnik]

On Nov 4, 2008, at 8:00 PM, erik quanstrom wrote:
> i don't think the kernel has this level of control.
>
> let's suppose that we have a process that gets a stop message
> that's doing i/o.  let's suppose that it's doing io to a particularly
> cranky device with lots of neat locks that really hates getting
> interrupted.  i don't know, something esoteric — say, ata.  are you
> really so sure that you can do what you want to this process
> when ata io is going on?

Well, may be not. But I would like my right to shoot myself in the
foot not to be revoked ;-)

> i sure would like to hear from someone who knows more about
> what went into the design of notes.  i don't think i've thought of
> all the scary dark corners.


Same here.

Now, There are  two issues on the table right now: the first
one has everything to do with notes, and I really would like to
better understand that area myself. Hence I'm all  ears.

The other issue (how to manipulate the running process) has
less to do with notes per se. As I said, I do appreciate the reasoning
that a process has to be in a frozen state in order for writes
on /proc/n/mem to succeed. By a frozen state I still mean the
kind of a state that would prevent a scheduler to place it
on a runqueue. Stopped is one such state, but may be it
shouldn't be the only one.

Would it be completely unreasonable for devproc.c to also
allow writes for the processes which are in process of being
stopped?

Thanks,
Roman.

Re: [9fans] Quick question on stopping a process that waits for IO

[erik quanstrom]

>>> I'm asking is -- "dear kernel, please don't advance this process even
>>> if you otherwise can". All I need is a frozen state so that I can
>>
>> not so easy on a multiprocessor.  (unless you turn all but one
>> processor off.)
>
> Hm. May be its getting late, but I can't quite see why that would
> be the case (or may be I didn't quite communicate the intent
> in my plea to the kernel ;-))
>
> The target process is *already* waiting for the IO stuck inside the
> kernel. It is not on a runqueue, not it is considered to be places
> there.

since procwrite doesn't acquire anything other than the debug lock,
how do you know?  the proc could start up again before you notice.

you could start grabbing locks, stealing stuff off queues or adding
if(up->procctl) statements here and there, but i have a strong feeling
thair be dragons in that direction.

the place to catch processes is on syscall entry and/or exit.  stop
messages are only checked on syscall entry.

i have a feeling that you want something other than notes for
your problem.  cpu arranges that del works.

- erik

Re: [9fans] Quick question on stopping a process that waits for IO

[Nathaniel W Filardo]

[-- Attachment #1: Type: text/plain, Size: 2778 bytes --]

On Sun, Nov 02, 2008 at 09:55:16PM -0800, Roman Shaposhnik wrote:
> Guys,
>
> when somebody tries to stop a process that is waiting for the IO the  
> process
> doesn't get transferred to a Stopped state immediately but only when
> the scheduler sees it for the first time. This leads to a process  
> writing to
> a /proc/n/ctl being put in a Stopwait state which is a bit inconvenient.
>
> Is there any way I can poke the target process so that it gets attention
> from the scheduler an can be put in a Stopped state?
>
> Thanks,
> Roman.

Thinking aloud, and looking mostly at the port/ and pc/ code:

port/devproc.c:/^procstopwait will be called when we write "stop" into
/proc/n/ctl.  Modulo details, this function will:
  qlock p->debug
  mark us as p's debugger
  set p->procctl to Proc_stopme
  drop p->debug
  wait until p->state == Stopped by using procstopped()
  return

If a process is stopped on I/O, that wait is going to take "a while" (until
I/O completes).  Eventually, however, it will return into
pc/trap.c:/^syscall and will eventually run the code:

>  if(scallnr!=RFORK && (up->procctl || up->nnote)){
>          splhi();
>          notify(ureg);
>  }

whereupon we expect [1] that it sees that up->procctl is nonzero and the
process will put itself to sleep inside notify(), awake the waiter above,
and everything will be fine.

So it seems like it wouldn't be very hard to define a "soonstop" command
which considered processes blocked on I/O "as good as stopped" since there
are two outcomes:
  After we set up->procctl, the target process returned to userland and will
  sleep on the next trap they take or syscall they make.  In this case, we
  should set ourselves as p's debugger and wait as we have a bounded wait:
  exactly one time quantum at most.

  After we set up->procctl, the process is still blocked (or blocked again)
  on I/O.  They will not make it back to userland after this I/O completes.
  In this case, we do not need to register ourselves as p's debugger and
  may immediately return.

Subject to appropriate fencing (if required, I'm not sure; sim. [1]) it
seems that we can distinguish these cases in the requesting process.

User memory and registers (probably just the return register, tho') would be
subject to change by the kernel during the period where the process is
asleep, but this is not too dissimilar from the state of the world when
shared memory is in flight.

Would this suffice?  Did I miss something obvious?
--nwf;

[1] It may be necessary to insert a memory fence after the setting of
    p->procctl or before the read of up->procctl, since several places
    do not acquire up->debug before reading up->procctl.  Can somebody
    comment?

[-- Attachment #2: Type: application/pgp-signature, Size: 204 bytes --]

Re: [9fans] Quick question on stopping a process that waits for IO

[Roman Shaposhnik]

On Nov 5, 2008, at 9:40 PM, Nathaniel W Filardo wrote:
> Would this suffice?

It sounds like exactly the kind of thing I was talking about.

> Did I miss something obvious?

And this would be a million dollar question here. I don't
think you did (although Eric constantly warns us of
dragons), but on the other hand I have very little
experience with the kernel itself.

Thanks,
Roman.

Re: [9fans] Quick question on stopping a process that waits for IO

[Roman Shaposhnik]

On Nov 5, 2008, at 4:55 AM, erik quanstrom wrote:
>>>> I'm asking is -- "dear kernel, please don't advance this process
>>>> even
>>>> if you otherwise can". All I need is a frozen state so that I can
>>>
>>> not so easy on a multiprocessor.  (unless you turn all but one
>>> processor off.)
>>
>> Hm. May be its getting late, but I can't quite see why that would
>> be the case (or may be I didn't quite communicate the intent
>> in my plea to the kernel ;-))
>>
>> The target process is *already* waiting for the IO stuck inside the
>> kernel. It is not on a runqueue, not it is considered to be places
>> there.
>
> since procwrite doesn't acquire anything other than the debug lock,
> how do you know?  the proc could start up again before you notice.

How? If there's a stop message already written to /proc/n/ctl. Once
that is done, the process is guaranteed to be in 2 states and those
states only: continue waiting for the I/O, being actually Stopped.
Both of the don't let the scheduler take it to the runqueue.

> the place to catch processes is on syscall entry and/or exit.  stop
> messages are only checked on syscall entry.
>
> i have a feeling that you want something other than notes for
> your problem.

If by the problem you mean stopping a process doing I/O than,
I would gladly accept any solution be it notes or otherwise.

> cpu arranges that del works.


Well, it doesn't on Stopped processes.

Thanks,
Roman.

Re: [9fans] Quick question on stopping a process that waits for IO

[Roman Shaposhnik]

On Nov 4, 2008, at 4:34 AM, erik quanstrom wrote:
>> So the question remains -- what is the proper way of putting a
>> process
>> that waits for an IO into a Stopped state?
>
> i don't think it's possible without changing the kernel.
> but it's a good question, why does it work this way?
> obviously one doesn't want to stop a process and kernel work in
> the middle of i/o.  there may be locks involved, etc.
> but one could easily check for Proc_stopme on syscall
> exit.
>
> does anyone remember the thinking that went into this?

I was waiting for somebody to chime in, but I guess no takers :-(

Well, one last thing to report before we put this thread to rest
would be the fact that acid(1) seems to be suffering from the
same issue. If the process is doing I/O I can't modify the variables.
Not a huge limitation, but a surprising one nonetheless if one
draws from experience of working with conventional debuggers.

Thanks,
Roman.

Re: [9fans] Quick question on stopping a process that waits for IO

[Nathaniel W Filardo]

[-- Attachment #1: Type: text/plain, Size: 1854 bytes --]

On Thu, Nov 06, 2008 at 09:18:47PM -0800, Roman Shaposhnik wrote:
> On Nov 5, 2008, at 9:40 PM, Nathaniel W Filardo wrote:
>> Would this suffice?
>
> It sounds like exactly the kind of thing I was talking about.

OK.  To reiterate what I said earlier, these kinds of "soonstop"-ed
processes may still make some amount of progress in the kernel before
becoming stopped, but they will indeed stop at their next scheduling
opportunity, and ideally well before (when they go to return to userland).

One could imagine a parallel "soonnote" which considered the note delivered
even if the process was asleep in I/O.  This might be the more appropriate
thing for sending kill requests from the shell?  Having not looked at the
shell, this might be a bad idea; I'm not sure.

>> Did I miss something obvious?
>
> And this would be a million dollar question here. I don't
> think you did (although Eric constantly warns us of
> dragons), but on the other hand I have very little
> experience with the kernel itself.

I hope somebody comments on the fencing that is or isn't needed here.  Since
we have parts of the kernel peeking witout locks at ->procctl, I worry about
races, and wonder how, or if, the current design avoids them.

( Incidentally, if you're curious about in-kernel development and want to
look at a strange system, it is exactly the species of dragons we have
encountered here -- where the kernel has hijacked a userland thread to do
driver operations, and is potentially sleeping with driver resources held or
in local store -- that drove the design of the Coyotos kernel to being a
strictly transactional system.  AFAIR there it is always possible to tell a
thread to kill itself after its current transaction or, if asleep or in
commit phase, to force it out of stall queues and then kill it. )

--nwf;

[-- Attachment #2: Type: application/pgp-signature, Size: 204 bytes --]

Re: [9fans] Quick question on stopping a process that waits for IO

[Roman V. Shaposhnik]

erik quanstrom wrote:
>> How? If there's a stop message already written to /proc/n/ctl. Once
>> that is done, the process is guaranteed to be in 2 states and those
>> states only: continue waiting for the I/O, being actually Stopped.
>> Both of the don't let the scheduler take it to the runqueue.
>>
>
> here's the senerio, i think  (works fine on a single processor)
> a			b
> acquire debug lock
> sleep			complete io
> 			sched
> 			run a bit
> 			syscall
> wakeup
But how "run a bit" could possibly happen if after the "stop" message
being sent right after the "complete io" the "b" process goes into
a "Stopped" state?

Thanks,
Roman.

Re: [9fans] Quick question on stopping a process that waits for IO

[erik quanstrom]

> >> The target process is *already* waiting for the IO stuck inside the
> >> kernel. It is not on a runqueue, not it is considered to be places
> >> there.
> >
> > since procwrite doesn't acquire anything other than the debug lock,
> > how do you know?  the proc could start up again before you notice.
>
> How? If there's a stop message already written to /proc/n/ctl. Once
> that is done, the process is guaranteed to be in 2 states and those
> states only: continue waiting for the I/O, being actually Stopped.
> Both of the don't let the scheduler take it to the runqueue.

here's the senerio, i think  (works fine on a single processor)
a			b
acquire debug lock
sleep			complete io
			sched
			run a bit
			syscall
wakeup

- erik

Re: [9fans] Quick question on stopping a process that waits for IO

[erik quanstrom]

> >> Did I miss something obvious?
> >
> > And this would be a million dollar question here. I don't
> > think you did (although Eric (sic) constantly warns us of
> > dragons), but on the other hand I have very little
> > experience with the kernel itself.
>
> I hope somebody comments on the fencing that is or isn't needed here.  Since
> we have parts of the kernel peeking witout locks at ->procctl, I worry about
> races, and wonder how, or if, the current design avoids them.

memory fencing only forces the instruction
stream to be read, write or r/w coherent with
the local processor.  it is of no help at all for
concurrent access.  supposing there is a problem,
memory fencing will not help.  you would need
an ilock.  i didn't spend enough time looking
at pc/trap.c to convince myself one way or the
other, but the main access (other than syscall
tracing) seemed to be a read of the variable
to transition to stopped state.  this race is
likely benign.  you'll just go around again and
catch it later.

notes are hard.  and the problem they're intended
to solve is how to interrupt a single-threaded process.
if you're writing something new and you think you
might want to extend notes, i think you would be
much happier using the thread library and living
in user space.

> in local store -- that drove the design of the Coyotos kernel to being a
> strictly transactional system.  AFAIR there it is always possible to tell a

it's easy to have a design where you let the dragons
play and you need to bring out some serious weaponry
to deal with your problems.  (and it is possible to get
carried away looking for concurrency problems.)
a good design like plan 9 keeps the dragons locked up,
mostly.

- erik