[9fans] sleep(2) historical question

[9fans] sleep(2) historical question

[erik quanstrom]

why is sleep(2) limited in resolution to HZ in the
portable code?  the underlying mechanism is often
much finer grained than HZ, and if there is a limit,
one would think that it's related to the hardware
mechanism, not the HZ clock.  i'm clearly missing
something.

- erik

Re: [9fans] sleep(2) historical question

[Charles Forsyth]

the relative unimportance of sleep?

On 27 November 2012 23:19, erik quanstrom <quanstro@quanstro.net> wrote:
> why is sleep(2) limited in resolution to HZ in the
> portable code?  the underlying mechanism is often
> much finer grained than HZ, and if there is a limit,
> one would think that it's related to the hardware
> mechanism, not the HZ clock.  i'm clearly missing
> something.
>
> - erik
>

Re: [9fans] sleep(2) historical question

[erik quanstrom]

On Wed Nov 28 02:38:39 EST 2012, charles.forsyth@gmail.com wrote:
> the relative unimportance of sleep?
>
> On 27 November 2012 23:19, erik quanstrom <quanstro@quanstro.net> wrote:
> > why is sleep(2) limited in resolution to HZ in the
> > portable code?  the underlying mechanism is often
> > much finer grained than HZ, and if there is a limit,
> > one would think that it's related to the hardware
> > mechanism, not the HZ clock.  i'm clearly missing
> > something.

that's not helpful.

- erik

Re: [9fans] sleep(2) historical question

[Charles Forsyth]

No, really, I'm quite serious. A grep of /sys/src/cmd/ suggests that
most sleeps are relatively large, and arbitrary.
None of the applications look likely to need microsecond let alone
nanosecond resolution, and that seems reasonable to me.
One exception is sleep(0), but that's yield()

If I want tight timing, I'll switch to a real-time discipline,
including scheduling.



On 28 November 2012 12:57, erik quanstrom <quanstro@quanstro.net> wrote:
> On Wed Nov 28 02:38:39 EST 2012, charles.forsyth@gmail.com wrote:
>> the relative unimportance of sleep?
>>
>> On 27 November 2012 23:19, erik quanstrom <quanstro@quanstro.net> wrote:
>> > why is sleep(2) limited in resolution to HZ in the
>> > portable code?  the underlying mechanism is often
>> > much finer grained than HZ, and if there is a limit,
>> > one would think that it's related to the hardware
>> > mechanism, not the HZ clock.  i'm clearly missing
>> > something.
>
> that's not helpful.
>
> - erik
>

Re: [9fans] sleep(2) historical question

[erik quanstrom]

On Wed Nov 28 08:11:43 EST 2012, charles.forsyth@gmail.com wrote:
> No, really, I'm quite serious. A grep of /sys/src/cmd/ suggests that
> most sleeps are relatively large, and arbitrary.
> None of the applications look likely to need microsecond let alone
> nanosecond resolution, and that seems reasonable to me.
> One exception is sleep(0), but that's yield()

you haven't explained how one can emulate a sub-ms
sleep with the edf scheduler.  i've got a legit problem.

and i don't see why the common case has anything to do
with this.  if you would look at some of the stuff we're doing
at coraid (which has actually been written this decade),
i think you'll find a number of short sleeps.  i've mentioned
before that doing realistic timeouts for aoe in userspace
would require about a 50-100µs sleep, as rtt on 10gbe
is in the neighborhood of 10-40µs.

under the covers, sleep multiplies ms * 1000000 to get ns,
and sets a timer.  it is a trivial thing that doesn't harm
any part of the system to allow one to specifiy ns directly.
this is a 100% artificial limit to resolution.

also, resolution != precision.  and clearly with the type
of hardware we've got, 1µs is going to be about the limit,
and 10µs is going to be the practical limit.  

in 1990 with 20Mhz processors, one couldn't sleep for 1ms,
probablly not even 10ms.

why do extra zeros on the right bother you?  i don't mind them since
the resolution of the interface is an absolute limit on how
short a sleep i can have.  we could if you want specify sleep
in 10µs units, which would accurately reflect todays hardware.
but i'd rather err on the side of too many zeros on the right
than the left.

- erik

Re: [9fans] sleep(2) historical question

[Charles Forsyth]

Because they claim to be providing a service that's not actually being provided.

On 28 November 2012 13:30, erik quanstrom <quanstro@quanstro.net> wrote:
> why do extra zeros on the right bother you?

Re: [9fans] sleep(2) historical question

[erik quanstrom]

On Wed Nov 28 09:36:36 EST 2012, charles.forsyth@gmail.com wrote:
> Because they claim to be providing a service that's not actually being provided.
>
> On 28 November 2012 13:30, erik quanstrom <quanstro@quanstro.net> wrote:
> > why do extra zeros on the right bother you?

sleep(2) says.

     DESCRIPTION
          Sleep suspends the current process for the number of mil-
          liseconds specified by the argument.  The actual suspension
          time may be a little more or less than the requested time.

from experience, i've found that stock 386 kernels on older hardware
yield a sleep resolution of 10ms or more.

so clearly the argument to sleep is not directly tied to the resolution of
the clock.  and i think tying the two together would invent a
new requirement and assume too much on the hardware.

- erik

Re: [9fans] sleep(2) historical question

[cinap_lenrek]

that use of sleep() in usb/kb is not a good example.
it basically polls the channel at 5ms intervals with
nbrecv() just to know when it should stop repeating.

a better way would be to just have a proc doing sleeps
at the key repeat rate (like these 160ms) and sending
ticks down a channel. and then have a alt() in this repeat
proc waiting for both repeat ticks from the timer channel
and for repeat on and repeat off.

if nobody consumes from the timer channel, it will
block and stop waking up all the time.

in any case, this 5ms sleep is not really needed in
usb/kb.

--
cinap

Re: [9fans] sleep(2) historical question

[Bakul Shah]

On Wed, 28 Nov 2012 08:30:49 EST erik quanstrom <quanstro@quanstro.net>  wrote:
> On Wed Nov 28 08:11:43 EST 2012, charles.forsyth@gmail.com wrote:
> > No, really, I'm quite serious. A grep of /sys/src/cmd/ suggests that
> > most sleeps are relatively large, and arbitrary.

Chicken and egg.

> > None of the applications look likely to need microsecond let alone
> > nanosecond resolution, and that seems reasonable to me.
> > One exception is sleep(0), but that's yield()
>
> you haven't explained how one can emulate a sub-ms
> sleep with the edf scheduler.  i've got a legit problem.

Why not add nsleep() with sleeptime in nanosecond units?  And
of course, any necessary kernel changes for better accuracy.

The whole idea of a kernel HZ clock seems a bit outdated now.
If the system has nothing do for the next N seconds, it should
may be go into low-power mode & just wait for an interrupt.

Re: [9fans] sleep(2) historical question

[Charles Forsyth]

No, I don't think it is, in this case. I really don't see many
applications deeply yearning for tiny sleeps and naplets.

On 28 November 2012 18:58, Bakul Shah <bakul@bitblocks.com> wrote:
> Chicken and egg.

Re: [9fans] sleep(2) historical question

[David Arnold]

On 28/11/2012, at 1:58 PM, Bakul Shah wrote:

> Why not add nsleep() with sleeptime in nanosecond units?  And
> of course, any necessary kernel changes for better accuracy.

Is POSIX' nanosleep() or old-POSIX/BSD/XOpen's usleep() out of the question?




d

Re: [9fans] sleep(2) historical question

[erik quanstrom]

10ms, the current limit to sleep resolution from sources,
is a massive chunk of time.  even 100µs is a rather large chunk
of time these days.  10 millisecond is
- 12.5 MB on a 10gbe network (1.25mb on gbe) or 1000 rtt (100rtt)
- 30 million instructions (per core), assuming only scalar.

On Wed Nov 28 14:11:58 EST 2012, charles.forsyth@gmail.com wrote:
> No, I don't think it is, in this case. I really don't see many
> applications deeply yearning for tiny sleeps and naplets.

even usb/kb uses sleep(5), which isn't going to work as intended.
that's getting pretty close to the limit.

ping is an example of a little program that really could use better
timing.  ping -i sets the interval between frames using sleep.
a minimum interval of 1000 rtt really limits the utility.

i think you're arguing for making anything about that granularity
*impossible*.

- erik

Re: [9fans] sleep(2) historical question

[erik quanstrom]

> > None of the applications look likely to need microsecond let alone
> > nanosecond resolution, and that seems reasonable to me.
> > One exception is sleep(0), but that's yield()
> 
> Chicken and egg.

i think so.

> > > None of the applications look likely to need microsecond let alone
> > > nanosecond resolution, and that seems reasonable to me.
> > > One exception is sleep(0), but that's yield()
> > 
> > you haven't explained how one can emulate a sub-ms
> > sleep with the edf scheduler.  i've got a legit problem.
> 
> Why not add nsleep() with sleeptime in nanosecond units?  And
> of course, any necessary kernel changes for better accuracy.

as it turns out, for anything i had in mind, the kernel already can
do about 10x what i need.  i need maybe 500µs, and i can get 50µs
at 4% error.

the only problem i see with just adding nsleep is it introduces a
second time base, and potentially any time-based call (tsemacquire)
would need to be doubled.  i would prefer for the end state to be
1 user space time base.  but the difficulty is getting there.  at the
least, the syscall# would change.

> The whole idea of a kernel HZ clock seems a bit outdated now.
> If the system has nothing do for the next N seconds, it should
> may be go into low-power mode & just wait for an interrupt.

and when you have things to do, the hz clock is just an unnecessary
interruption.

- erik

Re: [9fans] sleep(2) historical question

[Bakul Shah]

On Wed, 28 Nov 2012 19:10:19 GMT Charles Forsyth <charles.forsyth@gmail.com>  wrote:
> No, I don't think it is, in this case. I really don't see many
> applications deeply yearning for tiny sleeps and naplets.
>
> On 28 November 2012 18:58, Bakul Shah <bakul@bitblocks.com> wrote:
> > Chicken and egg.

I wrote a packet traffic/protocol simulator where a 250Hz
clock (seemed to be the default on Linux) was not fast enough.
On FreeBSD I could just up the clock to 10Khz.  RTT
computation as Erik pointed out.  If you want to minimize
latency in usermode drivers and deal with simple(r) hardware.

Lots of uses.

Re: [9fans] sleep(2) historical question

[Bakul Shah]

On Wed, 28 Nov 2012 14:29:19 EST erik quanstrom <quanstro@quanstro.net>  wrote:
>
> the only problem i see with just adding nsleep is it introduces a
> second time base, and potentially any time-based call (tsemacquire)
> would need to be doubled.  i would prefer for the end state to be
> 1 user space time base.  but the difficulty is getting there.  at the
> least, the syscall# would change.

If you want to maintain binary compatibility, new syscalls
would be needed. If you don't care, you can use wrapper
functions for lower precision calls.  Evolution is messy.

Re: [9fans] sleep(2) historical question

[erik quanstrom]

On Wed Nov 28 14:56:34 EST 2012, bakul@bitblocks.com wrote:
> On Wed, 28 Nov 2012 14:29:19 EST erik quanstrom <quanstro@quanstro.net>  wrote:
> >
> > the only problem i see with just adding nsleep is it introduces a
> > second time base, and potentially any time-based call (tsemacquire)
> > would need to be doubled.  i would prefer for the end state to be
> > 1 user space time base.  but the difficulty is getting there.  at the
> > least, the syscall# would change.
>
> If you want to maintain binary compatibility, new syscalls
> would be needed. If you don't care, you can use wrapper
> functions for lower precision calls.  Evolution is messy.

just the #, not the name.

- erik

Re: [9fans] sleep(2) historical question

[Charles Forsyth]

RTT computation isn't the problem. you're going to use a multicore
numa 64-bit processor to clock edges into a simple device register at
microsecond resolution without any attempt at timeliness? seriously?

On 28 November 2012 19:38, Bakul Shah <bakul@bitblocks.com> wrote:
> RTT
> computation as Erik pointed out.  If you want to minimize
> latency in usermode drivers and deal with simple(r) hardware.

Re: [9fans] sleep(2) historical question

[erik quanstrom]

On Wed Nov 28 15:59:05 EST 2012, charles.forsyth@gmail.com wrote:
> RTT computation isn't the problem. you're going to use a multicore
> numa 64-bit processor to clock edges into a simple device register at
> microsecond resolution without any attempt at timeliness? seriously?

what modern hardware likely to be on a 64-bit numa processor
is going to require one to clock hard-timed edges?

- erik

Re: [9fans] sleep(2) historical question

[Charles Forsyth]

If you look at the manual pages for nanosleep, clock_nanosleep and
others on Linux you'll see that they have many bits
of advice, BUGS notes, and other statements to the effect that the
standard things do not work well, and most need some
sort of real-time support to do anything sensible. An earlier variant
did a busy wait when needed (which is unsurprising,
but assumes processor wiring, and no scheduling away from the process).

On 28 November 2012 19:27, David Arnold <davida@pobox.com> wrote:
> Is POSIX' nanosleep() or old-POSIX/BSD/XOpen's usleep() out of the question?

Re: [9fans] sleep(2) historical question

[Charles Forsyth]

But that's exactly my point! Read the context of that remark.

On 28 November 2012 21:02, erik quanstrom <quanstro@quanstro.net> wrote:
> what modern hardware likely to be on a 64-bit numa processor
> is going to require one to clock hard-timed edges?

Re: [9fans] sleep(2) historical question

[erik quanstrom]

On Wed Nov 28 16:14:28 EST 2012, charles.forsyth@gmail.com wrote:
> But that's exactly my point! Read the context of that remark.
>
> On 28 November 2012 21:02, erik quanstrom <quanstro@quanstro.net> wrote:
> > what modern hardware likely to be on a 64-bit numa processor
> > is going to require one to clock hard-timed edges?

okay, then.  how would you solve my problem?  i need delays
less than 1ms  in user space.  i'd like a practical suggestion.

- erik

Re: [9fans] sleep(2) historical question

[erik quanstrom]

On Wed Nov 28 16:14:28 EST 2012, charles.forsyth@gmail.com wrote:
> But that's exactly my point! Read the context of that remark.
>
> On 28 November 2012 21:02, erik quanstrom <quanstro@quanstro.net> wrote:
> > what modern hardware likely to be on a 64-bit numa processor
> > is going to require one to clock hard-timed edges?
>

straw man.

- erik

Re: [9fans] sleep(2) historical question

[Bakul Shah]

On Wed, 28 Nov 2012 20:58:03 GMT Charles Forsyth <charles.forsyth@gmail.com>  wrote:
> RTT computation isn't the problem. you're going to use a multicore
> numa 64-bit processor to clock edges into a simple device register at
> microsecond resolution without any attempt at timeliness? seriously?

Not everyone is running plan9 on multicore numa machines!  I
want to run it specifically on low end machines, *for*
hardware control among other things. Other OS choices here are
not as palatable.

By your argument why even 100Hz? Why not have a 1sec clock?

If a finer resolution timers were provided, may be that'd be
an incentive to fix any plan9 issues with scheduling or
accuracy.  And I think fixing such issues will be *easier* in
plan9 than in Linux or BSD.

And in any case why stick to an artificial choice made decades
ago when it gets in the way of at lease some applications?
Seems best to get rid of the fixed 100Hz clock and allow as
fine a timer resolution (& accuracy) as a particular CPU +
kernel combination can do.

Re: [9fans] sleep(2) historical question

[Charles Forsyth]

well, that's ok.

On 28 November 2012 21:43, Bakul Shah <bakul@bitblocks.com> wrote:
> And in any case why stick to an artificial choice made decades
> ago when it gets in the way of at lease some applications?
> Seems best to get rid of the fixed 100Hz clock and allow as
> fine a timer resolution (& accuracy) as a particular CPU +
> kernel combination can do.

Re: [9fans] sleep(2) historical question

[arnold]

> Seems best to get rid of the fixed 100Hz clock and allow as
> fine a timer resolution (& accuracy) as a particular CPU +
> kernel combination can do.

So what this argues for is a query interface - hello mr. kernel, what can
you and your hardware do?  In whatever units make sense. Then userspace
can do the calculation it needs and ask for the sleep time that it will
know the kernel can really supply.

Erik's problem boils down to "the kernel knows what it can do and is
quite capable but i can't get it at from user space".  So (a) make
the information availble and (b) let user space figure it out. Classic
separation of mechanism from policy.

At least, that what it looks like to me. :-)

Arnold

Re: [9fans] sleep(2) historical question

[Charles Forsyth]

My problem is that the kernel is not currently capable of supplying
what Erik defines, to the process level even internally,
and even for his described applications (delays of a small interval,
tiny retransmission times), it's not clear that the traditional sleep,
which has
*always* been sloppy, is anyway the right model; and certainly not for
any more general application.
I had a quick read through some papers and theses last night, and it
certainly isn't clear cut.

What values could the kernel currently supply to a query interface?
There are no particular bounds
on anything, which was rather my point. I had a similar problem
*inside the kernel* when dealing with
a fast network interface on the Blue Gene.

The whole thing started with an "historical question": "why is
sleep(2) limited to resolution HZ in the portable code? ..."
and I gave what I think is the correct reply historically: "the
relative unimportance of sleep?", but as usual that didn't go down
very well.

For relatively small delays, sleep -> tsleep isn't right. Most of the
ways of making "high-precision" timers available at user level
contain a lot of overhead, or frustration when trying to use them (eg,
POSIX sends a signal, never a harbinger of speed or ease of use).
I've probably just had my fill of general-purpose systems that confuse
"not too often slow" with "timely".

It's less of a problem for Plan 9, than Nix, where I'd like to see any
new things added be sound.

On 29 November 2012 08:12,  <arnold@skeeve.com> wrote:
> Erik's problem boils down to "the kernel knows what it can do and is
> quite capable but i can't get it at from user space".

Re: [9fans] sleep(2) historical question

[erik quanstrom]

On Thu Nov 29 04:20:52 EST 2012, charles.forsyth@gmail.com wrote:
> My problem is that the kernel is not currently capable of supplying
> what Erik defines, to the process level even internally, and even for
> his described applications (delays of a small interval, tiny
> retransmission times), it's not clear that the traditional sleep,
> which has *always* been sloppy, is anyway the right model; and
> certainly not for any more general application.  I had a quick read
> through some papers and theses last night, and it certainly isn't
> clear cut.

i ran a little test.  the meat of the program is this

	cycles(t+0);
	for(i = 0; i < 5000000000ull; i += s)
		nsleep(s);
	cycles(t+1);

	print("%g\n", cyclestoseconds(t[1]-t[0]));

i get quite predictable results; there is no cliff.  

sleep in ns	elapsed time	calculated syscall overhead
1000000000	5.000012		# 2.40µs / syscall
100000000	5.000106		# 2.12µs / syscall
10000000	5.001097		# 2.19µs / syscall
1000000		5.01078		# 2.16µs / syscall
100000		5.102487		# 2.16µs / syscall
10000		6.081568		# 2.16µs / syscall

- erik