[9fans] critique of sockets API

[9fans] critique of sockets API

[Bhanu Nagendra Pisupati]

Interesting read:
http://cacm.acm.org/magazines/2009/6/28495-whither-sockets/fulltext

If I am right, the filesystem based networking interface offered by Plan 9
has the three limitations discussed here:
* performance overhead: app requesting data from a socket typically needs
to perform 2 system calls (select/read or alt/read)
* lack of an "kernel up-call API": which allows the kernel to inform an
app each time network data is available
* hard to implement "multi homing" with support for multiple network
interfaces

Any thoughts/comments?

Thanks,
-Bhanu

Re: [9fans] critique of sockets API

[erik quanstrom]

On Tue Jun  9 14:50:58 EDT 2009, bpisupat@cs.indiana.edu wrote:
> Interesting read:
> http://cacm.acm.org/magazines/2009/6/28495-whither-sockets/fulltext
>
> If I am right, the filesystem based networking interface offered by Plan 9
> has the three limitations discussed here:
> * performance overhead: app requesting data from a socket typically needs
> to perform 2 system calls (select/read or alt/read)

alt — which is not required — is not a system call.  only a read or write is
required.

> * lack of an "kernel up-call API": which allows the kernel to inform an
> app each time network data is available

plan 9 uses synchronous i/o, so this statement doesn't make sense
in plan 9.  you can use threads to do i/o asynch w.r.t. your application,
but the i/o itself is still synchronous w.r.t. the kernel.

> * hard to implement "multi homing" with support for multiple network
> interfaces

i have no idea how this relates to the use of a fs in implementing the
network stack.  why would using a filsystem (or not) make any difference
in the ability to multihome?

by the way, plan 9 beats the pants off anything else i've used for multiple
network / interface support.  it's support for mulitple ip stacks is quite
neat.

- erik

Re: [9fans] critique of sockets API

[Russ Cox]

> * performance overhead: app requesting data from a socket typically needs to
> perform 2 system calls (select/read or alt/read) * lack of an "kernel
> up-call API": which allows the kernel to inform an app each time network
> data is available

there is a mechanism.
user programs call read(2).
when data is available,
the read finishes.

Re: [9fans] critique of sockets API

[Bhanu Nagendra Pisupati]

>do you have a comparison to other similar projects like
>1. styx on a brick
>	http://www.vitanuova.com/inferno/rcx_paper.html
>2. 9p for imbedded devices
>	http://iwp9.inf.uth.gr/iwp9_proceedings08.pdf
>
>and or other information on your work?  in particular,
>what hardware are you using?

The "Styx on a Brick" paper was a definition inspiration for my work. I
have acknowledged as much in my publications.

I completed my PhD in '07 before the "embedFS" filesystem paper from IW9P
came out last year, and I have not done a formal comparison per se. But
reading through the text, I do see similarities in some of our ideas,
such as having one outstanding messag, of read/read message
size, using 9P (rather than 9p2000) etc.

The IW9P paper does not give size numbers, but the basic 9P filesystem in
my implementation was about 15KB in size when implemented on a 32-bit RISC
processor named Nios (an embedded processor from Altera).

Most of my work was done using the Nios embedded FPGA environment from
Altera. For more information:
http://www.cs.indiana.edu/cgi-bin/techreports/TRNNN.cgi?trnum=TR647
http://www.cs.indiana.edu/pub/techreports/TR647.html/embed.html

Re: [9fans] critique of sockets API

[Eris Discordia]

> s/could be/is/
>
> From real world product experience across multiple operating systems
> and architectures.

So there is at least one example to support the case for callbacks. I am
pretty convinced there are many more examples.

--On Thursday, June 11, 2009 19:34 -0400 "Devon H. O'Dell"
<devon.odell@gmail.com> wrote:

> 2009/6/11 Eris Discordia <eris.discordia@gmail.com>:
>>> but given that plan 9 is about having a system that's easy
>>> to understand and modify, i would think that it would be
>>> tough to demonstrate that asyncronous i/o or callbacks
>>> could make the system (or even applications) simplier.
>>> i doubt that they would make the system more efficient,
>>> either.
>>>
>>> do you have examples that demonstrate either?
>>
>> I can't claim I have anything in code which would be necessary for an
>> actual demonstration or for going beyond the "talk talk talk" stage. I
>> can, however, present one simple case: in some applications asynchronous
>> name resolving is a must and it can be realized by either of threads or
>> callbacks. Crawlers and scanners come to mind. Spawning threads for DNS
>> requests could be more costly than registering a set of callbacks within
>> one thread and then harvesting the results within that same thread.
>
> s/could be/is/
>
> From real world product experience across multiple operating systems
> and architectures.
>

Re: [9fans] critique of sockets API

[Eris Discordia]

> just to correct a basic fact, the size of the instruction set doesn't
> define RISC instruction set (http://en.wikipedia.org/wiki/RISC
> "instruction set size").

Heh. I did refer to exactly that article and consequently inserted "adding 
to the complexity of each primitive" as well as the modifier "strictly."

> can you cite any references that claim that the size of intel's
> instruction set has contributed to its success?

Try the Wikipedia article on CISC which really isn't a source, I admit. The 
general impression is that not only the total number, and diversity, of the 
instruction set but how much is done using a single instruction from the 
viewpoint of a programmer/compiler have a role there. How a CISC 
instruction set is implemented is besides this point. Some translation may 
take place in the process but it doesn't matter as long as it is 
transparent to the programmer/compiler.

> could it be that since transistors are very cheep, adding instructions is
> simply the cheapest go-faster trick?

Could be exactly that and further confirm my stance that added complexity, 
when it is carefully hidden and is exposed to the middle user only as a 
variety of orthogonal options, does not result in a bad system. In can 
actually result in a better system. I'm, of course, not claiming that the 
x86 instruction set represents an epitome of orthogonality or good design.

> what is a middle user?  and why would such a person be
> discouraged by having to learn fewer things?  does the myriad
> of different ways to write an if statement in perl make it more
> useable?  readable?

A middle user is someone who uses your product to create another product 
for an end user who very often doesn't create product on their own, 
although that may not be the case. A programmer who uses the language and 
compiler you designed, or the OS you created, or the API you developed. 
Having to learn fewer things is not discouraging, lack of expressiveness 
due to lack of options can be. In case of Perl its popularity establishes 
its merit for the area in which it is popular. Besides, we aren't talking 
about if redundancy makes a system more expressive (it could but we aren't 
talking about that), we are rather talking about addition of options that 
are orthogonal to existing ones: options that have no equivalent.

A callback infrastructure can be created in user land using threading but 
that would have two disadvantages: the very point of lowering initial cost 
using callback strategy is made moot, and people will tend to disagree on 
how that extra infrastructure should look like so a variety of 
incompatible, highly redundant implementations will pop up.

> i don't see how progress == complicated.  could you explain
> how they are one and the same?
>
> as a counter example, unix uses untyped files.  not only does
> lack of typing make the system simplier, it makes it more expressive
> as well.

I don't see how complex == complicated. You could have a very sophisticated 
system at hand that is rather easy to program. Increasing adaptivity 
through increased complexity is one well-known evolutionary path. The other 
well-known path is increasing resilience through decreased complexity. 
There is a point where resilience and adaptivity are just at the balance 
you desire for your audience. Making clear who your audience is should 
clear this problem as well.

Regarding your example I believe you are mixing redundancy with choice. 
While I could even argue for some redundancy I'm not trying to do so. File 
typing, as seen on Apple systems, is not orthogonal to other features 
already present in those systems; it is redundant. Windows does not have 
any file typing similar to Apple's. UNIX-like systems do make some 
distinctions between files which have become rather blurry with time. I 
brought up the the subject in my awkward manner some time ago when caching 
9P was being discussed on the list. It seemed to me that a form of typing 
that expressed at a reasonable level of detail the amenability of files to 
various caching strategies could have improved the situation of 9P caching.

> i don't see how a wage-earning programmer can't be a researcher as well.
> and being a wage-earning programmer, i appreciate simplicity and use
> it to advantage on a daily basis.

The vast majority of them aren't. That's a fact of life. You appreciate 
simplicity because you happen to work on a specific application for which 
your target system's API is exceptionally well-rounded. Try designing a 
complex UI for some CAD software and tell me how amenable your simple 
system is to that purpose. The platforms targeted for, say, creating the 
frontend to a CAD system are not chosen by luck really. They have been made 
suitable by designers to that and other purposes. Adding of orthogonal 
options, when done wisely, should not take away or negatively affect the 
core primitives you use and are content with.

> several times, i've needed to get a particular bit of h/w working.
> given a choice, i go with the one with the thinner manual.

I bet a 8051's manual is thinner that a 80x86's. Does that mean the 8051 
will be your platform of choice when it absolutely does not cut the task? I 
think it's somebody's motto: simple enough, but no(t) simpler.

--On Thursday, June 11, 2009 19:41 -0400 erik quanstrom 
<quanstro@quanstro.net> wrote:

>> > i think you're trying to argue that — a priori — choice is good?
>>
>> I believe it is. How many of us are using strictly RISC machines on our
>> desks today? Extending the set of available primitives and adding to the
>> complexity of each primitive both are natural steps in the development
>> of  computer systems as they see more use.
>
> just to correct a basic fact, the size of the instruction set doesn't
> define RISC instruction set (http://en.wikipedia.org/wiki/RISC
> "instruction set size").
>
> can you cite any references that claim that the size of intel's
> instruction set has contributed to its success?
>
> could it be that since transistors are very cheep, adding instructions is
> simply the cheapest go-faster trick?
>
>> Limiting options doesn't seem to me
>> to be an effective way of encouraging good programming practice. It can,
>> however, successfully discourage potential middle users.
>
> what is a middle user?  and why would such a person be
> discouraged by having to learn fewer things?  does the myriad
> of different ways to write an if statement in perl make it more
> useable?  readable?
>
>> > that's not the position i subscribe to.  and since plan 9
>> > is simple and easy to change, it makes an ideal system
>> > for someone who wants to try new things.
>>
>> And after the new things are tried out and, one may hope, proven
>> advantageous? I think the next step would be incorporating them into the
>> system. A process that in due time will make the system less simple and
>> easy to change but more immediately useful.
>
> i don't see how progress == complicated.  could you explain
> how they are one and the same?
>
> as a counter example, unix uses untyped files.  not only does
> lack of typing make the system simplier, it makes it more expressive
> as well.
>
>> I see this more as a difference of intended audience than a difference
>> of  taste or philosophy. The real question is whom (sic) you imagine as
>> your system's  middle user: a wage-earning programmer or a researcher.
>> Were I a programmer  who worked for pay I'd very much appreciate being
>> given every possible  option that would let me do my job easier, faster,
>> and, of course, more  properly.
>
> i don't see how a wage-earning programmer can't be a researcher as well.
> and being a wage-earning programmer, i appreciate simplicity and use
> it to advantage on a daily basis.
>
> several times, i've needed to get a particular bit of h/w working.
> given a choice, i go with the one with the thinner manual.
>
> - erik
>

Re: [9fans] critique of sockets API

[Paul Lalonde]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Even more than transistors are so cheap that doing on-the-fly
translation to a RISC instruction set in hardware is an essentially
invisible cost.  Plus you get to change the target microarchitecture
to exploit new thinking in processor design without breaking existing
software.

That last reason, not choice, is how IA became dominant.

Paul

On Jun 11, 2009, at 4:41 PM, erik quanstrom wrote:

> could it be that since transistors are very cheep, adding
> instructions is
> simply the cheapest go-faster trick?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.3 (Darwin)

iD8DBQFKMdpjpJeHo/Fbu1wRAsPTAKCmeg6YnYvCNBHpWQCd9hCAM5+Y/wCgw8//
+MMUaZeL/qPutVO0W7sYTqM=
=D+2E
-----END PGP SIGNATURE-----

Re: [9fans] critique of sockets API

[erik quanstrom]

> > i think you're trying to argue that — a priori — choice is good?
>
> I believe it is. How many of us are using strictly RISC machines on our
> desks today? Extending the set of available primitives and adding to the
> complexity of each primitive both are natural steps in the development of
> computer systems as they see more use.

just to correct a basic fact, the size of the instruction set doesn't define RISC
instruction set (http://en.wikipedia.org/wiki/RISC "instruction set size").

can you cite any references that claim that the size of intel's
instruction set has contributed to its success?

could it be that since transistors are very cheep, adding instructions is
simply the cheapest go-faster trick?

> Limiting options doesn't seem to me
> to be an effective way of encouraging good programming practice. It can,
> however, successfully discourage potential middle users.

what is a middle user?  and why would such a person be
discouraged by having to learn fewer things?  does the myriad
of different ways to write an if statement in perl make it more
useable?  readable?

> > that's not the position i subscribe to.  and since plan 9
> > is simple and easy to change, it makes an ideal system
> > for someone who wants to try new things.
>
> And after the new things are tried out and, one may hope, proven
> advantageous? I think the next step would be incorporating them into the
> system. A process that in due time will make the system less simple and
> easy to change but more immediately useful.

i don't see how progress == complicated.  could you explain
how they are one and the same?

as a counter example, unix uses untyped files.  not only does
lack of typing make the system simplier, it makes it more expressive
as well.

> I see this more as a difference of intended audience than a difference of
> taste or philosophy. The real question is whom (sic) you imagine as your system's
> middle user: a wage-earning programmer or a researcher. Were I a programmer
> who worked for pay I'd very much appreciate being given every possible
> option that would let me do my job easier, faster, and, of course, more
> properly.

i don't see how a wage-earning programmer can't be a researcher as well.
and being a wage-earning programmer, i appreciate simplicity and use
it to advantage on a daily basis.

several times, i've needed to get a particular bit of h/w working.
given a choice, i go with the one with the thinner manual.

- erik

Re: [9fans] critique of sockets API

[Devon H. O'Dell]

2009/6/11 Eris Discordia <eris.discordia@gmail.com>:
>> but given that plan 9 is about having a system that's easy
>> to understand and modify, i would think that it would be
>> tough to demonstrate that asyncronous i/o or callbacks
>> could make the system (or even applications) simplier.
>> i doubt that they would make the system more efficient,
>> either.
>>
>> do you have examples that demonstrate either?
>
> I can't claim I have anything in code which would be necessary for an actual
> demonstration or for going beyond the "talk talk talk" stage. I can,
> however, present one simple case: in some applications asynchronous name
> resolving is a must and it can be realized by either of threads or
> callbacks. Crawlers and scanners come to mind. Spawning threads for DNS
> requests could be more costly than registering a set of callbacks within one
> thread and then harvesting the results within that same thread.

s/could be/is/

>From real world product experience across multiple operating systems
and architectures.

Re: [9fans] critique of sockets API

[Eris Discordia]

> but given that plan 9 is about having a system that's easy
> to understand and modify, i would think that it would be
> tough to demonstrate that asyncronous i/o or callbacks
> could make the system (or even applications) simplier.
> i doubt that they would make the system more efficient,
> either.
>
> do you have examples that demonstrate either?

I can't claim I have anything in code which would be necessary for an 
actual demonstration or for going beyond the "talk talk talk" stage. I can, 
however, present one simple case: in some applications asynchronous name 
resolving is a must and it can be realized by either of threads or 
callbacks. Crawlers and scanners come to mind. Spawning threads for DNS 
requests could be more costly than registering a set of callbacks within 
one thread and then harvesting the results within that same thread.

> one can build what's needed.  compare just r?fork and exec
> with all the spawn variants windows has.

The actual number of calls in CreateThread and CreateProcess family is 
rather small. Most calls are wrappers for more basic calls but we don't 
want to go into that.

> i think you're trying to argue that — a priori — choice is good?

I believe it is. How many of us are using strictly RISC machines on our 
desks today? Extending the set of available primitives and adding to the 
complexity of each primitive both are natural steps in the development of 
computer systems as they see more use. Limiting options doesn't seem to me 
to be an effective way of encouraging good programming practice. It can, 
however, successfully discourage potential middle users.

> that's not the position i subscribe to.  and since plan 9
> is simple and easy to change, it makes an ideal system
> for someone who wants to try new things.

And after the new things are tried out and, one may hope, proven 
advantageous? I think the next step would be incorporating them into the 
system. A process that in due time will make the system less simple and 
easy to change but more immediately useful.

I see this more as a difference of intended audience than a difference of 
taste or philosophy. The real question is whom you imagine as your system's 
middle user: a wage-earning programmer or a researcher. Were I a programmer 
who worked for pay I'd very much appreciate being given every possible 
option that would let me do my job easier, faster, and, of course, more 
properly.

--On Thursday, June 11, 2009 14:24 -0400 erik quanstrom 
<quanstro@quanstro.net> wrote:

>> I might as well repeat myself: choice of strategy depends on the
>> application. Given choice programmers can decide on which strategy or
>> combination of strategies works best. Without choice, well, they will
>> just  live with what's available.
>
> this is a very deep philosophical divide between windows and
> systems like plan 9, and research unix.  the approach the labs
> took was to provide a minimal set of primatives from which
> one can build what's needed.  compare just r?fork and exec
> with all the spawn variants windows has.
>
> i think you're trying to argue that — a priori — choice is good?
>
> but given that plan 9 is about having a system that's easy
> to understand and modify, i would think that it would be
> tough to demonstrate that asyncronous i/o or callbacks
> could make the system (or even applications) simplier.
> i doubt that they would make the system more efficient,
> either.
>
> do you have examples that demonstrate either?
>
>> One Right Way" always leaves open the question of whether a different
>> choice of strategy on the same platform, were a different choice
>> available,  would have yielded better results.
>
> clearly if that position is accepted, computer science is a
> solved problem; we should all put our heads down and just
> code up the accepted wisdom.
>
> that's not the position i subscribe to.  and since plan 9
> is simple and easy to change, it makes an ideal system
> for someone who wants to try new things.
>
> - erik
>

Re: [9fans] critique of sockets API

[erik quanstrom]

> I might as well repeat myself: choice of strategy depends on the
> application. Given choice programmers can decide on which strategy or
> combination of strategies works best. Without choice, well, they will just
> live with what's available.

this is a very deep philosophical divide between windows and
systems like plan 9, and research unix.  the approach the labs
took was to provide a minimal set of primatives from which
one can build what's needed.  compare just r?fork and exec
with all the spawn variants windows has.

i think you're trying to argue that — a priori — choice is good?

but given that plan 9 is about having a system that's easy
to understand and modify, i would think that it would be
tough to demonstrate that asyncronous i/o or callbacks
could make the system (or even applications) simplier.
i doubt that they would make the system more efficient,
either.

do you have examples that demonstrate either?

> One Right Way" always leaves open the question of whether a different
> choice of strategy on the same platform, were a different choice available,
> would have yielded better results.

clearly if that position is accepted, computer science is a
solved problem; we should all put our heads down and just
code up the accepted wisdom.

that's not the position i subscribe to.  and since plan 9
is simple and easy to change, it makes an ideal system
for someone who wants to try new things.

- erik

Re: [9fans] critique of sockets API

[Eris Discordia]

> the signal context is the original calling thread.  unless
> ms' diagram is incorrect, this is a single threaded operation;
> only the i/o originator can process the event.  so the plan 9

Of course it is single-threaded operation. That's the very idea behind 
using callbacks. Originally they were used to allow applications to do 
asynchronous I/O on earlier Windows incarnations that had no concept of 
threading. When threading became available it was correctly considered 
hazardous, as with any other form of concurrency, so it was and is avoided 
unless absolutely necessary. Callbacks survived the introduction of 
threading.

There are actually three thinkable options here: read-and-wait 
(synchronous), subscribe-and-continue (asynchronous using callback), and 
continue-until-notified (asynchronous using message queue). My point was 
that as of some time ago, more than a decade I believe, Windows began 
offering all three options which may be used on their own or in tandem. The 
three options can be sorted in order of increasing initial cost: callback, 
message queue, thread (and/or "lightweight process"). Average cost reverses 
that ordering. The application determines which strategy is better.

On the subject of who gets to process a certain event on Windows it should 
be noted that any process/thread that subscribes for an event will be 
notified, if an opportunity for subscription is provided in the first 
place. In case of a read other processes/threads have no notion that a 
specific read has been requested and they will never want to process the 
results of a read they haven't requested. Interface and other events that 
may concern more than one process/thread, on the other hand, can be 
processed by any worker that opts to process them. A chain of handlers with 
varying priorities is very easy to construct within this framework.

> there's plenty of overlapped i/o in plan 9 — it's in the
> disk and network device drivers.  even stripping away
> the register fiddling, it's not an i/o model that's attractive;
> that's the reason all the details are hidden in the kernel.

I might as well repeat myself: choice of strategy depends on the 
application. Given choice programmers can decide on which strategy or 
combination of strategies works best. Without choice, well, they will just 
live with what's available. It is common with Windows programmers to use a 
combination of threading, callbacks, and synchronous I/O that best 
represents the application's workings. GUI is often run in its own thread 
which is made to wait only on I/O that crucially affects future 
interactions, e.g. a user "Open File" request. Processing of sporadic 
network events, e.g. reports from a network resource, is done in auxiliary 
message queues. Light network operations are delegated to a single thread 
that uses callbacks. Only heavy network operations are aggressively 
threaded.

Most notable here is the fact that allowing for various compatible options 
to grow within the same system can only enrich it while insisting on "The 
One Right Way" always leaves open the question of whether a different 
choice of strategy on the same platform, were a different choice available, 
would have yielded better results.

> are you saying the author(s) had their windows blinders on
> and might not have considered other options?

I believe at this point it is clear that all thinkable options are 
available on the platform you refer to as (horse) "blinders" but that 
really isn't important. The author(s) had criticized a specific I/O model 
without mentioning, or probably even knowing, that alternatives existed and 
could be profiled for tangible, definitive results. I merely pointed that 
out.

--On Thursday, June 11, 2009 08:16 -0400 erik quanstrom 
<quanstro@quanstro.net> wrote:

> On Thu Jun 11 04:12:13 EDT 2009, eris.discordia@gmail.com wrote:
>> > i don't think i understand what you're getting at.
>> > it could be that the blog was getting at the fact that select
>> > funnels a bunch of independent i/o down to one process.
>> > it's an effective technique when (a) threads are not available
>> > and (b) processing is very fast.
>>
>> This might help: what he is getting at is probably the question of why
>> not  make possible network applications that consist of a bunch of
>> callbacks or  a mix of callbacks and listener/worker threads. Windows
>> implements both  synchronous and asynchronous I/O. Threads are
>> available. Callbacks, too, as  well as message queues.
>
> are you saying the author(s) had their windows blinders on
> and might not have considered other options?
>
> my windows-fu is very low.  but according to microsoft
> http://msdn.microsoft.com/en-us/library/aa365683(VS.85).aspx
> windows "asynchronous" (overlapped) i/o signals the calling thread, so
> the signal context is the original calling thread.  unless
> ms' diagram is incorrect, this is a single threaded operation;
> only the i/o originator can process the event.  so the plan 9
> model would seem to me to be better threaded and i think
> the CSP-style of the plan 9 model makes it easier to
> reason about.  and has already been implemented under
> unix without changing the kernel.  see p9p.
>
>> Ideally, it is the programmer's informed choice
>> based on their understanding of their application's priorities whether
>> to  use callbacks, listener/worker threads, message queues, or a
>> combination.  Someone may find it worth the effort to compare these
>> approaches on a  platform that provides both. (COM is notorious for
>> implementing things  through callback that get some wrapping of one's
>> head around them before  making sense.)
>
> there's plenty of overlapped i/o in plan 9 — it's in the
> disk and network device drivers.  even stripping away
> the register fiddling, it's not an i/o model that's attractive;
> that's the reason all the details are hidden in the kernel.
>
> - erik

Re: [9fans] critique of sockets API

[erik quanstrom]

On Wed Jun 10 18:52:19 EDT 2009, bpisupat@cs.indiana.edu wrote:
>
> > Did you do this on Plan 9 or bring some of the filesystem sanity of another OS?
>
> Actually my work was based on 9P based synthetic filesystems
> implemented using Npfs (see
> http://docs.google.com/Doc?id=dcb7zf48_503q8j84)
>
> These filesystems resided in resource constrained embedded devices which
> could not run Plan 9 in whole.
>
>

do you have a comparison to other similar projects like

1. styx on a brick
	http://www.vitanuova.com/inferno/rcx_paper.html
2. 9p for imbedded devices
	http://iwp9.inf.uth.gr/iwp9_proceedings08.pdf

and or other information on your work?  in particular,
what hardware are you using?

- erik

Re: [9fans] critique of sockets API

[erik quanstrom]

On Thu Jun 11 04:12:13 EDT 2009, eris.discordia@gmail.com wrote:
> > i don't think i understand what you're getting at.
> > it could be that the blog was getting at the fact that select
> > funnels a bunch of independent i/o down to one process.
> > it's an effective technique when (a) threads are not available
> > and (b) processing is very fast.
>
> This might help: what he is getting at is probably the question of why not
> make possible network applications that consist of a bunch of callbacks or
> a mix of callbacks and listener/worker threads. Windows implements both
> synchronous and asynchronous I/O. Threads are available. Callbacks, too, as
> well as message queues.

are you saying the author(s) had their windows blinders on
and might not have considered other options?

my windows-fu is very low.  but according to microsoft
http://msdn.microsoft.com/en-us/library/aa365683(VS.85).aspx
windows "asynchronous" (overlapped) i/o signals the calling thread, so
the signal context is the original calling thread.  unless
ms' diagram is incorrect, this is a single threaded operation;
only the i/o originator can process the event.  so the plan 9
model would seem to me to be better threaded and i think
the CSP-style of the plan 9 model makes it easier to
reason about.  and has already been implemented under
unix without changing the kernel.  see p9p.

> Ideally, it is the programmer's informed choice
> based on their understanding of their application's priorities whether to
> use callbacks, listener/worker threads, message queues, or a combination.
> Someone may find it worth the effort to compare these approaches on a
> platform that provides both. (COM is notorious for implementing things
> through callback that get some wrapping of one's head around them before
> making sense.)

there's plenty of overlapped i/o in plan 9 — it's in the
disk and network device drivers.  even stripping away
the register fiddling, it's not an i/o model that's attractive;
that's the reason all the details are hidden in the kernel.

- erik

Re: [9fans] critique of sockets API

[Eris Discordia]

> i don't think i understand what you're getting at.
> it could be that the blog was getting at the fact that select
> funnels a bunch of independent i/o down to one process.
> it's an effective technique when (a) threads are not available
> and (b) processing is very fast.

This might help: what he is getting at is probably the question of why not
make possible network applications that consist of a bunch of callbacks or
a mix of callbacks and listener/worker threads. Windows implements both
synchronous and asynchronous I/O. Threads are available. Callbacks, too, as
well as message queues. Ideally, it is the programmer's informed choice
based on their understanding of their application's priorities whether to
use callbacks, listener/worker threads, message queues, or a combination.
Someone may find it worth the effort to compare these approaches on a
platform that provides both. (COM is notorious for implementing things
through callback that get some wrapping of one's head around them before
making sense.)

--On Tuesday, June 09, 2009 20:07 -0400 erik quanstrom
<quanstro@quanstro.net> wrote:

> On Tue Jun  9 19:22:39 EDT 2009, bpisupat@cs.indiana.edu wrote:
>> Well, select() or alt might or might not be required depending on
>> whether  you want your thread to wait till the read operation waiting
>> for data from the network completes.
>
> your thread will always wait until any system call completes;
> they're all synchronous all the time.  if you want your application
> to do something else at the same time, you're going to need two
> threads and a synch device (like a lock + shared memory or a channel).
>
>> read()->process()->read()... alternating sequence of operations that is
>> required, wherein the application has to explicitly go fetch data from
>> the network  using the read operation. To borrow text from the paper:
>> <snip>
>> The API does not provide the programmer a way in which to say, "Whenever
>> there is data for me, call me to process it directly."
>> </snip>
>
> i don't think i understand what you're getting at.
> it could be that the blog was getting at the fact that select
> funnels a bunch of independent i/o down to one process.
> it's an effective technique when (a) threads are not available
> and (b) processing is very fast.
>
> perhaps you think this is doging the question, but the cannonical
> plan 9 approach to this is to note that it's easy (trivial) to have a n
> reader threads and m worker threads s.t. i/o threads don't block
> unless (a) there's nothing to i/o, or (b) the workers aren't draining
> the queue fast enough; and s.t. worker threads don't block
> unless (a) the i/o threads can't keep up.  in this case, there is no
> work to do anyway.  consider these two types of threads;
> let mb be a pointer to a message buffer.
>
> thread type 1
> 	for(;;)
> 		mb <- freechan
> 		read(fd, mb->wp, BUFSIZE);
> 		mb -> fullchan
> thread type 2;
> 	for(;;)
> 		mb <- fullchan
> 		do stuff
> 		mb -> freechan
>
> if your server issues responses, it's easy to add thread type 3.
>
> as you can see, this is a simple generalization of your case.  (if we
> have a queue depth of 0 and one thread 1 and one thread 2,
> we will get your loop.)  yet there should be no waiting.
>
>> The question was meant to ask as to how easy it is to programmatically
>> use the filesystem interface in a multi home network. But I agree that
>> support for  multiple network interfaces in Plan9 is way superior.
>
> i think the answer to your question is that programs don't care.
> programs that take a cannonical network address are just as
> happy to accept /net/tcp!www.example.com!http as they are to
> accept /net.alt/tcp!www.example.com!http.  for a short while
> i ran a 10gbit network on /net.10g.
>
> - erik
>

Re: [9fans] critique of sockets API

[Bhanu Nagendra Pisupati]

> perhaps you think this is doging the question, but the cannonical
> plan 9 approach to this is to note that it's easy (trivial) to have a n
> reader threads and m worker threads s.t. i/o threads don't block

I'll agree. With multi threading the network read/write operations
could be pipelined to minimize the overhead.

Re: [9fans] critique of sockets API

[Bhanu Nagendra Pisupati]

> Did you do this on Plan 9 or bring some of the filesystem sanity of another OS?

Actually my work was based on 9P based synthetic filesystems
implemented using Npfs (see
http://docs.google.com/Doc?id=dcb7zf48_503q8j84)

These filesystems resided in resource constrained embedded devices which
could not run Plan 9 in whole.

Re: [9fans] critique of sockets API

[Gary Wright]

On Jun 9, 2009, at 7:33 PM, Devon H. O'Dell wrote:
> I think you mean `networks with multiple gateways'? I don't see how
> Plan 9's model would affect this idea one way or the other, but
> perhaps I (or all of us) misunderstand this part of your questions.


Based on scanning the original article, I think the question was how
Plan 9 could support protocols such as: SCTP (Stream Control
Transport Protocol. In SCTP a communication endpoint is represented by
more than one IP address.

Gary Wright

Re: [9fans] critique of sockets API

[erik quanstrom]

> i don't think i understand what you're getting at.
> it could be that the blog was getting at the fact that select
> funnels a bunch of independent i/o down to one process.
> it's an effective technique when (a) threads are not available
> and (b) processing is very fast.

ack.  but, you can select on the same fd in many processes.
select usage is so idiomatic that this use is easy to forget,
though i've written programs that take advantage of this myself.

- erik

Re: [9fans] critique of sockets API

[erik quanstrom]

On Tue Jun  9 19:22:39 EDT 2009, bpisupat@cs.indiana.edu wrote:
> Well, select() or alt might or might not be required depending on whether
> you want your thread to wait till the read operation waiting
> for data from the network completes.

your thread will always wait until any system call completes;
they're all synchronous all the time.  if you want your application
to do something else at the same time, you're going to need two
threads and a synch device (like a lock + shared memory or a channel).

> read()->process()->read()... alternating sequence of operations that is
> required, wherein the application has to explicitly go fetch data from the network
> using the read operation. To borrow text from the paper:
> <snip>
> The API does not provide the programmer a way in which to say, "Whenever
> there is data for me, call me to process it directly."
> </snip>

i don't think i understand what you're getting at.
it could be that the blog was getting at the fact that select
funnels a bunch of independent i/o down to one process.
it's an effective technique when (a) threads are not available
and (b) processing is very fast.

perhaps you think this is doging the question, but the cannonical
plan 9 approach to this is to note that it's easy (trivial) to have a n
reader threads and m worker threads s.t. i/o threads don't block
unless (a) there's nothing to i/o, or (b) the workers aren't draining
the queue fast enough; and s.t. worker threads don't block
unless (a) the i/o threads can't keep up.  in this case, there is no
work to do anyway.  consider these two types of threads;
let mb be a pointer to a message buffer.

thread type 1
	for(;;)
		mb <- freechan
		read(fd, mb->wp, BUFSIZE);
		mb -> fullchan
thread type 2;
	for(;;)
		mb <- fullchan
		do stuff
		mb -> freechan

if your server issues responses, it's easy to add thread type 3.

as you can see, this is a simple generalization of your case.  (if we
have a queue depth of 0 and one thread 1 and one thread 2,
we will get your loop.)  yet there should be no waiting.

> The question was meant to ask as to how easy it is to programmatically
> use the filesystem interface in a multi home network. But I agree that support for
> multiple network interfaces in Plan9 is way superior.

i think the answer to your question is that programs don't care.
programs that take a cannonical network address are just as
happy to accept /net/tcp!www.example.com!http as they are to
accept /net.alt/tcp!www.example.com!http.  for a short while
i ran a 10gbit network on /net.10g.

- erik

Re: [9fans] critique of sockets API

[Devon H. O'Dell]

>> i have no idea how this relates to the use of a fs in implementing the
>> network stack.  why would using a filsystem (or not) make any difference
>> in the ability to multihome?
>>
>> by the way, plan 9 beats the pants off anything else i've used for
>> multiple
>> network / interface support.  it's support for mulitple ip stacks is quite
>> neat.
>
> The question was meant to ask as to how easy it is to programmatically use
> the filesystem interface in a multi home network. But I agree that support
> for multiple network interfaces in Plan9 is way superior.

I think you mean `networks with multiple gateways'? I don't see how
Plan 9's model would affect this idea one way or the other, but
perhaps I (or all of us) misunderstand this part of your questions.

--me

Re: [9fans] critique of sockets API

[J.R. Mauro]

On Tue, Jun 9, 2009 at 7:20 PM, Bhanu Nagendra
Pisupati<bpisupat@cs.indiana.edu> wrote:
> First off, I really am a big fan of filesystem interfaces as used in Plan 9
> - after all my PhD work was based on the model :)

Did you do this on Plan 9 or bring some of the filesystem sanity of another OS?

> My objective here is to debate and understand how the points made in the
> paper relate to the Plan9 networking model.
>
>>> * performance overhead: app requesting data from a socket typically needs
>>> to perform 2 system calls (select/read or alt/read)
>>
>> alt ? which is not required ? is not a system call.  only a read or write
>> is
>> required.
>
> Well, select() or alt might or might not be required depending on whether
> you want your thread to wait till the read operation waiting for data from
> the network completes. You may argue that since threads are "cheap" in Plan9
> you can afford to have a thread wait on the read operation. But that to me
> is a different question...
>
>>> * lack of an "kernel up-call API": which allows the kernel to inform an
>>> app each time network data is available
>>
>> plan 9 uses synchronous i/o, so this statement doesn't make sense
>> in plan 9.  you can use threads to do i/o asynch w.r.t. your application,
>> but the i/o itself is still synchronous w.r.t. the kernel.
>
> Whether the IO is synchronous or not, there is this
> read()->process()->read()... alternating sequence of operations that is
> required, wherein the application has to explicitly go fetch data from the
> network using the read operation. To borrow text from the paper:
> <snip>
> The API does not provide the programmer a way in which to say, "Whenever
> there is data for me, call me to process it directly."
> </snip>
>
>
>>> * hard to implement "multi homing" with support for multiple network
>>> interfaces
>>
>> i have no idea how this relates to the use of a fs in implementing the
>> network stack.  why would using a filsystem (or not) make any difference
>> in the ability to multihome?
>>
>> by the way, plan 9 beats the pants off anything else i've used for
>> multiple
>> network / interface support.  it's support for mulitple ip stacks is quite
>> neat.
>
> The question was meant to ask as to how easy it is to programmatically use
> the filesystem interface in a multi home network. But I agree that support
> for multiple network interfaces in Plan9 is way superior.
>
>

Re: [9fans] critique of sockets API

[Bhanu Nagendra Pisupati]

First off, I really am a big fan of filesystem interfaces as used in Plan
9 - after all my PhD work was based on the model :)
My objective here is to debate and understand how the points made in the
paper relate to the Plan9 networking model.

>> * performance overhead: app requesting data from a socket typically
>> needs to perform 2 system calls (select/read or alt/read)
>
> alt ? which is not required ? is not a system call.  only a read or write is
> required.

Well, select() or alt might or might not be required depending on whether
you want your thread to wait till the read operation waiting
for data from the network completes. You may argue that since threads are
"cheap" in Plan9 you can afford to have a thread wait on the read
operation. But that to me is a different question...

>> * lack of an "kernel up-call API": which allows the kernel to inform an
>> app each time network data is available
>
> plan 9 uses synchronous i/o, so this statement doesn't make sense
> in plan 9.  you can use threads to do i/o asynch w.r.t. your application,
> but the i/o itself is still synchronous w.r.t. the kernel.

Whether the IO is synchronous or not, there is this
read()->process()->read()... alternating sequence of operations that is
required, wherein the application has to explicitly go fetch data from the network
using the read operation. To borrow text from the paper:
<snip>
The API does not provide the programmer a way in which to say, "Whenever
there is data for me, call me to process it directly."
</snip>


>> * hard to implement "multi homing" with support for multiple network
>> interfaces
>
> i have no idea how this relates to the use of a fs in implementing the
> network stack.  why would using a filsystem (or not) make any difference
> in the ability to multihome?
>
> by the way, plan 9 beats the pants off anything else i've used for multiple
> network / interface support.  it's support for mulitple ip stacks is quite
> neat.

The question was meant to ask as to how easy it is to programmatically
use the filesystem interface in a multi home network. But I agree that support for
multiple network interfaces in Plan9 is way superior.