[9fans] Why do we need syspipe() ?

[9fans] Why do we need syspipe() ?

[Roman V. Shaposhnik]

I'm confused. Is there any part of syspipe() that can NOT
be done from userspace? Why does it have to be a syscall?

Thanks,
Roman.

P.S. I would also argue that sysdup() would seem to be superfluous
if more feature-rich devdup was available.

Re: [9fans] Why do we need syspipe() ?

[Russ Cox]

On Sat, Jan 3, 2009 at 9:04 PM, Roman V. Shaposhnik <rvs@sun.com> wrote:
> I'm confused. Is there any part of syspipe() that can NOT
> be done from userspace? Why does it have to be a syscall?
>
> P.S. I would also argue that sysdup() would seem to be superfluous
> if more feature-rich devdup was available.

I don't believe you can write a race-free implementation of
the pipe system call using #|.

I also don't believe you can implement the dup system call
(remember, it has two arguments) using #d.

If you disagree, show me the code.

Russ

Re: [9fans] Why do we need syspipe() ?

[Roman V. Shaposhnik]

On Sat, 2009-01-03 at 22:56 -0800, Russ Cox wrote:
> On Sat, Jan 3, 2009 at 9:04 PM, Roman V. Shaposhnik <rvs@sun.com> wrote:
> > I'm confused. Is there any part of syspipe() that can NOT
> > be done from userspace? Why does it have to be a syscall?
> >
> > P.S. I would also argue that sysdup() would seem to be superfluous
> > if more feature-rich devdup was available.
>
> I don't believe you can write a race-free implementation of
> the pipe system call using #|.

Could you, please, elaborate on what particular race do you have
in mind? Indeed, I ran into a problem with devpipe implementation,
but it isn't a race, its a dreaded implicit ->attach that namec()
does when it evaluates names with the first character being #.

> I also don't believe you can implement the dup system call
> (remember, it has two arguments) using #d.

Agreed. That's why I mentioned that a more feature-rich devdup
is needed. Of course, now I've also discovered that the current
implementation of devpipe is also not sufficient enough for me
to be able to produce a 100% user-space version of pipe(2).

> If you disagree, show me the code.

I can't :-( Sorry for the noise. With the current implementation
my question has a definite answer of why Plan9 kernel needs a syspipe().

If I'm allowed to change the implementation of devpipe, I think
I can show you the code. Would that still count?

Thanks,
Roman.

Re: [9fans] Why do we need syspipe() ?

[Russ Cox]

>> I don't believe you can write a race-free implementation of
>> the pipe system call using #|.
>
> Could you, please, elaborate on what particular race do you have
> in mind? Indeed, I ran into a problem with devpipe implementation,
> but it isn't a race, its a dreaded implicit ->attach that namec()
> does when it evaluates names with the first character being #.

The closest you can come in user space to implementing pipe is:

int
pipe(int *fd)
{
	bind("#|", "/mnt", MREPL);
	fd[0] = open("/mnt/data", ORDWR);
	fd[1] = open("/mnt/data1", ORDWR);
	unmount("/mnt");
	return 0;
}

but if there are multiple processes running pipe()
in the same name space, the binds will step
on each other and the pipes might get crossed.
Even if not, maybe something else was already
mounted on /mnt (or whatever mount point you
choose), and now there's nothing there.

>> I also don't believe you can implement the dup system call
>> (remember, it has two arguments) using #d.
>
> Agreed. That's why I mentioned that a more feature-rich devdup
> is needed. Of course, now I've also discovered that the current
> implementation of devpipe is also not sufficient enough for me
> to be able to produce a 100% user-space version of pipe(2).

Sorry, I thought you were saying that #d was already more
feature rich than dup (it is, in a way, since it has the ctl files now).
I was trying to say that although that is true, it doesn't have the
dup features.

There are some devices in Plan 9 that simply don't "virtualize",
because at a deep level they are tied to process state that
doesn't go through the file system.  Dup manipulates the file
descriptor table, not files themselves.  Pipe accesses files that
have no name in the file system.   The pid returned by getpid
needs to match the pid returned by the parent's fork; it really
needs to be the process's actual pid.  For example, suppose
a process wants to know .  If getpid read from /dev/pid
instead of #c/pid, then running "iostats rc -c 'echo $pid'"
would show iostats's pid, not rc's.  What then if rc wants to send
itself (or, more likely, its note group) a note, or fiddle with
one of its /proc files?  It would be manipulating iostats, not
itself.

A write to devsrv is even more magical: when you write "23"
to #s/newfile, your process's fd 23 gets taken over by the
kernel.  For this reason you can't use iostats on any program
that writes to /srv/newfile instead of #s/newfile--when the program
writes "23", the kernel sees the request come from iostats
instead of the original program, and it takes over the wrong fd.
(Most of those programs are 9P servers that fork into the
background, and iostats isn't too useful on those anyway,
so no one has bothered to address this.)

The tls device and ssl devices #a and #D use the same trick,
so you can't interpose on traffic destined to them.  Happily,
the libraries use #a and #D directly, so using iostats on them
simply misses that i/o rather than causing the program to
execute incorrectly.

You could add a special message to #d to make the dup
system call unnecessary, but it wouldn't be any cleaner than
having the system call, since you'd have to hard code #d
instead of using /fd, or else you'd have the same problems
as #s, #a, and #D already do.

The # device syntax is very useful to mean the kernel device
and none other in these situations.  There's definitely
something unsatisfactory about it, but it works.

Russ

Re: [9fans] Why do we need syspipe() ?

[lucio]

> The # device syntax is very useful to mean the kernel device
> and none other in these situations.  There's definitely
> something unsatisfactory about it, but it works.

Smacks of overloading; at the very least it seems to suggest that
there ought to be official sanction as well as documented situations
exactly as you described.  To pepper the libraries with direct calls
to the kernel drivers is likely to cause consternation at the very
minimum.  I appreciate that a solution isn't yet obvious and I hope
that further thought will eventually lead to the necessary elegance.

++L

Re: [9fans] Why do we need syspipe() ?

[roger peppe]

i've sometimes thought that the trick used by #d etc could
be made more transparent by providing a genuine capability
service for fds, in the form of a system call, for instance

  getfdcap(int fd, char *buf, int len)

then instead of just writing the fd itself, you'd write
the capability - thus the write can bridge several
namespaces, as long as it ends up in the same kernel,
which can then utilise the capability.

the fact that this also provides the possibility of implementing
sendfd might or might not be an advantage.

Re: [9fans] Why do we need syspipe() ?

[Nathaniel W Filardo]

[-- Attachment #1: Type: text/plain, Size: 1183 bytes --]

On Sun, Jan 04, 2009 at 10:20:55PM -0800, Russ Cox wrote:
> There are some devices in Plan 9 that simply don't "virtualize",
> because at a deep level they are tied to process state that
> doesn't go through the file system.  Dup manipulates the file
> descriptor table, not files themselves.  Pipe accesses files that
> have no name in the file system.   The pid returned by getpid
> needs to match the pid returned by the parent's fork; it really
> needs to be the process's actual pid.  For example, suppose
> a process wants to know .  If getpid read from /dev/pid
> instead of #c/pid, then running "iostats rc -c 'echo $pid'"
> would show iostats's pid, not rc's.  What then if rc wants to send
> itself (or, more likely, its note group) a note, or fiddle with
> one of its /proc files?  It would be manipulating iostats, not
> itself.
>
>[snip #s #a and #D]

This just means that these services need to be mounted at the canonical
place in the namepsace atop the root provided by iostats.  That yields
equivalent behavior -- the kernel sees the right process making the call and
iostats sees nothing at all -- but it is, I agree, unsatisfactory.

--nwf;

[-- Attachment #2: Type: application/pgp-signature, Size: 204 bytes --]

Re: [9fans] Why do we need syspipe() ?

[Charles Forsyth]

>This just means that these services need to be mounted at the canonical

there is no point binding #a or #D into the name space.
they can be used only locally and might as well
be accessed directly.  they might be considered similar to
"push" in streams.  #s has a similar difficulty to #a and #D when creating the names,
and requires special hacks to make existing files usable when exported.

Re: [9fans] Why do we need syspipe() ?

[Roman V. Shaposhnik]

On Tue, 2009-01-06 at 20:37 +0000, Charles Forsyth wrote:
> >This just means that these services need to be mounted at the canonical
>
> there is no point binding #a or #D into the name space.
> they can be used only locally and might as well
> be accessed directly.  they might be considered similar to
> "push" in streams.  #s has a similar difficulty to #a and #D when creating the names,
> and requires special hacks to make existing files usable when exported.

Well, that's been the case for as long as Plan9 existed, and I don't
want to say that it doesn't work.

Although in the alternative universe I can see how implementing #X
as *channels* capable of 9P messages, could enable things like mounting
them on external hosts and letting these hosts manipulate physical
devices attached to yours (I agree that remote mounting of the kernel
services, which do not correspond to physical devices, is less useful).
Just like exporting my *local* /srv can be a useful things at times.

Thanks,
Roman.

Re: [9fans] Why do we need syspipe() ?

[erik quanstrom]

> Although in the alternative universe I can see how implementing #X
> as *channels* capable of 9P messages, could enable things like mounting
> them on external hosts and letting these hosts manipulate physical
> devices attached to yours (I agree that remote mounting of the kernel
> services, which do not correspond to physical devices, is less useful).
> Just like exporting my *local* /srv can be a useful things at times.

you mean like this?
	import -E ssl minooka.coraid.com '#æ' /n/minookaæ

- erik

Re: [9fans] Why do we need syspipe() ?

[lucio]

> mounting
> them on external hosts and letting these hosts manipulate physical
> devices attached to yours

That's the function of "import/export" and is one of Plan 9's strong
suits.  Surely I cannot have been fooled all these years?

++L

Re: [9fans] Why do we need syspipe() ?

[Roman Shaposhnik]

On Jan 4, 2009, at 10:20 PM, Russ Cox wrote:
>>> I don't believe you can write a race-free implementation of
>>> the pipe system call using #|.
>>
>> Could you, please, elaborate on what particular race do you have
>> in mind? Indeed, I ran into a problem with devpipe implementation,
>> but it isn't a race, its a dreaded implicit ->attach that namec()
>> does when it evaluates names with the first character being #.
>
> The closest you can come in user space to implementing pipe is:
>
> int
> pipe(int *fd)
> {
> 	bind("#|", "/mnt", MREPL);
> 	fd[0] = open("/mnt/data", ORDWR);
> 	fd[1] = open("/mnt/data1", ORDWR);
> 	unmount("/mnt");
> 	return 0;
> }
>
> but if there are multiple processes running pipe()
> in the same name space, the binds will step
> on each other and the pipes might get crossed.
> Even if not, maybe something else was already
> mounted on /mnt (or whatever mount point you
> choose), and now there's nothing there.

Well, strictly speaking, there's another possibility:

int
pipe(int *fd)
{
        chdir("#|");
        fd[0] = open("data", ORDWR);
        fd[1] = open("data1", ORDWR);
        return 0;
}

which avoids a race, but trashes your current directory.

On related note: it is a bit sad, though, that I can't stash
a channel to the current directory away in a file
descriptor and use it instead of a symbolic name
in a call similar to chdir. I can amost do it:
     int stash_cwd = open(".", ORDWR);
but not quite :-(

>>>
> There are some devices in Plan 9 that simply don't "virtualize",
> because at a deep level they are tied to process state that
> doesn't go through the file system.

Agreed. I think it would be fair to call them drivers for the
kernel services. As opposed to drivers for physical devices.

> Dup manipulates the file descriptor table, not files themselves.
> Pipe accesses files that have no name in the file system.

I have always thought of pipes as in-kernel buffers. That's
a service the kernel provides. #| is just a driver for that service.

> For example, suppose a process wants to know .  If getpid read
> from /dev/pid
> instead of #c/pid, then running "iostats rc -c 'echo $pid'"
> would show iostats's pid, not rc's.  What then if rc wants to send
> itself (or, more likely, its note group) a note, or fiddle with
> one of its /proc files?  It would be manipulating iostats, not
> itself.

This is a very good point. Thank you for bringing it up. I now
see how being able to *always* go directly to the  #c/pid
makes things easier implementation-wise. And may be
it is a good-enough justification for #X.  But before I agree
100%  and shut up ;-) here's one though: isn't the
situation here,  to quote Charles, rather simple: you asked
for fish; you get fish? If iostats tries to interpose on everything
in the filesystem -- well, may be that should be *everything*.
Not ifs or buts. And yes it would mean the weird behaviour
of rc you've alluded to. Because it *is* everything.

> A write to devsrv is even more magical: when you write "23"
> to #s/newfile, your process's fd 23 gets taken over by the
> kernel.  For this reason you can't use iostats on any program
> that writes to /srv/newfile instead of #s/newfile--when the program
> writes "23", the kernel sees the request come from iostats
> instead of the original program, and it takes over the wrong fd.
> (Most of those programs are 9P servers that fork into the
> background, and iostats isn't too useful on those anyway,
> so no one has bothered to address this.)

Right. But its not that the problem is insurmountable. Its just that,
literally, nobody bothered to complicate the implementation
of iostats.


> The # device syntax is very useful to mean the kernel device
> and none other in these situations.  There's definitely
> something unsatisfactory about it, but it works.


Once again -- thanks a million for a very clear post. It is now
obvious that #X has more, how shall I put it, capabilities
than a pure namespace-based approach. It really wasn't
obvious to me before.

Thanks,
Roman.

Re: [9fans] Why do we need syspipe() ?

[Charles Forsyth]

the example that prompted the `fish' remark was: cd location; pwd shows location.

Re: [9fans] Why do we need syspipe() ?

[Roman V. Shaposhnik]

On Mon, 2009-01-05 at 11:00 +0000, roger peppe wrote:
> i've sometimes thought that the trick used by #d etc could
> be made more transparent by providing a genuine capability
> service for fds, in the form of a system call, for instance
>
>   getfdcap(int fd, char *buf, int len)
>
> then instead of just writing the fd itself, you'd write
> the capability - thus the write can bridge several
> namespaces, as long as it ends up in the same kernel,
> which can then utilise the capability.
>
> the fact that this also provides the possibility of implementing
> sendfd might or might not be an advantage.

In light of the recent discussion, I now see how providing this
basic capability could be a very useful building block for the
rest of the system.

Not in a sense, that it'll be plugging a particular gaping hole,
but rather that thrown into the mix, it might help foster better
ideas/implementations.

Thanks,
Roman.

Re: [9fans] Why do we need syspipe() ?

[Roman V. Shaposhnik]

On Tue, 2009-01-06 at 18:15 -0500, erik quanstrom wrote:
> > Although in the alternative universe I can see how implementing #X 
> > as *channels* capable of 9P messages, could enable things like mounting
> > them on external hosts and letting these hosts manipulate physical
> > devices attached to yours (I agree that remote mounting of the kernel
> > services, which do not correspond to physical devices, is less useful).
> > Just like exporting my *local* /srv can be a useful things at times.
> 
> you mean like this?
> 	import -E ssl minooka.coraid.com '#æ' /n/minookaæ

Well, I tend not to like the proliferation of exportfs', but may
be in this case it is actually better that what I had in mind.

Thanks,
Roman.

Re: [9fans] Why do we need syspipe() ?

[erik quanstrom]

>> > Although in the alternative universe I can see how implementing #X
>> > as *channels* capable of 9P messages, could enable things like mounting
>> > them on external hosts and letting these hosts manipulate physical
>> > devices attached to yours (I agree that remote mounting of the kernel
>> > services, which do not correspond to physical devices, is less useful).
>> > Just like exporting my *local* /srv can be a useful things at times.
>>
>> you mean like this?
>> 	import -E ssl minooka.coraid.com '#æ' /n/minookaæ
>
> Well, I tend not to like the proliferation of exportfs', but may
> be in this case it is actually better that what I had in mind.

why?

- erik

Re: [9fans] Why do we need syspipe() ?

[Roman V. Shaposhnik]

On Thu, 2009-01-08 at 18:48 -0500, erik quanstrom wrote:
> >> > Although in the alternative universe I can see how implementing #X 
> >> > as *channels* capable of 9P messages, could enable things like mounting
> >> > them on external hosts and letting these hosts manipulate physical
> >> > devices attached to yours (I agree that remote mounting of the kernel
> >> > services, which do not correspond to physical devices, is less useful).
> >> > Just like exporting my *local* /srv can be a useful things at times.
> >> 
> >> you mean like this?
> >> 	import -E ssl minooka.coraid.com '#æ' /n/minookaæ
> > 
> > Well, I tend not to like the proliferation of exportfs', but may
> > be in this case it is actually better that what I had in mind.
> 
> why?

Multiplexing. If devices exposed channel interface, and got
exported there would be no kernel protecting from clients
sending random sequences of 9P messages (on a single host
you can't mount a channel and then continue reading/writing
9P messages over it).

So the way devices are right now, actually seems to be better
compared to what I have in mind. Sorry for the noise.

Thanks,
Roman.

Re: [9fans] Why do we need syspipe() ?

[roger peppe]

On Fri, Jan 9, 2009 at 2:05 AM, Roman V. Shaposhnik <rvs@sun.com> wrote:
> Multiplexing. If devices exposed channel interface, and got
> exported there would be no kernel protecting from clients
> sending random sequences of 9P messages (on a single host
> you can't mount a channel and then continue reading/writing
> 9P messages over it).

a year or so back i mentioned a way that this could be done,
by having an "auth" (DMAUTH) file visible in the namespace representing the
channel to be mounted. you can turn this file into a new namespace
by opening it and handing it to an attach message (as the afid).

i've probably still got a working inferno kernel that i modified to
allow this, but i didn't take it any further.

this mechanism *could* potentially be used to replace the # namespace.