Re: [9fans] read/write offset hack

Re: [9fans] read/write offset hack

[erik quanstrom]

> On Fri, May 30, 2008 at 4:15 PM, erik quanstrom <quanstro@coraid.com> wrote:
> > why not put the timing information inband?  this would allow the timed
> > sound to be saved to a file also without 1.7mb of zeros.
>
> because then you'd need to quote the audio data in case it
> contained something that looked like a timing command?

if the size of the chunks is not obvious, say due to compression,
run-length encoding would solve that problem.

- erik

Re: [9fans] read/write offset hack

[Stefan Hajnoczi]

Excellent, excellent response Russ.  If a question like this comes up
again, your post is a great reference.

Stefan

On Fri, May 30, 2008 at 4:35 PM, erik quanstrom <quanstro@coraid.com> wrote:
>> On Fri, May 30, 2008 at 4:15 PM, erik quanstrom <quanstro@coraid.com> wrote:
>> > why not put the timing information inband?  this would allow the timed
>> > sound to be saved to a file also without 1.7mb of zeros.
>>
>> because then you'd need to quote the audio data in case it
>> contained something that looked like a timing command?
>
> if the size of the chunks is not obvious, say due to compression,
> run-length encoding would solve that problem.
>
> - erik
>
>

Re: [9fans] read/write offset hack

[Nyang A. Phra]

On May 30, 6:06 pm, r...@swtch.com (Russ Cox) wrote:
> If you want to violate a convention, Plan 9 won't stop you,
> but in doing so you give up compatibility with programs that
> depend on that convention (bind /net/tcp /proc; ps).
> Sure, you could replace ctl and clone and other special files with reads
> and writes at magic offsets, but in doing so you give up accessing
> those files with standard programs like echo and cat.  For me,
> the main benefit of user-level file servers is exactly that I can
> interact with them from a wide variety of programs, including
> scripts and interactive shell sessions.  I'd need a pretty compelling
> reason before I gave that up.

Thanks Russ for such a thorough and pragmatic response, I'm happy to
have learned something :)

Nyang

Re: [9fans] read/write offset hack

[Roman Shaposhnik]

On Fri, 2008-05-30 at 15:04 -0400, a@9srv.net wrote:
> Very well explained. I've seen this question come up lots of times
> when introducing fs-based interfaces to people. When we had some
> off-shore devs to bring up to speed, they kept coming back to it: "so
> can we just define a protocol to put all these streams in one file?". I
> tried to explaing that having them as distinct files *was* that
> protocol, but it never stuck. I wish I'd had this explanation then.

I believe that part of the difficulty might come from the fact that
Plan9 is exceptional in a sense that it offers an extremely effective
environment in a basic package. You don't have to devise complicated
GUI "shells" to be productive. Acme is as fancy as it gets.
Unfortunately not everybody is lucky enough to be able to run Plan9
environment. People come from backgrounds of different level of
suckiness. The environments where GUI "shells" are a necessity
(did you know both KDE and GNOME implement incompatible filesystems
at the level of their Explorers?). That's why these folks never trust
their basic environment. They never expect the tools that come with
it to be of any significant use. Their default reasoning is always
to add more "protocols" to the bloated monsters they call "GUI shells".
They just don't know any better.

Thanks,
Roman.

Re: [9fans] read/write offset hack

[a]

Very well explained. I've seen this question come up lots of times
when introducing fs-based interfaces to people. When we had some
off-shore devs to bring up to speed, they kept coming back to it: "so
can we just define a protocol to put all these streams in one file?". I
tried to explaing that having them as distinct files *was* that
protocol, but it never stuck. I wish I'd had this explanation then.
Anthony

Re: [9fans] read/write offset hack

[Robert William Fuller]

Russ Cox wrote:

<snip>

> If you want to violate a convention, Plan 9 won't stop you,
> but in doing so you give up compatibility with programs that
> depend on that convention (bind /net/tcp /proc; ps).
> Sure, you could replace ctl and clone and other special files with reads
> and writes at magic offsets, but in doing so you give up accessing
> those files with standard programs like echo and cat.  For me,
> the main benefit of user-level file servers is exactly that I can
> interact with them from a wide variety of programs, including
> scripts and interactive shell sessions.  I'd need a pretty compelling
> reason before I gave that up.

Thank you.  Posts like these are the reason I follow this list, even
though I do not have Plan9 installed!  (I have an interest in operating
system design and a background in filesystems.)

Re: [9fans] read/write offset hack

[ron minnich]

On Fri, May 30, 2008 at 1:43 AM, Nyang A. Phra <naphra@gmail.com> wrote:
> Poking around Plan 9 and 9P, I was wondering whether it would be a
> neat hack or some sort of abuse to read and write dynamically served
> files at different offsets to get different semantics, instead of
> reading and writing different files (ctl, clone, etc.) to do that.

it's a step backwards in time to very old ideas. And it's a terrible
idea. Heed your spider sense.

Just consider the QID  as your differentiator.

ron

Re: [9fans] read/write offset hack

[Russ Cox]

> why not put the timing information inband?  this would allow the timed
> sound to be saved to a file also without 1.7mb of zeros.

The Plan 9 audio device data format is stereo, 16-bit
little-endian PCM samples.  Given that convention, my
point was only that seeking in that kind of data stream
does make a little bit of sense.  The whole mail was
about the power of working within and respecting
existing conventions, not about redesigning the audio device.

Others might choose different conventions in different situations.
For example, I believe Nemo's Plan B audio device accepts
MP3 format audio to reduce bandwidth requirements for
sending audio to devices elsewhere on the network.
The Plan 9 audio format has the benefit of being close to
the native format for most hardware.

As always, your own context dictates whether or not
breaking established conventions is a good idea.  Of course,
the Sparc /dev/audio is not just a good idea: it's μ-law.

Russ

Re: [9fans] read/write offset hack

[roger peppe]

On Fri, May 30, 2008 at 4:15 PM, erik quanstrom <quanstro@coraid.com> wrote:
> why not put the timing information inband?  this would allow the timed
> sound to be saved to a file also without 1.7mb of zeros.

because then you'd need to quote the audio data in case it
contained something that looked like a timing command?

Re: [9fans] read/write offset hack

[erik quanstrom]

> P. P. S.  The usb audio use of offsets is not as bad as it first sounds.
> The device consumes written data at a constant rate (say, 176,400
> bytes per second for CD audio).  You can make a noise ten seconds
> from now by writing 1.7MB bytes of zeros (silence) followed by your
> sound data.  Being able to seek ahead 1.7MB instead just avoids the
> need to write the zeros.  The file offset is still bytes, not nanoseconds.
>

why not put the timing information inband?  this would allow the timed
sound to be saved to a file also without 1.7mb of zeros.

- erik

Re: [9fans] read/write offset hack

[Russ Cox]

> Poking around Plan 9 and 9P, I was wondering whether it would be a
> neat hack or some sort of abuse to read and write dynamically served
> files at different offsets to get different semantics, instead of
> reading and writing different files (ctl, clone, etc.) to do that.
>
> Given that the system encourages to perceive files as having arbitrary
> semantics (as opposed to having regular sequential file semantics) it
> would make sense (to me) to have reads and writes at arbitrary offsets
> to have arbitrary semantics as well -- that's, after all, what offset
> (kind of) does on a regular file, too, although in a rather trivial
> way.

The semantics of files on Plan 9 are not as ad hoc as you suggest.

In Unix there is essentially one file server, and thus essentially
one file interface.  (There are of course little differences;
for example, can you cat a directory?)

In Plan 9 there are many different file servers, but there are
relatively few different file interfaces.  The paucity of distinct
interfaces is essential to being able to reuse common tools
across multiple file systems.  Of course, each file server does
implement some special behavior (that's why it exists) but
the most effective ones implement those behaviors in a way
that existing tools continue to work.

A network directory is a special kind of tree, but the
many file servers that provide networks agree on a single
interface.  Programs that dial(3) work just as well with
the kernel-provided /net/tcp, /net/il, and /net/ether0
as they do with drawterm's networks (bind /mnt/term/net/tcp /net/tcp)
or with programs like sshnet.  If every network server used
a different interface, every program that used the network
would have to be changed when a new kind of network
was introduced.  (In Unix they do, and it's very frustrating!)

File servers are of course free to introduce whole new
intrfaces, but they usually do it in a way that simple
programs can continue to use their files.  For example,
even though some network directory files have special
semantics, the /net/xxx/0/data file behaves no different
from a pipe, so that programs that can talk to a pipe can
talk to the network, assuming some other program sets up
the connection.

As another example, upas/fs presents your mail box as a file tree.
The mail readers--nedmail and Mail--are deeply aware of
the structure of the tree, but mail-ignorant programs can
still access the files.  Upas/fs would be much less useful if
	file /mail/fs/mbox/1/2/3/body.jpg
	tar c /mail/fs/mbox/1/2/3/body.jpg >/usr/rsc/jpg.tar
	lp /mail/fs/mbox/1/2/3/body.jpg
	page /mail/fs/mbox/1/2/3/body.jpg
did not work.  (In p9p on Unix they don't, and it's very frustrating!)

File servers sometimes have good reason to buck convention,
but doing so always comes at a cost.  For example, pipes and
network devices cannot provide random access, so they ignore
the file offset in reads and writes; in doing so they become
incompatible with programs that read the data multiple times
(like page in the above examples).  Of course, pipes and networks
are so common, on both Plan 9 and Unix, that many programs
avoid the assumption of random access when possible.

Another common violation is to return a file size of 0 in stat
even though reading the file would produce bytes.  File servers
do this because it is not always simple to compute the size
of dynamically-generated file contents; in doing so, they become
incompatible with programs that depend on the file size being
accurate.  If upas/fs returned a file size of 0 for body.jpg
(it doesn't), then the tar command above would not work
as well, because it needs to write the file size into the tar file
before it copies in the file contents.  (Why can't it seek back?
Because it avoids random access on its output for compatibility
with pipes and networks.)  Not-actually-zero-length files
are so common that many programs, on Plan 9 at least, avoid
the assumption that stat's length matches the length of read.

If you want to violate a convention, Plan 9 won't stop you,
but in doing so you give up compatibility with programs that
depend on that convention (bind /net/tcp /proc; ps).
Sure, you could replace ctl and clone and other special files with reads
and writes at magic offsets, but in doing so you give up accessing
those files with standard programs like echo and cat.  For me,
the main benefit of user-level file servers is exactly that I can
interact with them from a wide variety of programs, including
scripts and interactive shell sessions.  I'd need a pretty compelling
reason before I gave that up.

Russ


P. S.  I'll ignore the fact that you've also replaced the human-friendly
names with human-unfriendly magic numbers.)

P. P. S.  The usb audio use of offsets is not as bad as it first sounds.
The device consumes written data at a constant rate (say, 176,400
bytes per second for CD audio).  You can make a noise ten seconds
from now by writing 1.7MB bytes of zeros (silence) followed by your
sound data.  Being able to seek ahead 1.7MB instead just avoids the
need to write the zeros.  The file offset is still bytes, not nanoseconds.

Re: [9fans] read/write offset hack

[Nick LaForge]

>> Nyang: I must say one thing: you are simply going to LOVE an
>> abomination of an acme feature i am working on!
>
> Do let us in on the secret (if you wish) :)
>
>

lets discuss it as soon as i get the code out... watch this space ;)

nkl

Re: [9fans] read/write offset hack

[Nyang A. Phra]

> Nyang: I must say one thing: you are simply going to LOVE an
> abomination of an acme feature i am working on!

Do let us in on the secret (if you wish) :)

Re: [9fans] read/write offset hack

[roger peppe]

> Given that the system encourages to perceive files as having arbitrary
> semantics (as opposed to having regular sequential file semantics) it
> would make sense (to me) to have reads and writes at arbitrary offsets
> to have arbitrary semantics as well -- that's, after all, what offset
> (kind of) does on a regular file, too, although in a rather trivial
> way.

the nearest i've seen to this (abusing offset semantics)
in the system i've found in the usb device. from usb(3):

: S is the size of the DMA operations on the device (i.e., the
: minimum useful size for reads and writes), b is the number
: of bytes currently buffered for input or output, and o and t
: should be interpreted to mean that byte offset o was/will be
: reached at time t (nanoseconds since the epoch).  The data
: can be used to map file offsets to points in time.  To play
: or record samples exactly at some predetermined time, use o
: and t with the sampling rate to calculate the offset to seek
: to.

this works ok if you're always getting exactly the same number
of bytes per second, but wouldn't if you were streaming variable bit-rate
data.

the advantage of this over using a control file is that there's no
additional latency
or syscall overhead to exactly specifiying a sample time, which could make
a significant difference, i guess, particularly if you were using the device
over a network.

is this an abuse? i'm not sure, though rob pike definitely seemed to think
it was when i mentioned this to him at usenix one time.

Re: [9fans] read/write offset hack

[Nick LaForge]

>>> we haven't got a task force to rip
>>> keyboards from careless hands

>>this is excellent!

> but Nick LaForge reminds us we can use more effective methods

>       Vercotti: Doug (takes a drink) Well, I was terrified.
>       Everyone was terrified of Doug.  I've seen grown men pull
>       their own heads off rather than see Doug.  Even Dinsdale was
>       frightened of Doug.

>       2nd Interviewer: What did he do?

>       Vercotti: He used...  sarcasm.  He knew all the tricks,
>       dramatic irony, metaphor, bathos, puns, parody, litotes and...
>       satire.  He was vicious.

my apologies. :)

Nyang: I must say one thing: you are simply going to LOVE an
abomination of an acme feature i am working on!

Re: [9fans] read/write offset hack

[Charles Forsyth]

>> we haven't got a task force to rip
>> keyboards from careless hands

>this is excellent!

but Nick LaForge reminds us we can use more effective methods

	Vercotti: Doug (takes a drink) Well, I was terrified.
	Everyone was terrified of Doug.  I've seen grown men pull
	their own heads off rather than see Doug.  Even Dinsdale was
	frightened of Doug.

	2nd Interviewer: What did he do?

	Vercotti: He used...  sarcasm.  He knew all the tricks,
	dramatic irony, metaphor, bathos, puns, parody, litotes and...
	satire.  He was vicious.

Re: [9fans] read/write offset hack

[Nick LaForge]

> Poking around Plan 9 and 9P, I was wondering whether it would be a
> neat hack or some sort of abuse to read and write dynamically served
> files at different offsets to get different semantics, instead of
> reading and writing different files (ctl, clone, etc.) to do that.

> Given that the system encourages to perceive files as having arbitrary
> semantics (as opposed to having regular sequential file semantics) it
> would make sense (to me) to have reads and writes at arbitrary offsets
> to have arbitrary semantics as well -- that's, after all, what offset
> (kind of) does on a regular file, too, although in a rather trivial
> way.

> ...but my spider-sense is telling me this would probably be either
> rather pointless, or troublesome, or prohibited. Please set me
> straight.

this is excellent!  The only thing you need now is a new name to mark
the address of the new block, a mechanism to seek to this address,
and an interface to standard I/O such that writes append to the block's
end.

nkl

Re: [9fans] read/write offset hack

[Charles Forsyth]

> ...but my spider-sense is telling me this would probably be either
> rather pointless, or troublesome, or prohibited. Please set me
> straight.

well, of those, it can't be prohibited, because we haven't got a task force to rip
keyboards from careless hands, although given the system's background,
alien abduction is always a possibility.

i suppose i don't see the point, so i'd go with the first one.

[9fans] read/write offset hack

[Nyang A. Phra]

Poking around Plan 9 and 9P, I was wondering whether it would be a
neat hack or some sort of abuse to read and write dynamically served
files at different offsets to get different semantics, instead of
reading and writing different files (ctl, clone, etc.) to do that.

Given that the system encourages to perceive files as having arbitrary
semantics (as opposed to having regular sequential file semantics) it
would make sense (to me) to have reads and writes at arbitrary offsets
to have arbitrary semantics as well -- that's, after all, what offset
(kind of) does on a regular file, too, although in a rather trivial
way.

...but my spider-sense is telling me this would probably be either
rather pointless, or troublesome, or prohibited. Please set me
straight.

Nyang