Re: [9fans] read/write offset hack

Re: [9fans] read/write offset hack

[erik quanstrom]

> P. P. S.  The usb audio use of offsets is not as bad as it first sounds.
> The device consumes written data at a constant rate (say, 176,400
> bytes per second for CD audio).  You can make a noise ten seconds
> from now by writing 1.7MB bytes of zeros (silence) followed by your
> sound data.  Being able to seek ahead 1.7MB instead just avoids the
> need to write the zeros.  The file offset is still bytes, not nanoseconds.
>

why not put the timing information inband?  this would allow the timed
sound to be saved to a file also without 1.7mb of zeros.

- erik

Re: [9fans] read/write offset hack

[roger peppe]

On Fri, May 30, 2008 at 4:15 PM, erik quanstrom <quanstro@coraid.com> wrote:
> why not put the timing information inband?  this would allow the timed
> sound to be saved to a file also without 1.7mb of zeros.

because then you'd need to quote the audio data in case it
contained something that looked like a timing command?

Re: [9fans] read/write offset hack

[Russ Cox]

> why not put the timing information inband?  this would allow the timed
> sound to be saved to a file also without 1.7mb of zeros.

The Plan 9 audio device data format is stereo, 16-bit
little-endian PCM samples.  Given that convention, my
point was only that seeking in that kind of data stream
does make a little bit of sense.  The whole mail was
about the power of working within and respecting
existing conventions, not about redesigning the audio device.

Others might choose different conventions in different situations.
For example, I believe Nemo's Plan B audio device accepts
MP3 format audio to reduce bandwidth requirements for
sending audio to devices elsewhere on the network.
The Plan 9 audio format has the benefit of being close to
the native format for most hardware.

As always, your own context dictates whether or not
breaking established conventions is a good idea.  Of course,
the Sparc /dev/audio is not just a good idea: it's μ-law.

Russ

[9fans] /dev/audio format (was: read/write offset hack)

[a]

// The Plan 9 audio device data format is stereo, 16-bit
// little-endian PCM samples.

While I was at the labs, I'm pretty sure we had a /dev/audio that could
take different audio formats (told the format via /dev/audioctl). I can't
remember where it came from, though. Was something like this ever in
Brazil, or was this specific to our group (Mesa; Geoff?)? I believe it was
limited to things like PCM, μ-law, and friends, at different combinations
of stereo, mono, and bitrates, not anthing as complex as mp3.

Mostly just trying to fill in a memory gap.
Anthony

Re: [9fans] /dev/audio format (was: read/write offset hack)

[geoff]

[-- Attachment #1: Type: text/plain, Size: 272 bytes --]

The devaudio Anthony remembers was in the Brazil kernel maintained by
the Mesa group and appears to have come from some Inferno BU kernel.
It supported exactly PCM, μ-law and a-law.

The current Inferno audio(3) claims to support "ulaw, alaw, pcm, pcm16
or adpcm".

[-- Attachment #2: Type: message/rfc822, Size: 2334 bytes --]

From: a@9srv.net
To: 9fans@9fans.net
Subject: [9fans] /dev/audio format (was: read/write offset hack)
Date: Fri, 30 May 2008 15:11:35 -0400
Message-ID: <3131a8932bcc91ca2802ea9d02a40c75@9srv.net>

// The Plan 9 audio device data format is stereo, 16-bit
// little-endian PCM samples.

While I was at the labs, I'm pretty sure we had a /dev/audio that could
take different audio formats (told the format via /dev/audioctl). I can't
remember where it came from, though. Was something like this ever in
Brazil, or was this specific to our group (Mesa; Geoff?)? I believe it was
limited to things like PCM, μ-law, and friends, at different combinations
of stereo, mono, and bitrates, not anthing as complex as mp3.

Mostly just trying to fill in a memory gap.
Anthony

Re: [9fans] /dev/audio format (was: read/write offset hack)

[Charles Forsyth]

> The current Inferno audio(3) claims to support "ulaw, alaw, pcm, pcm16
> or adpcm".

if the device supports it.  reading the ctl file reveals available formats.

Re: [9fans] /dev/audio format (was: read/write offset hack)

[erik quanstrom]

> The devaudio Anthony remembers was in the Brazil kernel maintained by
> the Mesa group and appears to have come from some Inferno BU kernel.
> It supported exactly PCM, μ-law and a-law.
>
> The current Inferno audio(3) claims to support "ulaw, alaw, pcm, pcm16
> or adpcm".

a search on "inferno bu" turns up this

	http://cgi.ebay.com/3Racing-Rear-Anti-Roll-Bar-for-Kyosho-Mini-Inferno-BU_W0QQitemZ160244218723QQcmdZViewItem?IMSfp=TL0805280137r6123

does anyone have any ptrs or information about either of the aforementioned?

- erik

Re: [9fans] /dev/audio format (was: read/write offset hack)

[andrey mirtchovski]

BU is the "Business Unit". the reason we're not using Plan 9 and
Inferno instead of Linux and Java ;)

Re: [9fans] /dev/audio format (was: read/write offset hack)

[Skip Tavakkolian]

> BU is the "Business Unit". the reason we're not using Plan 9 and
> Inferno instead of Linux and Java ;)

i don't think their marketing really understood what they had.

on the plus side, M$ is embracing and extending the ideas with their
singularity "innovation".  also, windows7 is coming out soon, and
maybe we'll get to windows9 in 4-6 years; one can hope windows9 would
be like plan9 only gui-er.

Re: [9fans] /dev/audio format (was: read/write offset hack)

[Bruce Ellis]

Correct. There were several generations of audio (and later
multimedia) drivers in both BU Inferno and Research Inferno. e.g. the
Windows driver supports any installed codec.

There was also the Inferno Audio File format which was rather fb like.
The file began with text lines describing the stuff and then after a
\n\n the rest was data. (The drivers didn't know the format, the
player and other tools did.)

An unofficial history of Inferno would be amusing. Maybe it'll be
penned up in a bar at a future IWP9. ericvh was lucky enough to have
been in the /dev/caveman meeting. The "third party fools" era was
entertaining too, though very sad.

brucee

On 5/31/08, geoff@plan9.bell-labs.com <geoff@plan9.bell-labs.com> wrote:
> The devaudio Anthony remembers was in the Brazil kernel maintained by
> the Mesa group and appears to have come from some Inferno BU kernel.
> It supported exactly PCM, μ-law and a-law.

Re: [9fans] /dev/audio format (was: read/write offset hack)

[Bruce Ellis]

Sorry if you were eating dinner.

On 5/31/08, Charles Forsyth <forsyth@terzarima.net> wrote:
> > The "third party fools" era was
> > entertaining too, though very sad.
>
> don't remind me.

Re: [9fans] /dev/audio format (was: read/write offset hack)

[Charles Forsyth]

> The "third party fools" era was
> entertaining too, though very sad.

don't remind me.

Re: [9fans] /dev/audio format (was: read/write offset hack)

[Enrico Weigelt]

* Bruce Ellis <bruce.ellis@gmail.com> wrote:

Hi,

> There was also the Inferno Audio File format which was rather fb
> like.The file began with text lines describing the stuff and then
> after a\n\n the rest was data.

Interesting idea.
We could also use something like HTTP or mail headers - easy to
understand for many people :)

Maybe an advanced audio server could even directly understand
mimetypes ? ,-O

BTW: some weeks ago, I started hacking on an 9P'ed pulseaudio.
It will be configurable completely via 9P and also support
audio I/O via 9P. But I'm still looking for a suitable interface.

The audio interface should be capable of completely replacing
other common interfaces (in GNU world), so eg. understand
all hw-supported formats, realtime-capable, etc.

Once this works, I intend to rewrite certain client libs for
other audio servers (pulse, esd, arts, ...) to go through
that new 9P interface.


Suggestions appreciated :)


cu
--
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [9fans] /dev/audio format (was: read/write offset hack)

[Bruce Ellis]

mime type of x-iaf was used successfully.

On 6/1/08, Enrico Weigelt <weigelt@metux.de> wrote:
> * Bruce Ellis <bruce.ellis@gmail.com> wrote:
>
> Hi,
>
> > There was also the Inferno Audio File format which was rather fb
> > like.The file began with text lines describing the stuff and then
> > after a\n\n the rest was data.
>
> Interesting idea.
> We could also use something like HTTP or mail headers - easy to
> understand for many people :)
>
> Maybe an advanced audio server could even directly understand
> mimetypes ? ,-O
>
> BTW: some weeks ago, I started hacking on an 9P'ed pulseaudio.
> It will be configurable completely via 9P and also support
> audio I/O via 9P. But I'm still looking for a suitable interface.
>
> The audio interface should be capable of completely replacing
> other common interfaces (in GNU world), so eg. understand
> all hw-supported formats, realtime-capable, etc.
>
> Once this works, I intend to rewrite certain client libs for
> other audio servers (pulse, esd, arts, ...) to go through
> that new 9P interface.
>
>
> Suggestions appreciated :)
>
>
> cu
> --
> ----------------------------------------------------------------------
>  Enrico Weigelt, metux IT service -- http://www.metux.de/
>
>  cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
> ----------------------------------------------------------------------
>  Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
> ----------------------------------------------------------------------
>
>

Re: [9fans] read/write offset hack

[Nyang A. Phra]

On May 30, 6:06 pm, r...@swtch.com (Russ Cox) wrote:
> If you want to violate a convention, Plan 9 won't stop you,
> but in doing so you give up compatibility with programs that
> depend on that convention (bind /net/tcp /proc; ps).
> Sure, you could replace ctl and clone and other special files with reads
> and writes at magic offsets, but in doing so you give up accessing
> those files with standard programs like echo and cat.  For me,
> the main benefit of user-level file servers is exactly that I can
> interact with them from a wide variety of programs, including
> scripts and interactive shell sessions.  I'd need a pretty compelling
> reason before I gave that up.

Thanks Russ for such a thorough and pragmatic response, I'm happy to
have learned something :)

Nyang

Re: [9fans] read/write offset hack

[Roman Shaposhnik]

On Fri, 2008-05-30 at 15:04 -0400, a@9srv.net wrote:
> Very well explained. I've seen this question come up lots of times
> when introducing fs-based interfaces to people. When we had some
> off-shore devs to bring up to speed, they kept coming back to it: "so
> can we just define a protocol to put all these streams in one file?". I
> tried to explaing that having them as distinct files *was* that
> protocol, but it never stuck. I wish I'd had this explanation then.

I believe that part of the difficulty might come from the fact that
Plan9 is exceptional in a sense that it offers an extremely effective
environment in a basic package. You don't have to devise complicated
GUI "shells" to be productive. Acme is as fancy as it gets.
Unfortunately not everybody is lucky enough to be able to run Plan9
environment. People come from backgrounds of different level of
suckiness. The environments where GUI "shells" are a necessity
(did you know both KDE and GNOME implement incompatible filesystems
at the level of their Explorers?). That's why these folks never trust
their basic environment. They never expect the tools that come with
it to be of any significant use. Their default reasoning is always
to add more "protocols" to the bloated monsters they call "GUI shells".
They just don't know any better.

Thanks,
Roman.

Re: [9fans] read/write offset hack

[a]

Very well explained. I've seen this question come up lots of times
when introducing fs-based interfaces to people. When we had some
off-shore devs to bring up to speed, they kept coming back to it: "so
can we just define a protocol to put all these streams in one file?". I
tried to explaing that having them as distinct files *was* that
protocol, but it never stuck. I wish I'd had this explanation then.
Anthony

Re: [9fans] read/write offset hack

[Robert William Fuller]

Russ Cox wrote:

<snip>

> If you want to violate a convention, Plan 9 won't stop you,
> but in doing so you give up compatibility with programs that
> depend on that convention (bind /net/tcp /proc; ps).
> Sure, you could replace ctl and clone and other special files with reads
> and writes at magic offsets, but in doing so you give up accessing
> those files with standard programs like echo and cat.  For me,
> the main benefit of user-level file servers is exactly that I can
> interact with them from a wide variety of programs, including
> scripts and interactive shell sessions.  I'd need a pretty compelling
> reason before I gave that up.

Thank you.  Posts like these are the reason I follow this list, even
though I do not have Plan9 installed!  (I have an interest in operating
system design and a background in filesystems.)

Re: [9fans] read/write offset hack

[ron minnich]

On Fri, May 30, 2008 at 1:43 AM, Nyang A. Phra <naphra@gmail.com> wrote:
> Poking around Plan 9 and 9P, I was wondering whether it would be a
> neat hack or some sort of abuse to read and write dynamically served
> files at different offsets to get different semantics, instead of
> reading and writing different files (ctl, clone, etc.) to do that.

it's a step backwards in time to very old ideas. And it's a terrible
idea. Heed your spider sense.

Just consider the QID  as your differentiator.

ron

Re: [9fans] read/write offset hack

[Stefan Hajnoczi]

Excellent, excellent response Russ.  If a question like this comes up
again, your post is a great reference.

Stefan

On Fri, May 30, 2008 at 4:35 PM, erik quanstrom <quanstro@coraid.com> wrote:
>> On Fri, May 30, 2008 at 4:15 PM, erik quanstrom <quanstro@coraid.com> wrote:
>> > why not put the timing information inband?  this would allow the timed
>> > sound to be saved to a file also without 1.7mb of zeros.
>>
>> because then you'd need to quote the audio data in case it
>> contained something that looked like a timing command?
>
> if the size of the chunks is not obvious, say due to compression,
> run-length encoding would solve that problem.
>
> - erik
>
>

Re: [9fans] read/write offset hack

[erik quanstrom]

> On Fri, May 30, 2008 at 4:15 PM, erik quanstrom <quanstro@coraid.com> wrote:
> > why not put the timing information inband?  this would allow the timed
> > sound to be saved to a file also without 1.7mb of zeros.
>
> because then you'd need to quote the audio data in case it
> contained something that looked like a timing command?

if the size of the chunks is not obvious, say due to compression,
run-length encoding would solve that problem.

- erik

Re: [9fans] read/write offset hack

[Russ Cox]

> Poking around Plan 9 and 9P, I was wondering whether it would be a
> neat hack or some sort of abuse to read and write dynamically served
> files at different offsets to get different semantics, instead of
> reading and writing different files (ctl, clone, etc.) to do that.
>
> Given that the system encourages to perceive files as having arbitrary
> semantics (as opposed to having regular sequential file semantics) it
> would make sense (to me) to have reads and writes at arbitrary offsets
> to have arbitrary semantics as well -- that's, after all, what offset
> (kind of) does on a regular file, too, although in a rather trivial
> way.

The semantics of files on Plan 9 are not as ad hoc as you suggest.

In Unix there is essentially one file server, and thus essentially
one file interface.  (There are of course little differences;
for example, can you cat a directory?)

In Plan 9 there are many different file servers, but there are
relatively few different file interfaces.  The paucity of distinct
interfaces is essential to being able to reuse common tools
across multiple file systems.  Of course, each file server does
implement some special behavior (that's why it exists) but
the most effective ones implement those behaviors in a way
that existing tools continue to work.

A network directory is a special kind of tree, but the
many file servers that provide networks agree on a single
interface.  Programs that dial(3) work just as well with
the kernel-provided /net/tcp, /net/il, and /net/ether0
as they do with drawterm's networks (bind /mnt/term/net/tcp /net/tcp)
or with programs like sshnet.  If every network server used
a different interface, every program that used the network
would have to be changed when a new kind of network
was introduced.  (In Unix they do, and it's very frustrating!)

File servers are of course free to introduce whole new
intrfaces, but they usually do it in a way that simple
programs can continue to use their files.  For example,
even though some network directory files have special
semantics, the /net/xxx/0/data file behaves no different
from a pipe, so that programs that can talk to a pipe can
talk to the network, assuming some other program sets up
the connection.

As another example, upas/fs presents your mail box as a file tree.
The mail readers--nedmail and Mail--are deeply aware of
the structure of the tree, but mail-ignorant programs can
still access the files.  Upas/fs would be much less useful if
	file /mail/fs/mbox/1/2/3/body.jpg
	tar c /mail/fs/mbox/1/2/3/body.jpg >/usr/rsc/jpg.tar
	lp /mail/fs/mbox/1/2/3/body.jpg
	page /mail/fs/mbox/1/2/3/body.jpg
did not work.  (In p9p on Unix they don't, and it's very frustrating!)

File servers sometimes have good reason to buck convention,
but doing so always comes at a cost.  For example, pipes and
network devices cannot provide random access, so they ignore
the file offset in reads and writes; in doing so they become
incompatible with programs that read the data multiple times
(like page in the above examples).  Of course, pipes and networks
are so common, on both Plan 9 and Unix, that many programs
avoid the assumption of random access when possible.

Another common violation is to return a file size of 0 in stat
even though reading the file would produce bytes.  File servers
do this because it is not always simple to compute the size
of dynamically-generated file contents; in doing so, they become
incompatible with programs that depend on the file size being
accurate.  If upas/fs returned a file size of 0 for body.jpg
(it doesn't), then the tar command above would not work
as well, because it needs to write the file size into the tar file
before it copies in the file contents.  (Why can't it seek back?
Because it avoids random access on its output for compatibility
with pipes and networks.)  Not-actually-zero-length files
are so common that many programs, on Plan 9 at least, avoid
the assumption that stat's length matches the length of read.

If you want to violate a convention, Plan 9 won't stop you,
but in doing so you give up compatibility with programs that
depend on that convention (bind /net/tcp /proc; ps).
Sure, you could replace ctl and clone and other special files with reads
and writes at magic offsets, but in doing so you give up accessing
those files with standard programs like echo and cat.  For me,
the main benefit of user-level file servers is exactly that I can
interact with them from a wide variety of programs, including
scripts and interactive shell sessions.  I'd need a pretty compelling
reason before I gave that up.

Russ


P. S.  I'll ignore the fact that you've also replaced the human-friendly
names with human-unfriendly magic numbers.)

P. P. S.  The usb audio use of offsets is not as bad as it first sounds.
The device consumes written data at a constant rate (say, 176,400
bytes per second for CD audio).  You can make a noise ten seconds
from now by writing 1.7MB bytes of zeros (silence) followed by your
sound data.  Being able to seek ahead 1.7MB instead just avoids the
need to write the zeros.  The file offset is still bytes, not nanoseconds.

Re: [9fans] read/write offset hack

[Nick LaForge]

>> Nyang: I must say one thing: you are simply going to LOVE an
>> abomination of an acme feature i am working on!
>
> Do let us in on the secret (if you wish) :)
>
>

lets discuss it as soon as i get the code out... watch this space ;)

nkl

Re: [9fans] read/write offset hack

[Nyang A. Phra]

> Nyang: I must say one thing: you are simply going to LOVE an
> abomination of an acme feature i am working on!

Do let us in on the secret (if you wish) :)

Re: [9fans] read/write offset hack

[roger peppe]

> Given that the system encourages to perceive files as having arbitrary
> semantics (as opposed to having regular sequential file semantics) it
> would make sense (to me) to have reads and writes at arbitrary offsets
> to have arbitrary semantics as well -- that's, after all, what offset
> (kind of) does on a regular file, too, although in a rather trivial
> way.

the nearest i've seen to this (abusing offset semantics)
in the system i've found in the usb device. from usb(3):

: S is the size of the DMA operations on the device (i.e., the
: minimum useful size for reads and writes), b is the number
: of bytes currently buffered for input or output, and o and t
: should be interpreted to mean that byte offset o was/will be
: reached at time t (nanoseconds since the epoch).  The data
: can be used to map file offsets to points in time.  To play
: or record samples exactly at some predetermined time, use o
: and t with the sampling rate to calculate the offset to seek
: to.

this works ok if you're always getting exactly the same number
of bytes per second, but wouldn't if you were streaming variable bit-rate
data.

the advantage of this over using a control file is that there's no
additional latency
or syscall overhead to exactly specifiying a sample time, which could make
a significant difference, i guess, particularly if you were using the device
over a network.

is this an abuse? i'm not sure, though rob pike definitely seemed to think
it was when i mentioned this to him at usenix one time.

Re: [9fans] read/write offset hack

[Nick LaForge]

>>> we haven't got a task force to rip
>>> keyboards from careless hands

>>this is excellent!

> but Nick LaForge reminds us we can use more effective methods

>       Vercotti: Doug (takes a drink) Well, I was terrified.
>       Everyone was terrified of Doug.  I've seen grown men pull
>       their own heads off rather than see Doug.  Even Dinsdale was
>       frightened of Doug.

>       2nd Interviewer: What did he do?

>       Vercotti: He used...  sarcasm.  He knew all the tricks,
>       dramatic irony, metaphor, bathos, puns, parody, litotes and...
>       satire.  He was vicious.

my apologies. :)

Nyang: I must say one thing: you are simply going to LOVE an
abomination of an acme feature i am working on!

Re: [9fans] read/write offset hack

[Charles Forsyth]

>> we haven't got a task force to rip
>> keyboards from careless hands

>this is excellent!

but Nick LaForge reminds us we can use more effective methods

	Vercotti: Doug (takes a drink) Well, I was terrified.
	Everyone was terrified of Doug.  I've seen grown men pull
	their own heads off rather than see Doug.  Even Dinsdale was
	frightened of Doug.

	2nd Interviewer: What did he do?

	Vercotti: He used...  sarcasm.  He knew all the tricks,
	dramatic irony, metaphor, bathos, puns, parody, litotes and...
	satire.  He was vicious.

Re: [9fans] read/write offset hack

[Nick LaForge]

> Poking around Plan 9 and 9P, I was wondering whether it would be a
> neat hack or some sort of abuse to read and write dynamically served
> files at different offsets to get different semantics, instead of
> reading and writing different files (ctl, clone, etc.) to do that.

> Given that the system encourages to perceive files as having arbitrary
> semantics (as opposed to having regular sequential file semantics) it
> would make sense (to me) to have reads and writes at arbitrary offsets
> to have arbitrary semantics as well -- that's, after all, what offset
> (kind of) does on a regular file, too, although in a rather trivial
> way.

> ...but my spider-sense is telling me this would probably be either
> rather pointless, or troublesome, or prohibited. Please set me
> straight.

this is excellent!  The only thing you need now is a new name to mark
the address of the new block, a mechanism to seek to this address,
and an interface to standard I/O such that writes append to the block's
end.

nkl

Re: [9fans] read/write offset hack

[Charles Forsyth]

> ...but my spider-sense is telling me this would probably be either
> rather pointless, or troublesome, or prohibited. Please set me
> straight.

well, of those, it can't be prohibited, because we haven't got a task force to rip
keyboards from careless hands, although given the system's background,
alien abduction is always a possibility.

i suppose i don't see the point, so i'd go with the first one.

[9fans] read/write offset hack

[Nyang A. Phra]

Poking around Plan 9 and 9P, I was wondering whether it would be a
neat hack or some sort of abuse to read and write dynamically served
files at different offsets to get different semantics, instead of
reading and writing different files (ctl, clone, etc.) to do that.

Given that the system encourages to perceive files as having arbitrary
semantics (as opposed to having regular sequential file semantics) it
would make sense (to me) to have reads and writes at arbitrary offsets
to have arbitrary semantics as well -- that's, after all, what offset
(kind of) does on a regular file, too, although in a rather trivial
way.

...but my spider-sense is telling me this would probably be either
rather pointless, or troublesome, or prohibited. Please set me
straight.

Nyang