[9fans] 9P writes for directories

[9fans] 9P writes for directories

[Roman Shaposhnik]

I could swear that I remember the following question being asked during
the times when 9P2000 was being introduced, but I simply can't find
any relevant threads. I do apologize if my google-foo is failing me,
but here it goes: what is the reason for not allowing writes to the
directories (with the append-only semantics and stat structures
payloads, of course)?

Is it simply because since we can't really get rid of create (because
of the transactional nature of create-open) it was deemed to be
unimportant to have directory writing semantics or was there a
subtler reason that I can't fully recall.

Thanks,
Roman.

Re: [9fans] 9P writes for directories

[Roman Shaposhnik]

On Mar 21, 2009, at 12:00 AM, Roman Shaposhnik wrote:
> I could swear that I remember the following question being asked
> during
> the times when 9P2000 was being introduced, but I simply can't find
> any relevant threads. I do apologize if my google-foo is failing me,
> but here it goes: what is the reason for not allowing writes to the
> directories (with the append-only semantics and stat structures
> payloads, of course)?
>
> Is it simply because since we can't really get rid of create (because
> of the transactional nature of create-open) it was deemed to be
> unimportant to have directory writing semantics or was there a
> subtler reason that I can't fully recall.

No takers, huh? Well, may be a bit of background would help so that
the question doesn't sound as a purely navel-gazing exercise.

The story here is that we are building a bunch of RESTful APIs
and my personal preference is to bend HTTP as close to 9P
as I can get (for obvious reasons). Now, the closest match
to "create" would be POST with a metadata payload on a
"subdirectory" URI. But of course, it is not a create at all.
It really is much closer to write on a subdirectory. Hence the
question: is there anything that HTTP makes us lose except
for the transactional nature of create?

Thanks,
Roman.

Re: [9fans] 9P writes for directories

[erik quanstrom]

> question: is there anything that HTTP makes us lose except
> for the transactional nature of create?

sanity?

> to "create" would be POST with a metadata payload on a
> "subdirectory" URI. But of course, it is not a create at all.

i'm not sure what fine hair you're trying to split.  why can't the
post contain the same information a create would?

- erik

Re: [9fans] 9P writes for directories

[Eric Van Hensbergen]

On Thu, Mar 26, 2009 at 2:31 PM, Roman Shaposhnik <rvs@sun.com> wrote:
> On Mar 21, 2009, at 12:00 AM, Roman Shaposhnik wrote:
>
> The story here is that we are building a bunch of RESTful APIs
> and my personal preference is to bend HTTP as close to 9P
> as I can get (for obvious reasons). Now, the closest match
> to "create" would be POST with a metadata payload on a
> "subdirectory" URI. But of course, it is not a create at all.
> It really is much closer to write on a subdirectory. Hence the
> question: is there anything that HTTP makes us lose except
> for the transactional nature of create?
>

So, my understanding is the function of POST to a collection/directory
was to create a new entry in the collection where the ID is assigned
automatically by the collection. The ID created is typically returned
by this operation.

This actually sounds more like a (devip style) clone operation to me.
Outside of the ID bit, why wouldn't create suffice?

         -eric

Re: [9fans] 9P writes for directories

[Roman Shaposhnik]

On Mar 26, 2009, at 12:35 PM, erik quanstrom wrote:
>> question: is there anything that HTTP makes us lose except
>> for the transactional nature of create?
>
> sanity?

That's dead and buried already :-( But I've got be honest
with you -- REST is actually the closes thing to 9P
that has a potential to excite PHBs and thus get approved.
HTTP is stateless, yes, but the biggest PITA that brought us
so far would be authentication, not the core functionality.

>> to "create" would be POST with a metadata payload on a
>> "subdirectory" URI. But of course, it is not a create at all.
>
> i'm not sure what fine hair you're trying to split.  why can't the
> post contain the same information a create would?


If "hair splitting" is the answer to the original question -- that's
completely fine. In fact, I was looking for that to be an answer.
It's just that everytime I see something being explicitly forbidden
in 9P spec there's usually a pretty good reason for that.

Thanks,
Roman.

Re: [9fans] 9P writes for directories

[erik quanstrom]

> >> to "create" would be POST with a metadata payload on a
> >> "subdirectory" URI. But of course, it is not a create at all.
> >
> > i'm not sure what fine hair you're trying to split.  why can't the
> > post contain the same information a create would?
>
>
> If "hair splitting" is the answer to the original question -- that's
> completely fine. In fact, I was looking for that to be an answer.
> It's just that everytime I see something being explicitly forbidden
> in 9P spec there's usually a pretty good reason for that.

i wasn't clear.  why can't you gather up the data from
the post and do a create rather than a write to the directory?

- erik

Re: [9fans] 9P writes for directories

[Roman Shaposhnik]

On Mar 26, 2009, at 12:44 PM, Eric Van Hensbergen wrote:
> On Thu, Mar 26, 2009 at 2:31 PM, Roman Shaposhnik <rvs@sun.com> wrote:
>> On Mar 21, 2009, at 12:00 AM, Roman Shaposhnik wrote:
>>
>> The story here is that we are building a bunch of RESTful APIs
>> and my personal preference is to bend HTTP as close to 9P
>> as I can get (for obvious reasons). Now, the closest match
>> to "create" would be POST with a metadata payload on a
>> "subdirectory" URI. But of course, it is not a create at all.
>> It really is much closer to write on a subdirectory. Hence the
>> question: is there anything that HTTP makes us lose except
>> for the transactional nature of create?
>>
>
> So, my understanding is the function of POST to a collection/directory
> was to create a new entry in the collection where the ID is assigned
> automatically by the collection. The ID created is typically returned
> by this operation.

If by ID you mean URI, that would be my current understanding as well.
Unlike create though, that is guaranteed to create an object in the
subdirectory
identified by an ID (fid), the POST can return any URI.

> This actually sounds more like a (devip style) clone operation to me.

I have thought about that too, but became convinced that POST is more
like create (or more like write on a subdirectory -- hence the original
question). With the clone operation it is the *opening* of the clone
device that provides you with a new fid. In HTTP that would be like
getting
a redirection on GET. Don't you think?

> Outside of the ID bit, why wouldn't create suffice?


It would (just as Erik pointed out). I guess I was just looking for
symmetry (if POST is really a write(*), it should translate into write
independent of whether the URI corresponds to a subdirectory or
not) and potential pitfalls that made 9P spec disallow writes on
subdirectories (and since nobody can identify any of those -- I'll
rest my case and proceed with translating POST into different
9P messages depending on the type of the URI).

Thanks,
Roman.

(*) Now, POST really is *not* a write. PUT is. But current generation
of browsers can only "write" local files to remote destinations
with POST, not PUT. And even with JavaScript that restriction is too
painful
to overcome :-(

Re: [9fans] 9P writes for directories

[erik quanstrom]

> It would (just as Erik pointed out). I guess I was just looking for
> symmetry (if POST is really a write(*), it should translate into write
> independent of whether the URI corresponds to a subdirectory or
> not) and potential pitfalls that made 9P spec disallow writes on
> subdirectories (and since nobody can identify any of those -- I'll
> rest my case and proceed with translating POST into different
> 9P messages depending on the type of the URI).

the only thing that write and create have in common
is the fact that they both translate to 9p messages.

the beauty of create is that it is atomic.  if you don't
care about atomicity, you can just open if create fails.
using write, the qid could not be returned and there
would always be a race.  since one would not have the
qid, it would be impossible to tell if the race had been
won.

- erik

Re: [9fans] 9P writes for directories

[Eric Van Hensbergen]

On Thu, Mar 26, 2009 at 3:23 PM, Roman Shaposhnik <rvs@sun.com> wrote:
> On Mar 26, 2009, at 12:44 PM, Eric Van Hensbergen wrote:
>>
>> On Thu, Mar 26, 2009 at 2:31 PM, Roman Shaposhnik <rvs@sun.com> wrote:
>>>
>>> On Mar 21, 2009, at 12:00 AM, Roman Shaposhnik wrote:
>>>
>>> The story here is that we are building a bunch of RESTful APIs
>>> and my personal preference is to bend HTTP as close to 9P
>>> as I can get (for obvious reasons). Now, the closest match
>>> to "create" would be POST with a metadata payload on a
>>> "subdirectory" URI. But of course, it is not a create at all.
>>> It really is much closer to write on a subdirectory. Hence the
>>> question: is there anything that HTTP makes us lose except
>>> for the transactional nature of create?
>>>
>>
>> So, my understanding is the function of POST to a collection/directory
>> was to create a new entry in the collection where the ID is assigned
>> automatically by the collection. The ID created is typically returned
>> by this operation.
>
> If by ID you mean URI, that would be my current understanding as well.
> Unlike create though, that is guaranteed to create an object in the
> subdirectory
> identified by an ID (fid), the POST can return any URI.

I think we have a semantic mismatch, if ID==URI then ID != FID.  I
don't see where FID's fit in to the REST model (making REST closer to
Op, then 9p)

>
>> This actually sounds more like a (devip style) clone operation to me.
>
> I have thought about that too, but became convinced that POST is more
> like create (or more like write on a subdirectory -- hence the original
> question). With the clone operation it is the *opening* of the clone
> device that provides you with a new fid. In HTTP that would be like getting
> a redirection on GET. Don't you think?
>

Except that Creates give an ID/path for creation instead of receiving
one -- that's the key thing that makes it like clone, the most
important bit being that this sort of mechanism avoids collision.
Whether or not that is critical depends on how you write your app.  I
think the main difference here is you are trying to map HTTP syntax to
9P syntax, and I've been thinking of semantics -- an HTTP POST to a
subdirectory would equal the opening of a clone file (within that
subdirectory), and writing the metadata, a read on that file would
return the ID -- this would be done atomically by the HTTP server to
service the POST not as a set of HTTP routines.  Again - this all
depends on what you are actually trying to do with your app as to
whether or not clone semantics are necessary.  I think the critical
aspect from the REST/CRUD perspective is that the POST has to be
idempotent which I don't think you can guarantee with just create
semantics due to the possibility of collision (but perhaps creates
with collision just fail), not sure if that is "expected" from a REST
perspective, but it is probably legal.

>> Outside of the ID bit, why wouldn't create suffice?
>
> It would (just as Erik pointed out). I guess I was just looking for
> symmetry (if POST is really a write(*), it should translate into write
> independent of whether the URI corresponds to a subdirectory or
> not) and potential pitfalls that made 9P spec disallow writes on
> subdirectories (and since nobody can identify any of those -- I'll
> rest my case and proceed with translating POST into different
> 9P messages depending on the type of the URI).
>

I don't think the symmetry is worth altering the semantics of the
protocol -- its likely more trouble than its worth in the long run.

        -eric

Re: [9fans] 9P writes for directories

[Roman Shaposhnik]

On Mar 26, 2009, at 1:54 PM, Eric Van Hensbergen wrote:
>> I have thought about that too, but became convinced that POST is more
>> like create (or more like write on a subdirectory -- hence the
>> original
>> question). With the clone operation it is the *opening* of the clone
>> device that provides you with a new fid. In HTTP that would be like
>> getting
>> a redirection on GET. Don't you think?
>>
>
> Except that Creates give an ID/path for creation instead of receiving
> one -- that's the key thing that makes it like clone, the most
> important bit being that this sort of mechanism avoids collision.

I believe we're on the same page, but the wording is  not always
accurate.

> Whether or not that is critical depends on how you write your app.  I
> think the main difference here is you are trying to map HTTP syntax to
> 9P syntax, and I've been thinking of semantics -- an HTTP POST to a
> subdirectory would equal the opening of a clone file (within that
> subdirectory), and writing the metadata,

Interesting point. I guess it all depends on what is the model for
new nodes to be created in the URI tree. In fact, if I were to complete
the analogy, then POST to an existing URI (although, nothing really
exists
in the world of URIs) corresponding to a "subdirectory" would be, in
fact,
what you say. A POST/PUT to a non-existing URI could be considered
a creation of the named resource in its parent. Although I'm not sure
that
such a thing should be always allowed.

> a read on that file would
> return the ID -- this would be done atomically by the HTTP server to
> service the POST not as a set of HTTP routines.

If you mean PRG (http://en.wikipedia.org/wiki/Post/Redirect/Get) then
yes.

> I think the critical aspect from the REST/CRUD perspective is that
> the POST
> has to be idempotent

Not as per HTTP RFC:
     http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.5

>>> Outside of the ID bit, why wouldn't create suffice?
>>
>> It would (just as Erik pointed out). I guess I was just looking for
>> symmetry (if POST is really a write(*), it should translate into
>> write
>> independent of whether the URI corresponds to a subdirectory or
>> not) and potential pitfalls that made 9P spec disallow writes on
>> subdirectories (and since nobody can identify any of those -- I'll
>> rest my case and proceed with translating POST into different
>> 9P messages depending on the type of the URI).
>>
>
> I don't think the symmetry is worth altering the semantics of the
> protocol -- its likely more trouble than its worth in the long run.

Agreed.

Thanks,
Roman.

P.S. I wonder if there's a general interest in REST from the Plan9
folks. I've seen your blog post on the subject, so there's that. It is
actually quite fun to see how things like Google App Engine and
http://konstrukt.dk/, etc can reap the same benefits from the elegance
of FS-aware design. Could it be that with the right approach
(lib9p/libixp plugin for apache?) writing web services could be
as much fun as writing filesystems for Plan9?

Re: [9fans] 9P writes for directories

[Uriel]

Some of us have been thinking about a 'sane' subset of HTTP plus some
conventions, that could reasonably map to 9p.

The main issue is the huge amounts of crud in the HTTP spec and how to
pick the sensible bits and discard the rest while remaining compatible
with existing implementations; the main convention that needs to be
added is a way to list directory contents, likely using something not
too insane, like JSON.

With some thought and care one could come up with a very simple,
RESTful, and almost 9p-mappable replacement for the stinking bloated
WebDAV and SOAP/XML-RPC abominations.

uriel

On Thu, Mar 26, 2009 at 11:05 PM, Roman Shaposhnik <rvs@sun.com> wrote:
> On Mar 26, 2009, at 1:54 PM, Eric Van Hensbergen wrote:
>>>
>>> I have thought about that too, but became convinced that POST is more
>>> like create (or more like write on a subdirectory -- hence the original
>>> question). With the clone operation it is the *opening* of the clone
>>> device that provides you with a new fid. In HTTP that would be like
>>> getting
>>> a redirection on GET. Don't you think?
>>>
>>
>> Except that Creates give an ID/path for creation instead of receiving
>> one -- that's the key thing that makes it like clone, the most
>> important bit being that this sort of mechanism avoids collision.
>
> I believe we're on the same page, but the wording is  not always
> accurate.
>
>> Whether or not that is critical depends on how you write your app.  I
>> think the main difference here is you are trying to map HTTP syntax to
>> 9P syntax, and I've been thinking of semantics -- an HTTP POST to a
>> subdirectory would equal the opening of a clone file (within that
>> subdirectory), and writing the metadata,
>
> Interesting point. I guess it all depends on what is the model for
> new nodes to be created in the URI tree. In fact, if I were to complete
> the analogy, then POST to an existing URI (although, nothing really exists
> in the world of URIs) corresponding to a "subdirectory" would be, in fact,
> what you say. A POST/PUT to a non-existing URI could be considered
> a creation of the named resource in its parent. Although I'm not sure that
> such a thing should be always allowed.
>
>> a read on that file would
>> return the ID -- this would be done atomically by the HTTP server to
>> service the POST not as a set of HTTP routines.
>
> If you mean PRG (http://en.wikipedia.org/wiki/Post/Redirect/Get) then
> yes.
>
>> I think the critical aspect from the REST/CRUD perspective is that the
>> POST
>> has to be idempotent
>
> Not as per HTTP RFC:
>    http://www.w3.org/Protocols/rfc2616/rfc2616-sec9.html#sec9.5
>
>>>> Outside of the ID bit, why wouldn't create suffice?
>>>
>>> It would (just as Erik pointed out). I guess I was just looking for
>>> symmetry (if POST is really a write(*), it should translate into write
>>> independent of whether the URI corresponds to a subdirectory or
>>> not) and potential pitfalls that made 9P spec disallow writes on
>>> subdirectories (and since nobody can identify any of those -- I'll
>>> rest my case and proceed with translating POST into different
>>> 9P messages depending on the type of the URI).
>>>
>>
>> I don't think the symmetry is worth altering the semantics of the
>> protocol -- its likely more trouble than its worth in the long run.
>
> Agreed.
>
> Thanks,
> Roman.
>
> P.S. I wonder if there's a general interest in REST from the Plan9
> folks. I've seen your blog post on the subject, so there's that. It is
> actually quite fun to see how things like Google App Engine and
> http://konstrukt.dk/, etc can reap the same benefits from the elegance
> of FS-aware design. Could it be that with the right approach
> (lib9p/libixp plugin for apache?) writing web services could be
> as much fun as writing filesystems for Plan9?
>

Re: [9fans] 9P writes for directories

[Roman Shaposhnik]

On Mar 27, 2009, at 6:36 PM, Uriel wrote:
> Some of us have been thinking about a 'sane' subset of HTTP plus some
> conventions, that could reasonably map to 9p.

Interestingly enough, that's exactly the quest I'm on. I'd appreciate
a chance
of talking to likeminded folks.

> The main issue is the huge amounts of crud in the HTTP spec and how to
> pick the sensible bits

Ever since I've read "JavaSript: the good parts" (and realized along the
way that the good parts were really quire close to Scheme) I tend
to thing that this approach of trying to identify "the good parts" is
the
only thing that can let me keep sanity in the modern world of
rampant web services. The good news (at least so far) seem to
be that there *are* good parts in HTTP.

> and discard the rest while remaining compatible
> with existing implementations; the main convention that needs to be
> added is a way to list directory contents, likely using something not
> too insane, like JSON.

Yeap.

> With some thought and care one could come up with a very simple,
> RESTful, and almost 9p-mappable replacement for the stinking bloated
> WebDAV and SOAP/XML-RPC abominations.


Can't agree more. Another reason to clean up this mess is a potential
to significantly simplify the way a typical web service is written. With
luck, there could be something as simple as lib9p/libixp (or even
execnet) to make writing webserives simple and fun.

  I'd love to exchange ideas on a more appropriate forum (if there's
any).

Thanks,
Roman.

Re: [9fans] 9P writes for directories

[Enrico Weigelt]

* Uriel <uriel99@gmail.com> wrote:

Hi folks,

<snip>

I've completely followed the whole thread, but just a few ideas
how it could look like:

Essentially we need the operations:

    #1: read from some file, ev. w/ some offset and a limit
    #2: write to some file, ev. w/ some offset
    #3: create a new file
    #4: remove some file
    #5: list directories

Okay, lets make a start:

#1: reading

>> GET /some/file HTTP/1.1
>> Offset: 512
>> Limit: 4096
>>
--
<< 300 OK
<< Content-Length: 433
<<
<< dajlfajelfhaewfhaewfhlfhawlfhalefhailheflaefhalehfalefhalfhawlefhaweflh
...

For synthetic files which dont return data immediately, we could add some
additional Timeout: header. If the timeout is over, the server replies
with some appropriate code, maybe 204 or 206.

#2: writing

>> PUT /some/file HTTP/1.1
>> Offet: 512
>> Content-Size: 1024
>>
>> dfljfldkjldfjadlfjaljfldjfaldjfldjfldjf
...

<< 300 OK
<<

#3: create new file

simple write to an non-existing file

#4: remove some file:

>> REMOVE /some/file HTTP/1.1
>>
...
<< 300 OK
<<

#5: list a directory

just read the resource, the listing will be replied in ls -l format.


Okay, this approach has the drawback, that we don't have the concept
of file descriptors, so working with synthethic filesystems still might
be ugly. To get out of this, we could put the actual files and fd's
in their own namespaces, eg:

/root/ -> contains the actual files
/fd/ -> contains the fd's (counted up)

As we now need some client/connection identification, we could use a
cookie for that. Simply GET / without a cookie, and get back an reply
with a new cookie. Maybe there could even be some special file/subdir
for things like authentication (if its not already done through SSL
or HTTP authentication headers).


cu
--
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------