[9fans] dbfs and web framework for plan 9 (was: web-based plan 9?)

[9fans] dbfs and web framework for plan 9 (was: web-based plan 9?)

[Giacomo Tesio]

[-- Attachment #1: Type: text/plain, Size: 6266 bytes --]

Since I know it was discussed in the past (but I could not find the post in
the archives) I hope to be "original" by not reiterate old ideas.

BTW I start from a reply to a previous post to explain what I would need to
use plan 9 for web development.

But first of all, I have to say that I'm not (yet) a Plan 9 user/developer,
just a programmer who studied it and loved its potentialities.


Actually I'm thinking about how to use Plan 9 in web development (real one,
I'm not talking about having a blog or a shinny flash site), and I'm
proposing a structure for data access towards a dbfs interface.

The scope is to be able to query the db towards a filesystem, with no need
to use different libraries for different db backend... each application
would use a filesystem, while each rdbms would have it's own filesystem (all
with the same structure, obviously)


Now the WHY I think such an interface matter:

On Fri, Nov 21, 2008 at 7:12 PM, <a@9srv.net> wrote:

> On db ↔ fs conversions:
> // datas (and their relations) MUST be kept consistents.
>
> there's two things you could mean here: either the
> relations between the data fields/types or the relations
> between their values.



Both are really important.

When a product change its code (aka pk) for business' reasons, I need have
it changed automatically in any place its referred in the database.
Or if a dumb employee try to delete a product category (supposing he's
allowed) I have to be sure that the remaining data don't loose meaning.


SQL databases are interesting because they don't do anything to keep the
> relations
> consistent, at a schema level.


They do, if you instruct them about the relations amont your tables.
(MS Access don't count as a db :-D)



> there are some cases where
> this is valuable, but most of the time it just puts more of
> the burden on the application.


Application should HANDLE data.
But datas are a rapresentation of a reality which have a meaning!
A relational database don't have to know how datas are collected or used.
But it has to keep meaningfull the rappresentation.



> enforcing consistency between values is a whole different
> matter.


Obviously I'll check user provided data types before sending them to the db.
BTW, db also have to, for different reasons.
I've to protect my database from malicious sql injections, it has to protect
datas from inconsistency (say a string in a date field).

and largely needs to be done in the ap anyway (say
> your DB rejects a submission; are you just giong to panic,
> say "try again", or try and figure out what it didn't like?).


Well, accoding to the needs, different strategies are followed.
Sometime the error will go (human translated :-D) to the user, others a
signalation will go to the developers.

Here is a disadvantage: user provided data are double checked (some time
checks are 3, when also a remote client do the check, say via javascript).
I reflected a lot about this, but (still) found no solution.

I just notice that the reasons for the checks are different:
1. DB check to keep data consistents
2. Application check to validate data against malicious ones
3. (eventually) browsers check to help the user and to save a little of band

If you need data consistency (anyone need I think) DB have to check.
If you can't trust the user application have to check.
If you don't have enouth Gbits you could also need to repeat the data check
in the browser.



> // And MUST be updated frequently, in transaction.
>
> just curious: what is it about "transactionality" that you care
> about? atomicity, logging, roll-back (atomic roll-back)?
>

Atomicity and rollback. (logging is useful too...)

I need to be able to atomically save related tuples in different tables,
since (for example) those different tables could be simply modelling ONE
real entity (say an order or a family).

// Application code cannot do that, since it's too subject to
> // modifications...
>
> i'm not convinced. you can simply... not modify it!


You are right... I like my applications... (almost all...)
I really don't want... but they paid me! :-D


> unless you've got a database you're exposing to multiple parties
> and you're worried one is going to mess things up for
> others, but in that case i think exposing the database
> directly is pretty risky regardless.


There why we use REST... but that's another history...


> the application i've worked on with the most complex data
> constraints had an "ontology layer" that everything that
> wanted to do data access went through.


'till now our "ontology layer" was the database itself.

But I would be curious about how this ontology layer was implemented.
Really!


> // Moreover such a dbfs should allow rapid development of
> // views over data (with different permissions) by simply
> // write the query in a file and then open the file for reading
> // the XML.
>
> somewhere around here i've got a /chan/pq for inferno that
> makes connections to a pq database. each open is a new
> logical connection; you write the query and read the result.
> results were not in XML, and pq has no updates (at current).


where? I could start from there... (even I'm sure that it has to be coded in
C)


> i think you've got some very interesting ideas on sessionfs
> in particular. i have more thinking to do there.


You're welcome!
I realized that the sessionfs has to cooperate with the rewrite engine...


> oh, probably an aside: from an application design point of
> view, DO NOT empty my shopping cart just because my
> session timed out. i should be able to pick up where i left
> off. i'm not sure how hypothetical that example was, but
> that's a common source of irritation for online shopping.


It was hypothetical. But, to keep back your cart you MUST be logged (some
times adding to the shopping cart don't require authentication) and the cart
status have to be saved somewhere (the database?).


Note that such a strategy could lead to problems too: what about if you fill
a shopping cart with discounted products and submit it a month later, when
discounts disappear?
Are you sure you would be check if the prices have changed? I think so...
you use Plan 9...

But customers, often, use Windows... they are not so smart! :-D



Giacomo

[-- Attachment #2: Type: text/html, Size: 8853 bytes --]

Re: [9fans] dbfs and web framework for plan 9 (was: web-based plan 9?)

[Enrico Weigelt]

* Giacomo Tesio <giacomo@tesio.it> wrote:

Hi,

> The scope is to be able to query the db towards a filesystem, with no need
> to use different libraries for different db backend... each application
> would use a filesystem, while each rdbms would have it's own filesystem (all
> with the same structure, obviously)

That's a really good idea. But there's still one problem to solve:
filesystem hierachies and relational algebra are fundamentally
different concepts. So, if you don't want to just the fs as some
universal sql-transport, data has to be modeled differently.

Actually, in many webapps, the real underlying models are neither
relational nor hierachic, but most times just happen to be
modelled relatiolally, because rdmbs'es are the tool of choice
for most people.

So what we have to do here is to get back to the original problems
to solve and think about how they could fit into an hierachy
instead of relation.

Simple example: a web-forum.

Just look at how things were done in "good old" usenet times:
boards/newsgroups are directories, postings are files. Maybe split
off a single posting into several files, eg. to access single header
lines or body parts (attachments, etc) separately. Now we're almost
at mailfs ;)

Okay, we've got some more data, eg. profile information, maybe all
the other "web-2.0" stuff (user galleries, friend relations, etc),
but IMHO this all easily fits in hierachies if we start thinking
afresh with a blank paper.

With hierachies we can do a lot of other fine things here, eg.
server side threading (= postings ordered in a mail thread) or
various kind of filtering/ordering - the client doesn't have to
this all on its own anymore. Using symlinks (w/ 9P2k) might make
it even more efficient. If we know certain objects/files won't
be changed, caching can be come almost trivial, and the average
traffic of some "web-2.0" application drops down to a very small
percentage ...

That's all possible, if we just can imagine it :)

> Both are really important.
>
> When a product change its code (aka pk) for business' reasons, I need have
> it changed automatically in any place its referred in the database.
> Or if a dumb employee try to delete a product category (supposing he's
> allowed) I have to be sure that the remaining data don't loose meaning.

That's exactly what constraints, rules in SQL etc are for.
Maybe some similar ruling system for filesystems would be fine :)
(any suggestions ?)

> Application should HANDLE data.
> But datas are a rapresentation of a reality which have a meaning!
> A relational database don't have to know how datas are collected or used.
> But it has to keep meaningfull the rappresentation.

Right. But the database has to know rules which data has to comply,
so a wide range of errors becomes impossible, mathematically provable.

Imagine a forum where each posting belongs into a board and is assiocated
with the author. If the constraints are defined properly, the database
won't ever allow postings w/o valid board or author. You could also define
some rule which automatically removes all postings when the board gets
deleted, at the same point of time. (of course, we're talking about real
databases, cardfiles like mysql don't count here)

> I need to be able to atomically save related tuples in different tables,
> since (for example) those different tables could be simply modelling ONE
> real entity (say an order or a family).

CREATE RULE .. AS ON INSERT TO ...


Again, a rule system for filesystems would be really cool :)

cu
--
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [9fans] dbfs and web framework for plan 9 (was: web-based plan 9?)

[Francisco J Ballesteros]

Well, Plan 9 has web forums, see http://lsub.org/magic/group as an example
It uses a directory per web group, with a file per article (and replies to it).
Several skeleton html files are used and a DYNAMIC tag is replaced with
contents using a C program.

We use rm to remove articles, etc. etc.
Also, because we use the real FS to keep the data, we don't need a synthetic
file system program to implement any interface whatsoever (unless you count the
web as such thing).

IIRC, this thing is in sources, in contrib/nemo within the planb distrib.

On Sun, Nov 23, 2008 at 11:06 PM, Enrico Weigelt <weigelt@metux.de> wrote:
> * Giacomo Tesio <giacomo@tesio.it> wrote:
>
> Hi,
>
>> The scope is to be able to query the db towards a filesystem, with no need
>> to use different libraries for different db backend... each application
>> would use a filesystem, while each rdbms would have it's own filesystem (all
>> with the same structure, obviously)
>
> That's a really good idea. But there's still one problem to solve:
> filesystem hierachies and relational algebra are fundamentally
> different concepts. So, if you don't want to just the fs as some
> universal sql-transport, data has to be modeled differently.
>
> Actually, in many webapps, the real underlying models are neither
> relational nor hierachic, but most times just happen to be
> modelled relatiolally, because rdmbs'es are the tool of choice
> for most people.
>
> So what we have to do here is to get back to the original problems
> to solve and think about how they could fit into an hierachy
> instead of relation.
>
> Simple example: a web-forum.
>
> Just look at how things were done in "good old" usenet times:
> boards/newsgroups are directories, postings are files. Maybe split
> off a single posting into several files, eg. to access single header
> lines or body parts (attachments, etc) separately. Now we're almost
> at mailfs ;)
>
> Okay, we've got some more data, eg. profile information, maybe all
> the other "web-2.0" stuff (user galleries, friend relations, etc),
> but IMHO this all easily fits in hierachies if we start thinking
> afresh with a blank paper.
>
> With hierachies we can do a lot of other fine things here, eg.
> server side threading (= postings ordered in a mail thread) or
> various kind of filtering/ordering - the client doesn't have to
> this all on its own anymore. Using symlinks (w/ 9P2k) might make
> it even more efficient. If we know certain objects/files won't
> be changed, caching can be come almost trivial, and the average
> traffic of some "web-2.0" application drops down to a very small
> percentage ...
>
> That's all possible, if we just can imagine it :)
>
>> Both are really important.
>>
>> When a product change its code (aka pk) for business' reasons, I need have
>> it changed automatically in any place its referred in the database.
>> Or if a dumb employee try to delete a product category (supposing he's
>> allowed) I have to be sure that the remaining data don't loose meaning.
>
> That's exactly what constraints, rules in SQL etc are for.
> Maybe some similar ruling system for filesystems would be fine :)
> (any suggestions ?)
>
>> Application should HANDLE data.
>> But datas are a rapresentation of a reality which have a meaning!
>> A relational database don't have to know how datas are collected or used.
>> But it has to keep meaningfull the rappresentation.
>
> Right. But the database has to know rules which data has to comply,
> so a wide range of errors becomes impossible, mathematically provable.
>
> Imagine a forum where each posting belongs into a board and is assiocated
> with the author. If the constraints are defined properly, the database
> won't ever allow postings w/o valid board or author. You could also define
> some rule which automatically removes all postings when the board gets
> deleted, at the same point of time. (of course, we're talking about real
> databases, cardfiles like mysql don't count here)
>
>> I need to be able to atomically save related tuples in different tables,
>> since (for example) those different tables could be simply modelling ONE
>> real entity (say an order or a family).
>
> CREATE RULE .. AS ON INSERT TO ...
>
>
> Again, a rule system for filesystems would be really cool :)
>
> cu
> --
> ----------------------------------------------------------------------
>  Enrico Weigelt, metux IT service -- http://www.metux.de/
>
>  cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
> ----------------------------------------------------------------------
>  Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
> ----------------------------------------------------------------------
>
>

Re: [9fans] dbfs and web framework for plan 9 (was: web-based plan 9?)

[Enrico Weigelt]

* Francisco J Ballesteros <nemo@lsub.org> wrote:

Hi,

> Well, Plan 9 has web forums, see http://lsub.org/magic/group as an example
> It uses a directory per web group, with a file per article (and replies to it).
> Several skeleton html files are used and a DYNAMIC tag is replaced with
> contents using a C program.

Yeah, simple, but effective :)
With modern browsers you could even move this logic into the client:
It just reads the files (using HTTP as fs) and builds up everything
else on its own, using javascript.

> Also, because we use the real FS to keep the data, we don't need a synthetic
> file system program to implement any interface whatsoever (unless you count the
> web as such thing).

But a synthetic filesystem would also be fine for creating everything
on-the-fly. POST requests could be sent through a special file.
The actual http stuff would just be a tiny frontend :)


cu
--
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [9fans] dbfs and web framework for plan 9 (was: web-based plan 9?)

[Giacomo Tesio]

[-- Attachment #1: Type: text/plain, Size: 5654 bytes --]

On Sun, Nov 23, 2008 at 11:06 PM, Enrico Weigelt <weigelt@metux.de> wrote:

> * Giacomo Tesio <giacomo@tesio.it> wrote:
>
> Hi,


Hi! Thanks for your reply.

A premise: I'm not talking about developing a web forum or the like.

I'm talking about a framework to develop large applications with "thin
clients" (as many call the browsers... :-D) which have more complex
requirement than a web forum.

Forums, I think, don't need a database... (even if a database _could_
increase developers productivity)


> > The scope is to be able to query the db towards a filesystem, with no
> need
> > to use different libraries for different db backend... each application
> > would use a filesystem, while each rdbms would have it's own filesystem
> (all
> > with the same structure, obviously)
>
>
> That's a really good idea. But there's still one problem to solve:
> filesystem hierachies and relational algebra are fundamentally
> different concepts. So, if you don't want to just the fs as some
> universal sql-transport, data has to be modeled differently.


Well... to keep things simples, yes...
I thinks there's no need to reinvent the wheel, since SQL is so diffused.

And, thinking a bit more, I'm wondering whether v9fs could help to build
such a filesystem on linux (where PostgreSQL run) and expose it to plan 9
clients
(actually I don't know what v9fs is and how it works... it would be "easy"
to develop a fuse synthetic filesystem for this, but exporting with v9fs it
would introduce an unnecessary layer, so build directly in v9fs would be
better, if possible).

I'm thinking about a (new) "real world" application of plan 9.
In the "real world" I know, men's productivity is really important (80% of
the costs)
Without insult for anyone, I don't want to get a new "toy" for university
research... (even I would be glad to work with students on it), I want
something which could improve seriously the productivity of a real team
(with eterogeneous skills) on reals web applications development.

A forum or a web calendar is not (per se) such a real application.
With all the forums out there, it would be more a didactical hack.




> Actually, in many webapps, the real underlying models are neither
> relational nor hierachic, but most times just happen to be
> modelled relatiolally, because rdmbs'es are the tool of choice
> for most people.


Aren't they?


> So what we have to do here is to get back to the original problems
> to solve and think about how they could fit into an hierachy
> instead of relation.
> ...
> Okay, we've got some more data, eg. profile information, maybe all
> the other "web-2.0" stuff (user galleries, friend relations, etc),
> but IMHO this all easily fits in hierachies if we start thinking
> afresh with a blank paper.
> ...
> That's all possible, if we just can imagine it :)
>

You're right, and I'm sure it would be a wonderful research.

Actually I've also thinked about such a solution (well... more or less) and
came to BerkleyDB XML.

Take a look to it. It could be a good source of ideas.



> Simple example: a web-forum.



Too simple. In previous post I talked about an ecommerce, and yet I was
thinking it was a too simple example.

BTW, if you miss a post on a forum, you would be hungry.
But what if you pay for and order without receving the product you paid for?



> > Both are really important.
> >
> > When a product change its code (aka pk) for business' reasons, I need
> have
> > it changed automatically in any place its referred in the database.
> > Or if a dumb employee try to delete a product category (supposing he's
> > allowed) I have to be sure that the remaining data don't loose meaning.
>
>
> That's exactly what constraints, rules in SQL etc are for.
> Maybe some similar ruling system for filesystems would be fine :)
> (any suggestions ?)


Mmm.. I'm not sure.

Too complex, for a _file_system.


>
>
> > Application should HANDLE data.
> > But datas are a rapresentation of a reality which have a meaning!
> > A relational database don't have to know how datas are collected or used.
> > But it has to keep meaningfull the rappresentation.
>
> Right. But the database has to know rules which data has to comply,
> so a wide range of errors becomes impossible, mathematically provable.


Sure. It was what I was tring to say... sorry for my english! :-D

This is WHY relational database are needed: to be istructed to check for
data integrity/consistency.



> > I need to be able to atomically save related tuples in different tables,
> > since (for example) those different tables could be simply modelling ONE
> > real entity (say an order or a family).
>
> CREATE RULE .. AS ON INSERT TO ...
>
>
> Again, a rule system for filesystems would be really cool :)


I think that rule (or triggers) on any filesystem would be too complex.

You would loose the simplicity of the paradigm.

But I must admit that omero (completely offtopic here) make me think a bit
about such a thing: what if we could add to each element interface/file
events in the form of rc scripts?

Say you have a button in the interface: the button is a directory (that's
what I've understood, never tried), what if when the button is cliccked the
click.rc script be executed?

Well... may be this is actually done. Or there may be optimal reasons to
avoid this.
It was just an idea.

But it seem to me wrong to generalize such a system.
You would loose in simplicity, which is what make Plan 9 concepts so good!



Giacomo Tesio

[-- Attachment #2: Type: text/html, Size: 7614 bytes --]

Re: [9fans] dbfs and web framework for plan 9 (was: web-based plan 9?)

[Giacomo Tesio]

[-- Attachment #1: Type: text/plain, Size: 111 bytes --]

>
> BTW, if you miss a post on a forum, you would be *hungry*.
>

LOOL! ! ! I was meaning... angry... :-D

[-- Attachment #2: Type: text/html, Size: 333 bytes --]

Re: [9fans] dbfs and web framework for plan 9

[matt]

> That's exactly what constraints, rules in SQL etc are for.
> Maybe some similar ruling system for filesystems would be fine :)
> (any suggestions ?)
>
That's what I was driving at. To map SQL <> FS you end up replicating
lots of SQL logic in your client FS. Reads are *always* out of date.
Writes can happen across tables but need to be atomic and able to roll back.

Funny you should mention rules for filesystems. I also did the opposite
and wrote a front end that mapped 9p onto SQL. I.E. a scehma that
represented a file system with QIDs etc. with a python marshaller
inbetween such that Postgres was the backing store for 9p.

I got a bit stuck when the file sizes got over the row size, the blob
spec wasn't very clearly documented to me; all you get is the C function
definitions rather than what goes over the wire so it's another project
in the unfinished pile :)

I've got a few weeks off soon and I'm going to revisit my py9p :)

Re: [9fans] dbfs and web framework for plan 9

[Giacomo Tesio]

[-- Attachment #1: Type: text/plain, Size: 1961 bytes --]

On Mon, Nov 24, 2008 at 11:58 AM, matt <mattmobile@proweb.co.uk> wrote:

>
>  That's exactly what constraints, rules in SQL etc are for. Maybe some
>> similar ruling system for filesystems would be fine :)
>> (any suggestions ?)
>>
>>
> That's what I was driving at. To map SQL <> FS you end up replicating lots
> of SQL logic in your client FS. Reads are *always* out of date. Writes can
> happen across tables but need to be atomic and able to roll back.
>

Probably I was not clear on what I'm thinking about.

I think that rebuild a relational database on a filesystem is (quite)
pointless.


What I'm proposing is to design/develop the interface to interact with
(any?) rdbms through a filesystem.

A kind of "proxy" to the db with a filesystem interface.

A "draft" could be (even if I've already found some problems in it):


>    - a "ctrl" file which accept only COMMIT, ROLLBACK and ABORT
>    - an "error" file
>    - a "notice" file (postgresql has RAISE NOTICE... may be others have it
>    too)
>    - a "data" file (append only in the transaction, but not outside) where
>    the INSERT, UPDATES, DELETE and all the DDL and DCL commands will be written
>    - each SELECT have to be written in a different file (named
>    sequentially): after writing the query, the same file could be read to get
>    the results (in xml...)
>    - on errors, sequents writes fails and error file will be readable
>
> The problems:

   - transaction -> directory conversion require creating a new connection
   to the backend (if I'm right thinking that transaction are connection wide)
   - xml output of fetchable results (SELECT, FETCH, SHOW...) require a tool
   to easily query such an output. It seem Plan 9 miss such a tool. xmlfs
   actually is unsuitable.
   (I'm thinking about an xpath command accepting xml in stdin and the xpath
   query as an argument, and return to stdout the results)


Giacomo

[-- Attachment #2: Type: text/html, Size: 2937 bytes --]