[9fans] venti workalike in Python

[9fans] venti workalike in Python

[John E. Barham]

I'm attempting to decipher the venti and vac code and write a workalike in
Python.  (packet.c had me bogged down for a while...)  At this point I don't
care that much about compatibility, and can't really test it as I don't have
a running instance of Plan 9.  But as has been noted here before, the venti
procol is relatively trivial--hello, ping, read, write, goodbye, error,
although the two auth messages appear to be unused for the moment?--so
making it compatible at some point shouldn't be too much of a problem.
Having said that, is there any documentation on the venti protocol apart
from the source?

I'm not planning on re-implementing venti's arena code but thought that it
would be quick and easy for testing purposes to implement the back-end
storage using something like dbm or, mea culpa, a relational database.  (I
added binary type support to the Python extension for SQLite with this in
mind.)  Thoughts?

    John

Re: [9fans] venti workalike in Python

[Russ Cox]

> I'm attempting to decipher the venti and vac code and write a workalike in
> Python.  (packet.c had me bogged down for a while...)  At this point I don't

The discussion of the Vac file system format in /sys/doc/fossil.ps
may be helpful.

> care that much about compatibility, and can't really test it as I don't have
> a running instance of Plan 9.  But as has been noted here before, the venti
> procol is relatively trivial--hello, ping, read, write, goodbye, error,
> although the two auth messages appear to be unused for the moment?--so
> making it compatible at some point shouldn't be too much of a problem.
> Having said that, is there any documentation on the venti protocol apart
> from the source?

Not really, but here's a summary.

Venti uses a simple RPC protocol with only a few messages:

	size[2] RError   tag[1] error[s]

	size[2] QPing    tag[1]
	size[2] RPing    tag[1]

	size[2] QHello   tag[1] version[s] uid[s] strength[1]
	                              ncrypto[1] crypto[ncrypto]
	                              ncodec[1] codec[ncodec]
	size[2] RHello   tag[1] sid[s] crypto[1] codec[1]

	size[2] QGoodbye tag[1]

	size[2] QRead    tag[1] score[20] type[1] 0[1] count[2]
	size[2] RRead    tag[1] data[]

	size[2] QWrite   tag[1] type[1] 0[3] data[]
	size[2] RWrite   score[20]

	size[2] QSync    tag[1]
	size[2] RSync    tag[1]

In the notation, field[s] indicates a string: a two-byte length followed
by that many bytes.

The actual message types are one-byte values:

enum
{
	VtRError = 1,

	VtQPing,
	VtRPing,
	VtQHello,
	VtRHello,
	VtQGoodbye,
	VtRGoodbye,	/* not used */
	VtQAuth0,	/* not used */
	VtRAuth0,	/* not used */
	VtQAuth1,	/* not used */
	VtRAuth1,	/* not used */
	VtQRead,
	VtRRead,
	VtQWrite,
	VtRWrite,
	VtQSync,
	VtRSync,

	VtMaxOp
};

Transactions and tags are like in 9P, except that tags are one byte
and the request messages have names beginning with Q instead of T.
The strength, crypto, and codec stuff was never fleshed out and
is essentially unused.

> I'm not planning on re-implementing venti's arena code but thought that it
> would be quick and easy for testing purposes to implement the back-end
> storage using something like dbm or, mea culpa, a relational database.  (I
> added binary type support to the Python extension for SQLite with this in
> mind.)  Thoughts?

We use Berkeley DB at MIT for another project to store 8kB blocks indexed
by SHA1 hash.  It's adequate.

Russ

Re: [9fans] venti workalike in Python

[Geoff Collyer]

Berkeley DB has grown a lot over the years.  I recently ported Ken's
dbm library from V7 to native Plan 9, and modified it to store all
integers on disk in big-endian order, so that dbm databases can be
shared across architectures (there are still locking problems to be
solved for concurrent access, though).  I did the same for Chris
Torek's extended dbm, mdbm, which adds multiple keys per value and the
ability to have multiple databases open simultaneously.  My mdbm port
will also run on Unix, so one can share my mdbm databases across
architectures there too.  All of this has been only lightly exercised,
but if you'd like a copy, let me know.

Re: [9fans] venti workalike in Python

[William Josephson]

On Tue, Dec 09, 2003 at 06:00:41PM -0800, John E. Barham wrote:
> I'm not planning on re-implementing venti's arena code but thought that it
> would be quick and easy for testing purposes to implement the back-end
> storage using something like dbm or, mea culpa, a relational database.  (I
> added binary type support to the Python extension for SQLite with this in
> mind.)  Thoughts?

I'd use Berkeley DB. In fact I have a small prototype system
that does just that using 4K data blocks.  For small amounts
of data it works fine and I'd expect it to be OK up to at
least a hundred gigabytes.

Re: [9fans] venti workalike in Python

[John E. Barham]

> > Having said that, is there any documentation on the venti protocol apart
> > from the source?
>
> Not really, but here's a summary.
>
> Venti uses a simple RPC protocol with only a few messages:
> ...

Many thanks.

> size[2] QSync    tag[1]
> size[2] RSync    tag[1]

These aren't in the source I have.  Presumably they were added relatively
recently.  I assume they mean save all transactions to disk?

John

Re: [9fans] venti workalike in Python

[John E. Barham]

> > Having said that, is there any documentation on the venti protocol apart
> > from the source?
>
> Not really, but here's a summary.
>
> Venti uses a simple RPC protocol with only a few messages:
> ...

Many thanks.

> size[2] QSync    tag[1]
> size[2] RSync    tag[1]

These aren't in the source I have.  Presumably they were added relatively
recently.  I assume they mean save all transactions to disk?

John

Re: [9fans] venti workalike in Python

[Russ Cox]

> > > Having said that, is there any documentation on the venti protocol apart
> > > from the source?
> >
> > Not really, but here's a summary.
> >
> > Venti uses a simple RPC protocol with only a few messages:
> > ...
>
> Many thanks.
>
> > size[2] QSync    tag[1]
> > size[2] RSync    tag[1]
>
> These aren't in the source I have.  Presumably they were added relatively
> recently.  I assume they mean save all transactions to disk?

I added them about a year ago.  Sync means "don't return until everything
I've sent up to this point is committed to disk".

Russ

Re: [9fans] venti workalike in Python

[Geoff Collyer]

I've had a request, so I've added libdbm.tar and libmdbm.tar to my
Plan 9 software page, http://www.collyer.net/~geoff/9.