* [9fans] venti workalike in Python
@ 2003-12-10 2:00 John E. Barham
2003-12-10 2:31 ` Russ Cox
` (3 more replies)
0 siblings, 4 replies; 8+ messages in thread
From: John E. Barham @ 2003-12-10 2:00 UTC (permalink / raw)
To: 9fans
I'm attempting to decipher the venti and vac code and write a workalike in
Python. (packet.c had me bogged down for a while...) At this point I don't
care that much about compatibility, and can't really test it as I don't have
a running instance of Plan 9. But as has been noted here before, the venti
procol is relatively trivial--hello, ping, read, write, goodbye, error,
although the two auth messages appear to be unused for the moment?--so
making it compatible at some point shouldn't be too much of a problem.
Having said that, is there any documentation on the venti protocol apart
from the source?
I'm not planning on re-implementing venti's arena code but thought that it
would be quick and easy for testing purposes to implement the back-end
storage using something like dbm or, mea culpa, a relational database. (I
added binary type support to the Python extension for SQLite with this in
mind.) Thoughts?
John
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [9fans] venti workalike in Python
2003-12-10 2:00 [9fans] venti workalike in Python John E. Barham
@ 2003-12-10 2:31 ` Russ Cox
2003-12-10 2:51 ` Geoff Collyer
2003-12-10 22:52 ` William Josephson
` (2 subsequent siblings)
3 siblings, 1 reply; 8+ messages in thread
From: Russ Cox @ 2003-12-10 2:31 UTC (permalink / raw)
To: 9fans
> I'm attempting to decipher the venti and vac code and write a workalike in
> Python. (packet.c had me bogged down for a while...) At this point I don't
The discussion of the Vac file system format in /sys/doc/fossil.ps
may be helpful.
> care that much about compatibility, and can't really test it as I don't have
> a running instance of Plan 9. But as has been noted here before, the venti
> procol is relatively trivial--hello, ping, read, write, goodbye, error,
> although the two auth messages appear to be unused for the moment?--so
> making it compatible at some point shouldn't be too much of a problem.
> Having said that, is there any documentation on the venti protocol apart
> from the source?
Not really, but here's a summary.
Venti uses a simple RPC protocol with only a few messages:
size[2] RError tag[1] error[s]
size[2] QPing tag[1]
size[2] RPing tag[1]
size[2] QHello tag[1] version[s] uid[s] strength[1]
ncrypto[1] crypto[ncrypto]
ncodec[1] codec[ncodec]
size[2] RHello tag[1] sid[s] crypto[1] codec[1]
size[2] QGoodbye tag[1]
size[2] QRead tag[1] score[20] type[1] 0[1] count[2]
size[2] RRead tag[1] data[]
size[2] QWrite tag[1] type[1] 0[3] data[]
size[2] RWrite score[20]
size[2] QSync tag[1]
size[2] RSync tag[1]
In the notation, field[s] indicates a string: a two-byte length followed
by that many bytes.
The actual message types are one-byte values:
enum
{
VtRError = 1,
VtQPing,
VtRPing,
VtQHello,
VtRHello,
VtQGoodbye,
VtRGoodbye, /* not used */
VtQAuth0, /* not used */
VtRAuth0, /* not used */
VtQAuth1, /* not used */
VtRAuth1, /* not used */
VtQRead,
VtRRead,
VtQWrite,
VtRWrite,
VtQSync,
VtRSync,
VtMaxOp
};
Transactions and tags are like in 9P, except that tags are one byte
and the request messages have names beginning with Q instead of T.
The strength, crypto, and codec stuff was never fleshed out and
is essentially unused.
> I'm not planning on re-implementing venti's arena code but thought that it
> would be quick and easy for testing purposes to implement the back-end
> storage using something like dbm or, mea culpa, a relational database. (I
> added binary type support to the Python extension for SQLite with this in
> mind.) Thoughts?
We use Berkeley DB at MIT for another project to store 8kB blocks indexed
by SHA1 hash. It's adequate.
Russ
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [9fans] venti workalike in Python
2003-12-10 2:31 ` Russ Cox
@ 2003-12-10 2:51 ` Geoff Collyer
0 siblings, 0 replies; 8+ messages in thread
From: Geoff Collyer @ 2003-12-10 2:51 UTC (permalink / raw)
To: 9fans
Berkeley DB has grown a lot over the years. I recently ported Ken's
dbm library from V7 to native Plan 9, and modified it to store all
integers on disk in big-endian order, so that dbm databases can be
shared across architectures (there are still locking problems to be
solved for concurrent access, though). I did the same for Chris
Torek's extended dbm, mdbm, which adds multiple keys per value and the
ability to have multiple databases open simultaneously. My mdbm port
will also run on Unix, so one can share my mdbm databases across
architectures there too. All of this has been only lightly exercised,
but if you'd like a copy, let me know.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [9fans] venti workalike in Python
2003-12-10 2:00 [9fans] venti workalike in Python John E. Barham
2003-12-10 2:31 ` Russ Cox
@ 2003-12-10 22:52 ` William Josephson
2003-12-11 7:20 ` John E. Barham
2003-12-11 9:46 ` John E. Barham
3 siblings, 0 replies; 8+ messages in thread
From: William Josephson @ 2003-12-10 22:52 UTC (permalink / raw)
To: 9fans
On Tue, Dec 09, 2003 at 06:00:41PM -0800, John E. Barham wrote:
> I'm not planning on re-implementing venti's arena code but thought that it
> would be quick and easy for testing purposes to implement the back-end
> storage using something like dbm or, mea culpa, a relational database. (I
> added binary type support to the Python extension for SQLite with this in
> mind.) Thoughts?
I'd use Berkeley DB. In fact I have a small prototype system
that does just that using 4K data blocks. For small amounts
of data it works fine and I'd expect it to be OK up to at
least a hundred gigabytes.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [9fans] venti workalike in Python
2003-12-10 2:00 [9fans] venti workalike in Python John E. Barham
2003-12-10 2:31 ` Russ Cox
2003-12-10 22:52 ` William Josephson
@ 2003-12-11 7:20 ` John E. Barham
2003-12-11 9:46 ` John E. Barham
3 siblings, 0 replies; 8+ messages in thread
From: John E. Barham @ 2003-12-11 7:20 UTC (permalink / raw)
To: 9fans
> > Having said that, is there any documentation on the venti protocol apart
> > from the source?
>
> Not really, but here's a summary.
>
> Venti uses a simple RPC protocol with only a few messages:
> ...
Many thanks.
> size[2] QSync tag[1]
> size[2] RSync tag[1]
These aren't in the source I have. Presumably they were added relatively
recently. I assume they mean save all transactions to disk?
John
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [9fans] venti workalike in Python
2003-12-10 2:00 [9fans] venti workalike in Python John E. Barham
` (2 preceding siblings ...)
2003-12-11 7:20 ` John E. Barham
@ 2003-12-11 9:46 ` John E. Barham
2003-12-11 16:23 ` Russ Cox
3 siblings, 1 reply; 8+ messages in thread
From: John E. Barham @ 2003-12-11 9:46 UTC (permalink / raw)
To: 9fans
> > Having said that, is there any documentation on the venti protocol apart
> > from the source?
>
> Not really, but here's a summary.
>
> Venti uses a simple RPC protocol with only a few messages:
> ...
Many thanks.
> size[2] QSync tag[1]
> size[2] RSync tag[1]
These aren't in the source I have. Presumably they were added relatively
recently. I assume they mean save all transactions to disk?
John
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [9fans] venti workalike in Python
2003-12-11 9:46 ` John E. Barham
@ 2003-12-11 16:23 ` Russ Cox
0 siblings, 0 replies; 8+ messages in thread
From: Russ Cox @ 2003-12-11 16:23 UTC (permalink / raw)
To: 9fans
> > > Having said that, is there any documentation on the venti protocol apart
> > > from the source?
> >
> > Not really, but here's a summary.
> >
> > Venti uses a simple RPC protocol with only a few messages:
> > ...
>
> Many thanks.
>
> > size[2] QSync tag[1]
> > size[2] RSync tag[1]
>
> These aren't in the source I have. Presumably they were added relatively
> recently. I assume they mean save all transactions to disk?
I added them about a year ago. Sync means "don't return until everything
I've sent up to this point is committed to disk".
Russ
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [9fans] venti workalike in Python
@ 2003-12-10 12:11 Geoff Collyer
0 siblings, 0 replies; 8+ messages in thread
From: Geoff Collyer @ 2003-12-10 12:11 UTC (permalink / raw)
To: 9fans
I've had a request, so I've added libdbm.tar and libmdbm.tar to my
Plan 9 software page, http://www.collyer.net/~geoff/9.
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2003-12-11 16:23 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2003-12-10 2:00 [9fans] venti workalike in Python John E. Barham
2003-12-10 2:31 ` Russ Cox
2003-12-10 2:51 ` Geoff Collyer
2003-12-10 22:52 ` William Josephson
2003-12-11 7:20 ` John E. Barham
2003-12-11 9:46 ` John E. Barham
2003-12-11 16:23 ` Russ Cox
2003-12-10 12:11 Geoff Collyer
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).