Re: [9fans] Venti over DHT

[Enrico Weigelt <weigelt@metux.de>]

Russ Cox wrote:

Hi,

> There was no real code to speak of.  It was a draft of a draft.
> I did some calculations of block-level commonality using a
> few trivial programs that hashed each block of every file in
> a tree, but you could recreate that in 100 lines of C or shell script.
> We never stored any blocks in the DHT.

I've also done some bits of works in that area
(nothing usable yet ;-o), but with different
requirements:

* storage near to the user (at least local mirrors)
* equal data should get equal score (even w/ encryption)
* automatic removal of stale blocks -> garbage collection
* efficient also on small data blocks

Especially the distributed GC together w/ encryption turned out
not be quite trivial ;-o
On the one hand we need things like timestamps, on the other hand
we need to trace the tree structures w/o decryption.

So I added several block types: eg. blob (payload data) and inode
(holding the tree). blobs are encrypted w/ their score, thus
reading takes scores (outer for retrieval, inner for decryption).
inodes have an public area, listing just the outer scores and an
encrypted one holding the inner scores. The tree formed by inodes
doesn't necessarily has to be fully balanced, so incremental writes
with special heuristics for preventing unncessary writes are possible.


If someone likes to hear more about this, just let me know :)


cu
--
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------