[9fans] neoventi: call for initial testing

[9fans] neoventi: call for initial testing

[noam]

My reimplementation of venti[1], presented previously at IWP9, has
gone from "this will DEFINITELY eat your data in the first five
seconds" to "this will probably eat your data in the first five
minutes."

Any brave souls interested in testing it are invited to do so.  I'll
definitely look at any stack traces or bugs or corruption you find,
but I can't promise to be able to recover data.  It'll _probably_ be
recoverable.  Probably.

Use at your own risk, yadda yadda, I'm only starting testing it with
fossil presently, but all the obvious tests I could think of offhand
are passing.

That means that all the remaining bugs are _subtle_, which is _more_
dangerous.

- Noam Preil

[1] https://git.sr.ht/~pixelherodev/neoventi




------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Mdc787dc2648256807ea74cbe
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[9fans] Re: neoventi: call for initial testing

[wb.kloke]

[-- Attachment #1: Type: text/plain, Size: 736 bytes --]

I am going to try your code. Just a question at the start: Presumably, I have to prepare arena space and indexsects by the original utilities, or else?

Perhaps you remember that I am a critic of the index handling in the original venti, because it will rewrite an entire disk block at random locations for every clump added. I would never use it on a SSD, only on magnetic discs. My 9front installation is on a FreeBSD15 bhyve vm on a ssd system. I would need to supply arena and index via NFS or AOE.


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M1e1bc4e3035480b7d1f54b81
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 1290 bytes --]

Re: [9fans] Venti: design discussion

[wb.kloke]

[-- Attachment #1: Type: text/plain, Size: 1186 bytes --]

Let me first give the address of my own solution (not plan9, but planport):
https://github.com/vestein463/plan9port
under src/cmd/venti/srv/mventi.c

The implementation uses the original arena format, but does not need indexsects or bloom filter (bloom filter is not useful in this case as the trie implementation of the index is just effective enough) , but of course sufficient main memory to hold the index.

I read the directories of the arenas at startup. This takes  some 3 Minutes for 80GB arenas.

The tricky thing is to avoid the storage (20 mostly useless bytes)  of the full scores in the index tree. If you read data, you expect the score to be present already on the arena partitition, so you only need the leading bytes of the score to find that. 

The other tricky thing is the decision whether to use 16-way or 8-way trie, because the last node will contain few entries, but the all of the upper ones are full or nearly full.


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Tb88459bd8ed4d095-Mdb8eefaae896b634360abdbf
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 1914 bytes --]

Re: [9fans] Venti: design discussion

[Steve Simon]

this is fortune-worthy.

> On 10 Dec 2025, at 7:55 pm, noam@pixelhero.dev wrote:
> 
> ideas are easy, code is easy, good design is hard.

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Tb88459bd8ed4d095-M71f8d362b0716d6550de1b31
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Re: neoventi: call for initial testing

[ori]

Quoth wb.kloke@gmail.com:
> I would never use it on a SSD, only on magnetic discs. 

Note that reasonable quality SSDs are rated for writing
something like 1000x their capacity. If you assume that
vendors are lying through their teeth, and you get only
10% of the advertised number of writes, that would let you
write the full disk from to back 100 times before you got
a disk failure.

Since venti is append-only, the amount of data written is
limited to the size of the disk; if venti writes 4 index
blcoks for each data block, and you filled the entire disk,
that would put your writes at 5x capacity, which is a long
way away from the rated 1000x capacity, or even the 100x
capacity we had assumed.

I think venti is fine for write capacity.

The bigger risk is that SSDs need to remain powered on
so that they don't lose data; thankfully, the index can
be rebuilt if needed, so if you want your data to be long
lived, I'd put the index purely on fast SSD, and the data
on spinnning rust.


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Mbc1059bea540fefe4c8a8e89
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Re: neoventi: call for initial testing

[wb.kloke]

[-- Attachment #1: Type: text/plain, Size: 579 bytes --]

Allow me to disagree. Arenas are, mostly, append-only. Index is write-random. For each clump about 40 bytes are changed at random locations. Therefore, you are are safe filling up the arena disc entirely,  but the index disc will be weared up, for nothing. Most ssd vendors recommend using a very low percentage of nominal capacity for mit use cases.
------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Mf5ca359ba61f866ed6ee4aa5
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 1077 bytes --]

Re: [9fans] Venti: design discussion

[wb.kloke]

[-- Attachment #1: Type: text/plain, Size: 310 bytes --]

Writes on the main area can be Cached, on the index caching is entirely useless.

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Tb88459bd8ed4d095-Mfb95aaaf61979016b39a6aad
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 802 bytes --]

Re: [9fans] Re: neoventi: call for initial testing

[ori]

Quoth wb.kloke@gmail.com:
> Allow me to disagree. Arenas are, mostly, append-only. Index is write-random.

Once a disk is full, there are no updates that can be done. Venti simply can't
produce enough writes to wear out an index on the same disk. You run out of
disk space first.

And an index on a separate disk is disposable; you can replace the disk and
generate a new index with no data loss.


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Ma543b8df302c225274aa014c
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Re: neoventi: call for initial testing

[ori]

Quoth noam@pixelhero.dev:
> and say "look, its worn out in less than a year from being used as an
> index"?

This would be nearly impossible to prove. But, smart stas showing that
there was significantly more IO than the manufacturer recommended would
be a good minimum demonstration.


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M2494c5542b657dff02bd93f2
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Re: neoventi: call for initial testing

[wb.kloke]

[-- Attachment #1: Type: text/plain, Size: 575 bytes --]

Please,  read before answering.  It doesn't matter, how full the arena disc is. Just add more arena space,  possibly on a new disc. The problem with indexsects is entirely different. They are either very sparsely populated, which makes the wear-up worse, or mostly full, which is good for almost purposes, except reliable space for New storage.
------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Me5311b475212c91c42730936
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 1079 bytes --]

Re: [9fans] Re: neoventi: call for initial testing

[wb.kloke]

[-- Attachment #1: Type: text/plain, Size: 430 bytes --]

It siems that you did not remember, that the whole thing startet by the report of one member on this list, that his installation, which provided plan9 software failed after using up more than 1 ssd.


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Me95c43ea69968713ba3e8e8c
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 954 bytes --]

Re: [9fans] Venti: design discussion

[wb.kloke]

[-- Attachment #1: Type: text/plain, Size: 264 bytes --]

For reading, maybe. For writing, no.
------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Tb88459bd8ed4d095-M537e519a9a5aa9feeee09d3e
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 752 bytes --]

Re: [9fans] Re: neoventi: call for initial testing

[ori]

Quoth wb.kloke@gmail.com:
> 
> 
> It siems that you did not remember, that the whole thing startet by the report of one member on this list, that his installation, which provided plan9 software failed after using up more than 1 ssd.
> 

Hardware dies sometimes. Especially SSDs, if they have spent a good portion of their lives powered off.
For example, some models used in production at Backblaze had failure rates approaching 10%/year:

        https://backblazeprod.wpenginepowered.com/wp-content/uploads/2023/09/7-Select-SSD-Lifetime-AFR-Table-1.png

What has been done to show that SSDs used under venti die at a higher rate than they would otherwise?

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M9da3859af50b07371f67729d
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Re: neoventi: call for initial testing

[wb.kloke]

[-- Attachment #1: Type: text/plain, Size: 379 bytes --]

Sorry for answering myself. My last to mails are answers to different entries before.
The thread I was recurring in the 2nd , was "thowing the towel".
------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M750a681b030ad27464a5d813
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 898 bytes --]

Re: [9fans] Venti: design discussion

[noam]

Quoth wb.kloke@gmail.com:

> Just a question at the start: Presumably, I have to
prepare arena space and indexsects by the original utilities, or else?

Ah, yes.

> Perhaps you remember that I am a critic of the index handling in the
original venti, because it will rewrite an entire disk block at random
locations for every clump added.  I would never use it on a SSD, only
on magnetic discs.

That - is a very fair point.

I'd be happy to discuss what the next version of the index should look
like.  My current goal is for neoventi to be, basically, compatible
and mostly-usable by the EOY, and then work on improved arena and
index formats as the next step.

I'm not sure how to avoid it with the index.  Bakcing away from
venti's implementation details for a second, let's state the
requirements, and then look at the problems

- We're storing a hash table on disk - The data blocks should be
stored in an append-only fashion

A lot of the useful properties come from those two designed-in
properties: deduplication, rapid sync, the ability to fork and reset a
file system, etc.

Even more simply: we want to store data in an append-only log, in
which the data is identified by its contents, not by its address.

That means, by necessity, that it is _not possible_ to know the
address of a given block, from its score!

The existence of an index is unavoidable.  The question is, can we
design one that does not require random writes, without unacceptable
trade offs?

Which is almost a trick question, as venti, in fact, already has one:
each arena contains a shortened summary that can be used to quickly
build the index on startup, without scanning the entire arena
contents.

But, if we relied on that to start up - and not as a backup with which
to construct the index - we'd be unable to handle any reads until we'd
read the entire thing from disk.  The current index is random-access,
which means we can be handling requests _without reading the whole
thing into memory._

We could maybe combine the two?

Have an index section on disk that's just a linear log of (key, value)
entries, that we load on startup - which would be faster than the
current arena scan, as it would be one single massive section to read
- and on startup, construct the entire index in memory from it?

...and, tbh, with how much faster neoventi already is, I can likely
experiment with just loading those arena sections on startup.  It
might be sufficiently fast anyways.  For 1TiB of data, that would be
~10GiB, and still _mostly_ linear [there's seeks between sectoins, but
sections are each a few MiB - so we might reasonably be able to hit,
say, 50MB/s on a HDD, or ~3 minutes?]

and that might be pretty fast on a modern drive, too, and eliminate
the random writes entirely?

As you can probalby see, I have a lot of _ideas_ on how to do this,
but - ideas are easy, code is easy, good design is hard.

- Noam Preil

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Tb88459bd8ed4d095-M2977805c85bddb876a811d44
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Venti: design discussion

[noam]

> That - is a very fair point.

Actually, thinking it over more, this doesn't really make sense.  The
index will receive less writes than the main arena; even accounting
for the multiple writes per bucket, the idea that this would cause
excessive wear seems quite silly.



------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Tb88459bd8ed4d095-M0897ee45450b22af8681ede0
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Venti: design discussion

[noam]

Quoth wb.kloke@gmail.com: > Writes on the main area can be Cached, on
the index caching is entirely useless.


That's empirically not true.  I have disk caching in neoventi and it
absolutely does help with the index as well, I can turn it off to
prove it :)

Not as _much_, but it does.  Note that each cached block, in venti too
IIRC, contains 8KiB.  It contains several unrelated index buckets;
something on the order of 250 index entries.  That's enough to index
nearly 1MiB.

If you scan through 100MiB of data, we may well only do 100-200 disk
reads from the index.

It's not great, but it's not that bad.


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Tb88459bd8ed4d095-M49206b86f071469356ae9569
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Re: neoventi: call for initial testing

[noam]

Quoth wb.kloke@gmail.com:

> Allow me to disagree.  Arenas are, mostly, append-only.  Index is
write-random.  For each clump about 40 bytes are changed at random
locations.  Therefore, you are are safe filling up the arena disc
entirely,  but the index disc will be weared up, for nothing.  Most
ssd vendors recommend using a very low percentage of nominal capacity
for mit use cases.


> for each clump, about 40 bytes are changed

Average clump size is on the order of 10KiB, or ~20 sectors.  Looking
at orders of magnitude, we're talking about roughly one (random)
sector written to the index for every 20 (sequential) sectors written
to the arena.

Moreover, arena writes ALSO have random writes!  Each write to the
arena, in addition to the sequential clump, seeks to the end of the
arena to add the index entry so that, if the index dies, it can be
rebuilt without scanning the actual data area.  There should be one of
those for each write to the index.

Yes, the index write is _more_ random, but there's also _so few of
them_ that it shouldn't matter.

keep in mind as well that the index is intended to be sized such that
it never hits 100% of usage.

If you have actual data to show that this is a problem, I'm happy to
take a look, but it seems to be speculative and based on ideas of how
SSDs _should_ work, not based on any actual SSD that you can point to
and say "look, its worn out in less than a year from being used as an
index"?



------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Mafd2cbacdf7b4c1799ce1c77
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Re: neoventi: call for initial testing

[noam]

Quoth ori@eigenstate.org:
> Quoth wb.kloke@gmail.com:
> > Allow me to disagree. Arenas are, mostly, append-only. Index is write-random.
> 
> Once a disk is full, there are no updates that can be done. Venti simply can't
> produce enough writes to wear out an index on the same disk. You run out of
> disk space first.
> And an index on a separate disk is disposable; you can replace the disk and
> generate a new index with no data loss.


Basically this.

Either you have the index on the same disk as the arenas, and the
arenas _will_ cause more wear; or, you have it on a separate disk, and
that disk can be treated as consumable.

I'm thinking about how to make it easy to build the index on startup
into a ramdisk, but this is not a priority right now.



------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M095a9f4ff18ea569dd602fe7
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Re: neoventi: call for initial testing

[noam]

Quoth wb.kloke@gmail.com:
> Please,  read before answering.

Please, don't assume that people are responding in bad faith :)

The fullness of the arena disc directly correlates to how bad writing
the index will be, because the index is a tiny fraction of the size of
the arenas.

> which makes the wear-up worse

Maybe this is true, but I've seen no evidence of it, and I've been
running old-venti on SSDs for several years without a single failure.
Do you have any evidence to back this up?


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M21bfc80d662011d4b0bc8313
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Re: neoventi: call for initial testing

[noam]

Quoth wb.kloke@gmail.com: > It siems that you did not remember, that
the whole thing startet by the report of one member on this list, that
his installation, which provided plan9 software failed after using up
more than 1 ssd.

What whole thing?  That might be before i was on the list.

_This_ whole thing, neoventi, started because I was fixing bugs in
venti and decided the code wasn't worth saving, it's too badly
designed and implemented.

I'd need more context to know how serious of a problem that is.  It's
entirely plausible that someone got multiple bad SSDs, that doesn't
actually prove venti caused the failure.  I'd be very curious just how
much data was stored on them to cause that.

I've had multiple SSDs fail running other file systems, within a year
or two, including ext4.  Some SSDs just aren't good, in the same way
that I've had hard drives fail within two weeks.  That doesn't mean
all hard drives are bad, let alone that ext4 ruins hard drives!  What
it really means is that QC isn't perfect and/or they're hoping people
just buy another one and don't refund it.


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M6d9e685eeb63eec56d1ef59f
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Re: neoventi: call for initial testing

[noam]

Quoth ori@eigenstate.org:
> Quoth noam@pixelhero.dev:
> > and say "look, its worn out in less than a year from being used as an
> > index"?
> This would be nearly impossible to prove. But, smart stas showing that
> there was significantly more IO than the manufacturer recommended would
> be a good minimum demonstration.

Yes, that is what I'm talking about - if you can show stats on the
disk that indicate it was being overused, that's a good indication.
Anything else is speculation.

> hardware fails sometimes

Sometimes you just get unlucky.  I had a hard drive
fail within 2 weeks of purchase.  I've had others that were over a
decade old when I got them, still working, a decade later.




------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Md5c5842adf58543cef8f50f4
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Re: neoventi: call for initial testing

[noam]

> > which makes the wear-up worse

I'm going to indulge in some speculation of my own, that actually, it
makes it better.

Any given sector will be written to very, very rarely.  ~40 bytes per
entry, ~512 byte sectors; as the disk approaches full, the average
write count of sectors will approach 13.  And, moreover, the random
distribution will make the wear leveller's job incredibly easy.

I don't really have any evidence to back this up, but, I see no
obvious reason that it should be less correct.


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M826ea895e52c0e99d3ab8136
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Re: neoventi: call for initial testing

[noam]

Anywho.

Neoventi will most likely not have code for building the index.
Instead, _if you want it_, you can specify that your index drive is
stored on a ramdisk and modify your bootrc to build the index on
startup.

neoventi won't care whether your index is on a ramdisk or a regular
one.  If you want to rebuild it on each boot, that's fine.  That
should already work, really, using venti/buildindex.

That said, I _will_ be looking at redesigning the index.  Fairly sure
from previous estimates that i can get at least 2x as many entries per
sector.  Probably more.  The structure needs some serious work,
venti's high-level design is good IMO, but the details are atrocious.

I also intend to make rebuilding the index, using neoventi/buildindex,
significantly faster than it is at present, so that won't be an
entirely unreasonable option.

But, consider the alternative: a 25TiB venti would require something
like 250GiB of RAM. 25TiB of HDD space is much, much, much more
practical than 250GiB of RAM, even today, even _before_ the recent
RAM spikes.

250GiB of SSD space, _even if you assume_ it will wear out quickly,
is still more practical. Rebuilding the index once every other year
would be much more practical than _requiring_ massive RAM usage.



------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Maf3fd5d8fe8f5636744cd684
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] neoventi: call for initial testing

[hahahahacker2009]

[-- Attachment #1: Type: text/plain, Size: 741 bytes --]

I made this change to compile on 4th edition, with Geoff's 9k.
I don't know what is procsetname and how it is different from
threadsetname.  Some format mismatches (%d for ulong and uintptr are
fixed), and there are still many warnings.  4th edition's cc doesn't
allow functions without return at the end.

vac and unvac works, and I tried to set up a fossil and snap -a, but
it constantly says archWalk: ...  unimplemented.  Perhaps make
neoventi fork to background when it is ready to use with fossil?

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te748e03814ed814b-M76bb39d91f848a59782bb890
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: message/rfc822, Size: 14048 bytes --]

From: noam@pixelhero.dev
To: 9fans@9fans.net
Subject: [9fans] neoventi: call for initial testing
Date: Mon, 08 Dec 2025 17:30:27 -0500
Message-ID: <15FCDAB3D76AF86D4FE56DF4E4BFA5E2@pixelhero.dev>

My reimplementation of venti[1], presented previously at IWP9, has
gone from "this will DEFINITELY eat your data in the first five
seconds" to "this will probably eat your data in the first five
minutes."

Any brave souls interested in testing it are invited to do so.  I'll
definitely look at any stack traces or bugs or corruption you find,
but I can't promise to be able to recover data.  It'll _probably_ be
recoverable.  Probably.

Use at your own risk, yadda yadda, I'm only starting testing it with
fossil presently, but all the obvious tests I could think of offhand
are passing.

That means that all the remaining bugs are _subtle_, which is _more_
dangerous.

- Noam Preil

[1] https://git.sr.ht/~pixelherodev/neoventi




------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Mdc787dc2648256807ea74cbe
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #3: neoventi.4e.diff --]
[-- Type: text/plain, Size: 4736 bytes --]

diff -ru /n/pi4/usr/glenda/neoventi/arena.c /usr/glenda/neoventi/arena.c
--- /n/pi4/usr/glenda/neoventi/arena.c	Tue Dec  9 13:45:28 2025
+++ /usr/glenda/neoventi/arena.c	Tue Dec  9 18:22:55 2025
@@ -36,7 +36,7 @@
 	int good;
 	n = arena->arenastats.clumps;
 	corruptchain = 0;
-	fprint(2, "clumps: %d, %d\n", n, arena->indexstats.clumps);
+	fprint(2, "clumps: %ld, %d\n", n, arena->indexstats.clumps);
 	if(n == arena->indexstats.clumps)
 		return 1;
 	buf = nil;
@@ -61,7 +61,7 @@
 		good = arenarepair(arena, (u8int*)ci, addr);
 		addr += (((u64int)(U16GET(ci + 1))) + 38);
 		if(!good){
-			fprint(2, "clump %d is broken, at address %llud: %r\n", m, addr);
+			fprint(2, "clump %ld is broken, at address %lld: %r\n", m, addr);
 			corruptchain += 1;
 			// Skip. old venti creates corruption in some easy-to-repro cases
 			// (e.g. write, kill venti within a few seconds) and I don't need people
@@ -73,14 +73,14 @@
 	}
 	if(addr != goodaddr){
 		assert(corruptchain > 0);
-		fprint(2, "corruption detected: the last %d clumps are corrupt. Dropping them.\n"
+		fprint(2, "corruption detected: the last %ld clumps are corrupt. Dropping them.\n"
 				  "\taddr dropped from %llud to %llud\n",
 		corruptchain, addr, goodaddr);
 		addr = goodaddr;
 		arena->arenastats.clumps -= corruptchain;
 	}
 	if(arena->arenastats.used != addr){
-		fprint(2, "corrupt: found addr %d, expected %d", addr, arena->arenastats.used);
+		fprint(2, "corrupt: found addr %lld, expected %lld", addr, arena->arenastats.used);
 		arena->arenastats.used = addr;
 	}
 	// TODO: we haven't updated indexstats->clumps, so this is safe, but we _do_
diff -ru /n/pi4/usr/glenda/neoventi/cache.c /usr/glenda/neoventi/cache.c
--- /n/pi4/usr/glenda/neoventi/cache.c	Tue Dec  9 13:45:28 2025
+++ /usr/glenda/neoventi/cache.c	Tue Dec  9 18:13:59 2025
@@ -62,13 +62,13 @@
 	if(data == (void*)-1)
 		sysfatal("failed to allocate cache space of %llux", (usize)ENTRYSIZE*ENTRIES);
 	buckets = sbrk(64 + 64*BUCKETS);
-	displacement = ((usize)buckets & 0b111111);
+	displacement = ((usize)buckets & (~(~0 << 6)) ); //0b111111 );
 	if(displacement != 0){
 		displacement = 64 - displacement;
 		buckets = (bucket_t*)((usize)buckets + displacement);
 	}
 	// alignment
-	assert(((usize)buckets&0b111111) == 0);
+	assert(((usize)buckets& (~(~0 << 6)) ) == 0);
 	for(int i = 0; i < BUCKETS; i += 1)
 		memset(buckets[i].index, 0xFF, 32);
 	bucketend = buckets + BUCKETS;
@@ -211,4 +211,5 @@
 	fprint(2, "impossible\n");
 	// DO NOT PROCEED if the cache might be corrupt!
 	abort();
+	return -1; /* shut up the compiler */
 }
diff -ru /n/pi4/usr/glenda/neoventi/disk.c /usr/glenda/neoventi/disk.c
--- /n/pi4/usr/glenda/neoventi/disk.c	Tue Dec  9 13:45:28 2025
+++ /usr/glenda/neoventi/disk.c	Tue Dec  9 18:13:20 2025
@@ -63,6 +63,7 @@
 	}
 	werrstr("internal corruption: arena not found for arenaindex");
 	abort();
+	return -1; /* shut up the compiler */
 }
 
 int
@@ -154,7 +155,7 @@
 	aindex = aindexfromaddr(addr->offset);
 	addr->s_arena = &arenas[aindex];
 	addr->offset -= (addr->s_arena->base - addr->s_arena->blocksize);
-	fprint(2, "entry found in bucket: arena %d, offset %d\n", aindex, addr->offset);
+	fprint(2, "entry found in bucket: arena %lld, offset %lld\n", aindex, addr->offset);
 	return 1;
 }
 
@@ -243,7 +244,7 @@
 {
 	index.arena->arenastats.sealed = 1;
 	if(index.arena->index+1 == numarenas)
-		sysfatal("TODO last arena, %d, full, at size %d!", index.arena->index, index.arena->arenastats.used);
+		sysfatal("TODO last arena, %d, full, at size %lld!", index.arena->index, index.arena->arenastats.used);
 	index.arena = &arenas[index.arena->index+1];
 	if(index.arena->block != 0 || index.arena->offset != 0 || index.arena->blockremain != index.arena->blocksize || index.arena->buf != nil)
 		sysfatal("TODO handle writing to venti which previously experienced nonlinear writes from other software?");
diff -ru /n/pi4/usr/glenda/neoventi/neoventi.c /usr/glenda/neoventi/neoventi.c
--- /n/pi4/usr/glenda/neoventi/neoventi.c	Tue Dec  9 13:45:28 2025
+++ /usr/glenda/neoventi/neoventi.c	Tue Dec  9 18:28:07 2025
@@ -5,7 +5,7 @@
 #include "neoventi.h"
 
 int mainstacksize = 128*1024;
-char *configpath = "/dev/kaladin/arenas";
+char *configpath = "/dev/sdC1/arenas";
 
 char *arenapath;
 char *isectpath;
diff -ru /n/pi4/usr/glenda/neoventi/server.c /usr/glenda/neoventi/server.c
--- /n/pi4/usr/glenda/neoventi/server.c	Tue Dec  9 13:45:28 2025
+++ /usr/glenda/neoventi/server.c	Tue Dec  9 13:47:49 2025
@@ -216,7 +216,7 @@
 	fd = announce(addr, adir);
 	if(fd < 0)
 		sysfatal("%r");
-	procsetname("neoventi/server");
+	threadsetname("neoventi/server");
 	for(ctl = listen(adir, dir); ctl >= 0; ctl = listen(adir, dir)){
 		handle(ctl, dir);
 		close(ctl);

Re: [9fans] neoventi: call for initial testing

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 3432 bytes --]

- displacement = ((usize)buckets & 0b111111);
+ displacement = ((usize)buckets & (~(~0 << 6)) ); //0b111111 );

you could instead just use the 0x equivalent to the 0b. it's funny that i
was wondering about adding 0b but apparently 9front already has done.


On Thu, 11 Dec 2025 at 23:02, <noam@pixelhero.dev> wrote:

> Quoth hahahahacker2009@gmail.com:
>
> > Warnings
>
> Yeah TBH a lot of those are just "I added something for debugging, it
> worked with warnings, didn't bother fixing it, hadn't removed the
> debug print yet".
>
> > Patch...
> - fprint(2, "entry found in bucket: arena %d, offset%d\n", aindex,
> addr->offset);
> + fprint(2, "entry found in bucket: arena %lld, offset %lld\n", aindex,
> addr->offset);
>
> Almost definitely going to apply all of these, thanks.
>
> - displacement = ((usize)buckets & 0b111111);
> + displacement = ((usize)buckets & (~(~0 << 6)) ); //0b111111 );
>
> I'm...  probably not going to merge patches that make the code more
> obtuse in order to be compatible with the older compiler.
>
> Actually, yes, I will - because neoventi is intended be merged into
> 9front as a replacement for venti outright Soon™, so I can use the
> external repo as the more-compatible version?
>
> + return -1; /* shut up the compiler */
>
> > 4th edition's cc doesn't allow functions without return at the end.
>
> Nor does 9front's!  If you remove the "abort()" call from the end of
> that function, kencc would complain on 9front as well - but abort
> cannot return, and is marked as such, so the compiler knows that there
> is no unreachable end to the function ;)
>
> I'll probably apply this fix to the non-9front version as well.
> Current intention is to get neoventi good enough to replace venti as
> the default implementation in 9front, then patch up the external repo
> for non9front _afterwards_. I might just add a 4e branch to the repo
> for now, and then make that the default later?
>
> -char *configpath = "/dev/kaladin/arenas";
> +char *configpath = "/dev/sdC1/arenas";
>
> There's a `-c` option, as with venti/venti; I'll probably change the
> default eventually, but not sure what to. Maybe just don't have one.
>
> > procsetname > procsetname(2):
>
>      DESCRIPTION
>           Procsetname overrides the current process arguments by writ-
>           ing to its /proc/pid/args file.  The process arguments are
>           informational only.
>
> procsetname sets process info that can be viewed easily (looking at a
> PID, which neoventi proc is it?); threadsetname is internal to
> libthread and AFAIK cannot be accessed from the outside.  I don't see
> much point in keeping it at that point.
>
> > archWalk: ...  unimplemented
>
> Interesting, there's likely a few places where I'm hanging up the
> connection on error / unknown op where I could be warning and moving
> on since I hadn't tested with a continuous write-mode connection, only
> vacfs.  Thanks, will look more :)
>
> > Perhaps make neoventi fork to background when it is ready to use
> with fossil?
> 
> Probably!  Haven't gotten there yet.
> 
> Thanks for taking a look, I'll apply the fixes soon :)
> 

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te748e03814ed814b-M53094346d95f81e753c012f3
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 5567 bytes --]

Re: [9fans] neoventi: call for initial testing

[Jacob Moody]

On 12/11/25 17:09, Charles Forsyth wrote:
> - displacement = ((usize)buckets & 0b111111);
> + displacement = ((usize)buckets & (~(~0 << 6)) ); //0b111111 );
> 
> you could instead just use the 0x equivalent to the 0b. it's funny that i was wondering about adding 0b but apparently 9front already has done.
> 
> 

I had added it when updating the power64 disassembler for POWER9. I didn't want to do the hex <-> binary conversions in my head while also dealing with
IBM's weird backwards notation.

If you're curious in it, I think the patch[0] should be mostly isolated. Although we did a slight refactor along with it.

[0] https://github.com/9front/9front/commit/3384b1e35a288f2e197d9ab75942cb5aa74fb5fc.diff

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te748e03814ed814b-M8a0c47c9821a2d81c0a5dfed
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] neoventi: call for initial testing

[noam]

Quoth hahahahacker2009@gmail.com:

> Warnings

Yeah TBH a lot of those are just "I added something for debugging, it
worked with warnings, didn't bother fixing it, hadn't removed the
debug print yet".

> Patch... 
- fprint(2, "entry found in bucket: arena %d, offset%d\n", aindex, addr->offset);
+ fprint(2, "entry found in bucket: arena %lld, offset %lld\n", aindex, addr->offset);

Almost definitely going to apply all of these, thanks.

- displacement = ((usize)buckets & 0b111111);
+ displacement = ((usize)buckets & (~(~0 << 6)) ); //0b111111 );

I'm...  probably not going to merge patches that make the code more
obtuse in order to be compatible with the older compiler.

Actually, yes, I will - because neoventi is intended be merged into
9front as a replacement for venti outright Soon™, so I can use the
external repo as the more-compatible version?

+ return -1; /* shut up the compiler */

> 4th edition's cc doesn't allow functions without return at the end.

Nor does 9front's!  If you remove the "abort()" call from the end of
that function, kencc would complain on 9front as well - but abort
cannot return, and is marked as such, so the compiler knows that there
is no unreachable end to the function ;)

I'll probably apply this fix to the non-9front version as well.
Current intention is to get neoventi good enough to replace venti as
the default implementation in 9front, then patch up the external repo
for non9front _afterwards_. I might just add a 4e branch to the repo
for now, and then make that the default later?

-char *configpath = "/dev/kaladin/arenas";
+char *configpath = "/dev/sdC1/arenas";

There's a `-c` option, as with venti/venti; I'll probably change the
default eventually, but not sure what to. Maybe just don't have one.

> procsetname > procsetname(2):

     DESCRIPTION
          Procsetname overrides the current process arguments by writ-
          ing to its /proc/pid/args file.  The process arguments are
          informational only.

procsetname sets process info that can be viewed easily (looking at a
PID, which neoventi proc is it?); threadsetname is internal to
libthread and AFAIK cannot be accessed from the outside.  I don't see
much point in keeping it at that point.

> archWalk: ...  unimplemented

Interesting, there's likely a few places where I'm hanging up the
connection on error / unknown op where I could be warning and moving
on since I hadn't tested with a continuous write-mode connection, only
vacfs.  Thanks, will look more :)

> Perhaps make neoventi fork to background when it is ready to use
with fossil?

Probably!  Haven't gotten there yet.

Thanks for taking a look, I'll apply the fixes soon :)


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te748e03814ed814b-M3891ced3b60cd010d1731427
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] neoventi: call for initial testing

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 1157 bytes --]

It's tiny (the biggest file is a notebook) and tidy, what's not to like?

On Mon, 8 Dec 2025 at 19:34, <noam@pixelhero.dev> wrote:

> My reimplementation of venti[1], presented previously at IWP9, has
> gone from "this will DEFINITELY eat your data in the first five
> seconds" to "this will probably eat your data in the first five
> minutes."
> 
> Any brave souls interested in testing it are invited to do so.  I'll
> definitely look at any stack traces or bugs or corruption you find,
> but I can't promise to be able to recover data.  It'll _probably_ be
> recoverable.  Probably.
> 
> Use at your own risk, yadda yadda, I'm only starting testing it with
> fossil presently, but all the obvious tests I could think of offhand
> are passing.
> 
> That means that all the remaining bugs are _subtle_, which is _more_
> dangerous.
> 
> - Noam Preil
> 
> [1] https://git.sr.ht/~pixelherodev/neoventi
> 

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Mfc18dbe9438846181a408dbc
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 2605 bytes --]

Re: [9fans] neoventi: call for initial testing

[wb.kloke]

[-- Attachment #1: Type: text/plain, Size: 586 bytes --]

I had a look into the neoventi code, and noticed some difference in the index handling vs. the original implementation.

So, I have some questions.

Am I right, that Noam's code uses linear search inside the buckets, where the original did binary search?

This is not necessarily bad, but will make the index files incompatible after writing to them.

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Md2e40caf66727601fd69b3b8
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 1189 bytes --]

Re: [9fans] neoventi: call for initial testing

[noam]

Quoth Charles Forsyth <charles.forsyth@gmail.com>:
> It's tiny (the biggest file is a notebook) and tidy, what's not to like?

Hm, where to start?  Oh, maybe at startup.

- The code for recovering arenas in inconsistent state is disabled for
being buggy.

- There's no compression support.  It can decompress whacked blocks
that were written with old venti, but will never write new ones.

- It still uses SHA1 for hashes.

- The disk format is still really bad.

- The protocol needs work too.

- It hasn't been tested with multiple connections, and there's been
enough bugs in the server handling code that there probably are still
some now.

- No formatter.  Still depends on old venti for that.

- Has not been tested nearly well enough to trust it with data.

- The cache implementation is entirely bespoke.  There may be dragons.
Also, I fully intend to bump it from the current eight entries per CPU
cache line up to at _least_ 10, which is probably going to be some...
questionable decision making.

- A lot of the code was written when my grasp on how venti works was
more rudimentary.  Probably going to need a lot of cleaning.  Then
again, it's still leagues better than old venti's code.

- Also, I wrote it.  That's probably a con, because if you reach out
about problems, history and self-awareness says i'm liable to respond
with "oh, that sucks, lemme look into that," chuck it into $home/todo,
and then forget about it for six months.

- Noam Preil


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Mfca52b5741020b8c08eec793
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] neoventi: call for initial testing

[noam]

> This is not necessarily bad, but will make the index files
incompatible after writing to them.

That's certainly not intentional, I'll take a look - but, also, why
would the method used for lookups affect compatibility of writes?  It
should still be writing to the...  ohhhhh wait, is venti keeping the
buckets _sorted_?  On disk?  But then you'd have to read and sort and
writeback multiple times per...  goddammit it's totally doing that
isn't it.

I'll take a look.  Le sigh.  Disregard previous email's claim about me
possibly taking six months to respond, was mostly joking ☺

Thanks :)


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Me8fcb976cc822836d17b5aef
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] neoventi: call for initial testing

[noam]

neoventi is now working with fossil.

Not fully tested, use at your own risk, but it is now successfully
writing snapshots :3

- Noam Preil

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M21c2b92bf72ac85c9ed306c3
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] neoventi: call for initial testing

[noam]

neoventi is now working with fossil.

Not fully tested, use at your own risk, but it is now successfully
writing snapshots :3

- Noam Preil

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Ma1e9baf1831db1bb30a66ba3
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] neoventi: call for initial testing

[wb.kloke]

[-- Attachment #1: Type: text/plain, Size: 1961 bytes --]

On Friday, 9 January 2026, at 2:38 AM, noam wrote:
> Not fully tested, use at your own risk, but it is now successfully
Some compiler warnings on 9front:

...
6c -FTVw disk.c
warning: server.c:54 auto declared and not used: i
warning: server.c:65 label declared and not used "fail"a
6c -FTVw cache.c
6c -FTVw arena.c
warning: arena.c:7 used and not set: tmp 
warning: arena.c:37 unreachable code AS
warning: arena.c:39 format mismatch d UVLONG, arg 3
warning: arena.c:64 format mismatch d UVLONG, arg 3
warning: arena.c:76 format mismatch d UVLONG, arg 3
warning: arena.c:83 format mismatch d UVLONG, arg 3
warning: arena.c:83 format mismatch d UVLONG, arg 4
warning: arena.c:27 set and not used: arena 
warning: arena.c:94 param declared and not used: uncsize
warning: arena.c:94 param declared and not used: score
warning: arena.c:94 param declared and not used: arena
warning: arena.c:93 set and not used: arena 
warning: disk.c:244 format mismatch d UVLONG, arg 3
warning: disk.c:375 auto declared and not used: blocks
6c -FTVw checkarena.c
...
Starrting neoventi, I get
cpu% 6.neoventi -c venti.conf
Initializing neoventi build 5... 6.neoventi: invalid config, bad tokenize on line 11
cpu% 
The venti.conf is:
cpu% cat venti.conf
index main
isect /home/venti/isect0
arenas /home/venti/arenas
arenas /home/venti/arenas1
arenas /home/venti/arenas2
arenas /home/venti/arenas3
arenas /home/venti/arenas4
addr tcp!*!17134
httpaddr tcp!*!8180

When I get around this message, I will retrieve the same data as in my own mventi, and post the times.
There will still be a difference because  I have no isect for the file /home/vent/newarenas . But otherwise the data reside on the same system.
------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Mce5bfe83fd6e5d56e4013a33
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 3285 bytes --]

Re: [9fans] neoventi: call for initial testing

[wb.kloke]

[-- Attachment #1: Type: text/plain, Size: 1263 bytes --]

On Friday, 9 January 2026, at 2:24 PM, wb.kloke wrote:
> When I get around this message, I will retrieve the same data as in my own mventi, and post the times.
> There will still be a difference because  I have no isect for the file /home/vent/newarenas . But otherwise the data reside on the same system.
I got  this fixed. Neoventi does  not like empty lines.

Now, I have a real problem going further. Both arenas and isect are readonly. Neoventi refuses this. I could try to add a writable empty partition, but this would still need a new isect partition, so this is not a real option for me.

For comparison, I checked the time on a conventional venti setup with this venti.conf. The real time is better than I expected.
Maybe the reason is the higher performance of the CPUs involved.
[wb@wbk1 ~/neoventi]$ time venti=cirno:17134 vcat ffs:f10fb5797e5ea50d6bc567772f9794706548f568 > /dev/null
1540096 blocks total
582848 blocks in use, 2122946 file reads

real    11m52,432s
user    1m28,386s
sys     0m48,180s

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M6b0a162bf53f21d28f7867dc
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 2155 bytes --]

Re: [9fans] neoventi: call for initial testing

[Noam Preil]

> warnings

yeah I'm going to look at those once the bugs are iroend out

...or impulsively in five minutes, one of the two

> Initializing neoventi build 5... 6.neoventi: invalid config, bad tokenize on line 11

A few things: it doesn't handle empty lines in the config right now, and
your config would be unsupported; it only supports a single arena
partition and a single index section partition at the moment.

(As an aside, the tcpaddr is ignored completely and will never be used
as venti's http server is a design flaw)

That also looks like a venti that has data on it: be very, very careful
with that, don't hook it up to a fossil, and don't try issuing writes.
Tehre's definitely remaining issues, I'd heavily suggest not trying to
use this to write to anything that you care about just yet.


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M073ba9de7135e16e752ed19f
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] neoventi: call for initial testing

[Noam Preil]

> read only, neoventi refuses this

neoventi will not write unless you issue a VtWrite command to it (e.g.
hook it up to fossil). Beyond that I'm not sure what you're expecting?

> benchmark

You keep posting benchmarks without any context to give them meaning.

11 minutes for 1 GiB of data would be awful, 11 minutes for 100GiB of
data would be pretty okay.


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M4b4c51c652e5e21b64630e76
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] neoventi: call for initial testing

[wb.kloke]

[-- Attachment #1: Type: text/plain, Size: 922 bytes --]

On Friday, 9 January 2026, at 5:08 PM, Noam Preil wrote:
> You keep posting benchmarks without any context to give them meaning.

11 minutes for 1 GiB of data would be awful, 11 minutes for 100GiB of
data would be pretty okay.

I gave the context in another thread. The size of the vcat'ed disk is about 50GB. (Multiply 1540096 with the FreeBSD blocksize of 32k)

In the meantime I changed the open in disk.c to OREAD if ORDWR fails (just like plan9 venti does).
This is, what I  got:

cpu% 6.neoventi -c venti.conf
Initializing neoventi build 5... 6.neoventi 2517: suicide: sys: trap: general protection violation pc=0x208b90

I an not going to test neoventi on writable arenas
------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M9a501e2c6854a6053bc4a9cd
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 1803 bytes --]

Re: [9fans] neoventi: call for initial testing

[Calvin via 9fans]

There are many security and privacy holes in venti's design that should be fixed. Anyone with access to a venti system has, in essence, full permissions to all data in every file system stored in the system. The only current practical mitigation is to simply prevent access to the venti in the first place, and only expose a file system on top of it, such as fossil, but this can greatly reduce the benefits of venti's deduplication.

Trying to do security in venti feels like it would be ripe for timing attacks - if we have security+global deduplication

Ex:

User 1 has a block written containing you know, something we want to see if the user has.

User 2 (the bad guy) writes an identical block, and it returns very quickly because no write was performed.

User 2 now knows the block exists on the system prior to them writing it.

I don't see how you can really do security with Venti, it seems like each user does need access to the whole system.

Which I guess is why Fossil did all of the heavy lifting.

Interesting stuff regardless, I am following along for fun. 

Relatedly (not really) is ZFS's deduplication 

https://despairlabs.com/blog/posts/2024-10-27-openzfs-dedup-is-good-dont-use-it/

Thanks
Calvin

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M36ee6e6fa151248a40e2949f
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] neoventi: call for initial testing

[wb.kloke]

[-- Attachment #1: Type: text/plain, Size: 1089 bytes --]

On Friday, 9 January 2026, at 6:14 PM, Calvin wrote:
> There are many security and privacy holes in venti's design that should be fixed. Anyone with access to a venti system has, in essence, full permissions to all data in every file system stored in the system. The only current practical mitigation is to simply prevent access to the venti in the first place, and only expose a file system on top of it, such as fossil, but this can greatly reduce the benefits of venti's deduplication.

Indeed, venti is insecure. But  it doesn't do 9p  and it is only intended as a backing store for real servers. If the real server (eg. a fossil) is on the same machine, address of 127.1 is a solution. Imho also adresses in 168.x.x.x nets are quite safe. This way it is not exposed outside your realm. 

Personally, I would even prefer a protocol similar to aoe.

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M29d8a4c82645a573eab876c3
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 1715 bytes --]

Re: [9fans] neoventi: call for initial testing

[Calvin via 9fans]

[-- Attachment #1: Type: text/plain, Size: 1092 bytes --]

> On Friday, 9 January 2026, at 6:14 PM, Calvin wrote:
>
>> There are many security and privacy holes in venti's design that should be fixed. Anyone with access to a venti system has, in essence, full permissions to all data in every file system stored in the system. The only current practical mitigation is to simply prevent access to the venti in the first place, and only expose a file system on top of it, such as fossil, but this can greatly reduce the benefits of venti's deduplication.
>
> Indeed, venti is insecure. But it doesn't do 9p and it is only intended as a backing store for real servers. If the real server (eg. a fossil) is on the same machine, address of 127.1 is a solution. Imho also adresses in 168.x.x.x nets are quite safe. This way it is not exposed outside your realm.

Sorry i was quoting Noam there, I am interested in proposed solutions.

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M2a5d8457b57f7ac8e8103593
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 1884 bytes --]

Re: [9fans] neoventi: call for initial testing

[Emery]

Regarding security, the ERIS spec describes some of these issues,
but I doubt the ERIS solution is compatible.

https://eris.codeberg.page/spec/#name-security-considerations

There is an ERIS 9P server but it's read-only and stores blocks in
its own database.
https://eris.codeberg.page/erishell.1.html#srv-

Excerpts from Calvin via 9fans's message of January 10, 2026 2:08 am:
>> On Friday, 9 January 2026, at 6:14 PM, Calvin wrote:
>>
>>> There are many security and privacy holes in venti's design that should be fixed. Anyone with access to a venti system has, in essence, full permissions to all data in every file system stored in the system. The only current practical mitigation is to simply prevent access to the venti in the first place, and only expose a file system on top of it, such as fossil, but this can greatly reduce the benefits of venti's deduplication.
>>
>> Indeed, venti is insecure. But it doesn't do 9p and it is only intended as a backing store for real servers. If the real server (eg. a fossil) is on the same machine, address of 127.1 is a solution. Imho also adresses in 168.x.x.x nets are quite safe. This way it is not exposed outside your realm.
> 
> Sorry i was quoting Noam there, I am interested in proposed solutions.

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M8702dae5eeef483f9277a8c1
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] neoventi: call for initial testing

[Noam Preil]

Thanks for that testing - I need to fix that and redesign the handling
of mapping virtual venti addresses to physical ones, it's definitely not
ready for usage just yet.

I really do appreciate the testing, though - now I should be able to fix
that over the weekend :D

- Noam Preil

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M7a5c54618c72c53bf38b2c9b
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription