[9fans] neoventi: call for initial testing

[9fans] neoventi: call for initial testing

[noam]

My reimplementation of venti[1], presented previously at IWP9, has
gone from "this will DEFINITELY eat your data in the first five
seconds" to "this will probably eat your data in the first five
minutes."

Any brave souls interested in testing it are invited to do so.  I'll
definitely look at any stack traces or bugs or corruption you find,
but I can't promise to be able to recover data.  It'll _probably_ be
recoverable.  Probably.

Use at your own risk, yadda yadda, I'm only starting testing it with
fossil presently, but all the obvious tests I could think of offhand
are passing.

That means that all the remaining bugs are _subtle_, which is _more_
dangerous.

- Noam Preil

[1] https://git.sr.ht/~pixelherodev/neoventi




------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Mdc787dc2648256807ea74cbe
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] neoventi: call for initial testing

[hahahahacker2009]

[-- Attachment #1: Type: text/plain, Size: 741 bytes --]

I made this change to compile on 4th edition, with Geoff's 9k.
I don't know what is procsetname and how it is different from
threadsetname.  Some format mismatches (%d for ulong and uintptr are
fixed), and there are still many warnings.  4th edition's cc doesn't
allow functions without return at the end.

vac and unvac works, and I tried to set up a fossil and snap -a, but
it constantly says archWalk: ...  unimplemented.  Perhaps make
neoventi fork to background when it is ready to use with fossil?

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te748e03814ed814b-M76bb39d91f848a59782bb890
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: message/rfc822, Size: 14048 bytes --]

From: noam@pixelhero.dev
To: 9fans@9fans.net
Subject: [9fans] neoventi: call for initial testing
Date: Mon, 08 Dec 2025 17:30:27 -0500
Message-ID: <15FCDAB3D76AF86D4FE56DF4E4BFA5E2@pixelhero.dev>

My reimplementation of venti[1], presented previously at IWP9, has
gone from "this will DEFINITELY eat your data in the first five
seconds" to "this will probably eat your data in the first five
minutes."

Any brave souls interested in testing it are invited to do so.  I'll
definitely look at any stack traces or bugs or corruption you find,
but I can't promise to be able to recover data.  It'll _probably_ be
recoverable.  Probably.

Use at your own risk, yadda yadda, I'm only starting testing it with
fossil presently, but all the obvious tests I could think of offhand
are passing.

That means that all the remaining bugs are _subtle_, which is _more_
dangerous.

- Noam Preil

[1] https://git.sr.ht/~pixelherodev/neoventi




------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Mdc787dc2648256807ea74cbe
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #3: neoventi.4e.diff --]
[-- Type: text/plain, Size: 4736 bytes --]

diff -ru /n/pi4/usr/glenda/neoventi/arena.c /usr/glenda/neoventi/arena.c
--- /n/pi4/usr/glenda/neoventi/arena.c	Tue Dec  9 13:45:28 2025
+++ /usr/glenda/neoventi/arena.c	Tue Dec  9 18:22:55 2025
@@ -36,7 +36,7 @@
 	int good;
 	n = arena->arenastats.clumps;
 	corruptchain = 0;
-	fprint(2, "clumps: %d, %d\n", n, arena->indexstats.clumps);
+	fprint(2, "clumps: %ld, %d\n", n, arena->indexstats.clumps);
 	if(n == arena->indexstats.clumps)
 		return 1;
 	buf = nil;
@@ -61,7 +61,7 @@
 		good = arenarepair(arena, (u8int*)ci, addr);
 		addr += (((u64int)(U16GET(ci + 1))) + 38);
 		if(!good){
-			fprint(2, "clump %d is broken, at address %llud: %r\n", m, addr);
+			fprint(2, "clump %ld is broken, at address %lld: %r\n", m, addr);
 			corruptchain += 1;
 			// Skip. old venti creates corruption in some easy-to-repro cases
 			// (e.g. write, kill venti within a few seconds) and I don't need people
@@ -73,14 +73,14 @@
 	}
 	if(addr != goodaddr){
 		assert(corruptchain > 0);
-		fprint(2, "corruption detected: the last %d clumps are corrupt. Dropping them.\n"
+		fprint(2, "corruption detected: the last %ld clumps are corrupt. Dropping them.\n"
 				  "\taddr dropped from %llud to %llud\n",
 		corruptchain, addr, goodaddr);
 		addr = goodaddr;
 		arena->arenastats.clumps -= corruptchain;
 	}
 	if(arena->arenastats.used != addr){
-		fprint(2, "corrupt: found addr %d, expected %d", addr, arena->arenastats.used);
+		fprint(2, "corrupt: found addr %lld, expected %lld", addr, arena->arenastats.used);
 		arena->arenastats.used = addr;
 	}
 	// TODO: we haven't updated indexstats->clumps, so this is safe, but we _do_
diff -ru /n/pi4/usr/glenda/neoventi/cache.c /usr/glenda/neoventi/cache.c
--- /n/pi4/usr/glenda/neoventi/cache.c	Tue Dec  9 13:45:28 2025
+++ /usr/glenda/neoventi/cache.c	Tue Dec  9 18:13:59 2025
@@ -62,13 +62,13 @@
 	if(data == (void*)-1)
 		sysfatal("failed to allocate cache space of %llux", (usize)ENTRYSIZE*ENTRIES);
 	buckets = sbrk(64 + 64*BUCKETS);
-	displacement = ((usize)buckets & 0b111111);
+	displacement = ((usize)buckets & (~(~0 << 6)) ); //0b111111 );
 	if(displacement != 0){
 		displacement = 64 - displacement;
 		buckets = (bucket_t*)((usize)buckets + displacement);
 	}
 	// alignment
-	assert(((usize)buckets&0b111111) == 0);
+	assert(((usize)buckets& (~(~0 << 6)) ) == 0);
 	for(int i = 0; i < BUCKETS; i += 1)
 		memset(buckets[i].index, 0xFF, 32);
 	bucketend = buckets + BUCKETS;
@@ -211,4 +211,5 @@
 	fprint(2, "impossible\n");
 	// DO NOT PROCEED if the cache might be corrupt!
 	abort();
+	return -1; /* shut up the compiler */
 }
diff -ru /n/pi4/usr/glenda/neoventi/disk.c /usr/glenda/neoventi/disk.c
--- /n/pi4/usr/glenda/neoventi/disk.c	Tue Dec  9 13:45:28 2025
+++ /usr/glenda/neoventi/disk.c	Tue Dec  9 18:13:20 2025
@@ -63,6 +63,7 @@
 	}
 	werrstr("internal corruption: arena not found for arenaindex");
 	abort();
+	return -1; /* shut up the compiler */
 }
 
 int
@@ -154,7 +155,7 @@
 	aindex = aindexfromaddr(addr->offset);
 	addr->s_arena = &arenas[aindex];
 	addr->offset -= (addr->s_arena->base - addr->s_arena->blocksize);
-	fprint(2, "entry found in bucket: arena %d, offset %d\n", aindex, addr->offset);
+	fprint(2, "entry found in bucket: arena %lld, offset %lld\n", aindex, addr->offset);
 	return 1;
 }
 
@@ -243,7 +244,7 @@
 {
 	index.arena->arenastats.sealed = 1;
 	if(index.arena->index+1 == numarenas)
-		sysfatal("TODO last arena, %d, full, at size %d!", index.arena->index, index.arena->arenastats.used);
+		sysfatal("TODO last arena, %d, full, at size %lld!", index.arena->index, index.arena->arenastats.used);
 	index.arena = &arenas[index.arena->index+1];
 	if(index.arena->block != 0 || index.arena->offset != 0 || index.arena->blockremain != index.arena->blocksize || index.arena->buf != nil)
 		sysfatal("TODO handle writing to venti which previously experienced nonlinear writes from other software?");
diff -ru /n/pi4/usr/glenda/neoventi/neoventi.c /usr/glenda/neoventi/neoventi.c
--- /n/pi4/usr/glenda/neoventi/neoventi.c	Tue Dec  9 13:45:28 2025
+++ /usr/glenda/neoventi/neoventi.c	Tue Dec  9 18:28:07 2025
@@ -5,7 +5,7 @@
 #include "neoventi.h"
 
 int mainstacksize = 128*1024;
-char *configpath = "/dev/kaladin/arenas";
+char *configpath = "/dev/sdC1/arenas";
 
 char *arenapath;
 char *isectpath;
diff -ru /n/pi4/usr/glenda/neoventi/server.c /usr/glenda/neoventi/server.c
--- /n/pi4/usr/glenda/neoventi/server.c	Tue Dec  9 13:45:28 2025
+++ /usr/glenda/neoventi/server.c	Tue Dec  9 13:47:49 2025
@@ -216,7 +216,7 @@
 	fd = announce(addr, adir);
 	if(fd < 0)
 		sysfatal("%r");
-	procsetname("neoventi/server");
+	threadsetname("neoventi/server");
 	for(ctl = listen(adir, dir); ctl >= 0; ctl = listen(adir, dir)){
 		handle(ctl, dir);
 		close(ctl);

Re: [9fans] neoventi: call for initial testing

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 3432 bytes --]

- displacement = ((usize)buckets & 0b111111);
+ displacement = ((usize)buckets & (~(~0 << 6)) ); //0b111111 );

you could instead just use the 0x equivalent to the 0b. it's funny that i
was wondering about adding 0b but apparently 9front already has done.


On Thu, 11 Dec 2025 at 23:02, <noam@pixelhero.dev> wrote:

> Quoth hahahahacker2009@gmail.com:
>
> > Warnings
>
> Yeah TBH a lot of those are just "I added something for debugging, it
> worked with warnings, didn't bother fixing it, hadn't removed the
> debug print yet".
>
> > Patch...
> - fprint(2, "entry found in bucket: arena %d, offset%d\n", aindex,
> addr->offset);
> + fprint(2, "entry found in bucket: arena %lld, offset %lld\n", aindex,
> addr->offset);
>
> Almost definitely going to apply all of these, thanks.
>
> - displacement = ((usize)buckets & 0b111111);
> + displacement = ((usize)buckets & (~(~0 << 6)) ); //0b111111 );
>
> I'm...  probably not going to merge patches that make the code more
> obtuse in order to be compatible with the older compiler.
>
> Actually, yes, I will - because neoventi is intended be merged into
> 9front as a replacement for venti outright Soon™, so I can use the
> external repo as the more-compatible version?
>
> + return -1; /* shut up the compiler */
>
> > 4th edition's cc doesn't allow functions without return at the end.
>
> Nor does 9front's!  If you remove the "abort()" call from the end of
> that function, kencc would complain on 9front as well - but abort
> cannot return, and is marked as such, so the compiler knows that there
> is no unreachable end to the function ;)
>
> I'll probably apply this fix to the non-9front version as well.
> Current intention is to get neoventi good enough to replace venti as
> the default implementation in 9front, then patch up the external repo
> for non9front _afterwards_. I might just add a 4e branch to the repo
> for now, and then make that the default later?
>
> -char *configpath = "/dev/kaladin/arenas";
> +char *configpath = "/dev/sdC1/arenas";
>
> There's a `-c` option, as with venti/venti; I'll probably change the
> default eventually, but not sure what to. Maybe just don't have one.
>
> > procsetname > procsetname(2):
>
>      DESCRIPTION
>           Procsetname overrides the current process arguments by writ-
>           ing to its /proc/pid/args file.  The process arguments are
>           informational only.
>
> procsetname sets process info that can be viewed easily (looking at a
> PID, which neoventi proc is it?); threadsetname is internal to
> libthread and AFAIK cannot be accessed from the outside.  I don't see
> much point in keeping it at that point.
>
> > archWalk: ...  unimplemented
>
> Interesting, there's likely a few places where I'm hanging up the
> connection on error / unknown op where I could be warning and moving
> on since I hadn't tested with a continuous write-mode connection, only
> vacfs.  Thanks, will look more :)
>
> > Perhaps make neoventi fork to background when it is ready to use
> with fossil?
> 
> Probably!  Haven't gotten there yet.
> 
> Thanks for taking a look, I'll apply the fixes soon :)
> 

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te748e03814ed814b-M53094346d95f81e753c012f3
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 5567 bytes --]

Re: [9fans] neoventi: call for initial testing

[Jacob Moody]

On 12/11/25 17:09, Charles Forsyth wrote:
> - displacement = ((usize)buckets & 0b111111);
> + displacement = ((usize)buckets & (~(~0 << 6)) ); //0b111111 );
> 
> you could instead just use the 0x equivalent to the 0b. it's funny that i was wondering about adding 0b but apparently 9front already has done.
> 
> 

I had added it when updating the power64 disassembler for POWER9. I didn't want to do the hex <-> binary conversions in my head while also dealing with
IBM's weird backwards notation.

If you're curious in it, I think the patch[0] should be mostly isolated. Although we did a slight refactor along with it.

[0] https://github.com/9front/9front/commit/3384b1e35a288f2e197d9ab75942cb5aa74fb5fc.diff

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te748e03814ed814b-M8a0c47c9821a2d81c0a5dfed
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] neoventi: call for initial testing

[noam]

Quoth hahahahacker2009@gmail.com:

> Warnings

Yeah TBH a lot of those are just "I added something for debugging, it
worked with warnings, didn't bother fixing it, hadn't removed the
debug print yet".

> Patch... 
- fprint(2, "entry found in bucket: arena %d, offset%d\n", aindex, addr->offset);
+ fprint(2, "entry found in bucket: arena %lld, offset %lld\n", aindex, addr->offset);

Almost definitely going to apply all of these, thanks.

- displacement = ((usize)buckets & 0b111111);
+ displacement = ((usize)buckets & (~(~0 << 6)) ); //0b111111 );

I'm...  probably not going to merge patches that make the code more
obtuse in order to be compatible with the older compiler.

Actually, yes, I will - because neoventi is intended be merged into
9front as a replacement for venti outright Soon™, so I can use the
external repo as the more-compatible version?

+ return -1; /* shut up the compiler */

> 4th edition's cc doesn't allow functions without return at the end.

Nor does 9front's!  If you remove the "abort()" call from the end of
that function, kencc would complain on 9front as well - but abort
cannot return, and is marked as such, so the compiler knows that there
is no unreachable end to the function ;)

I'll probably apply this fix to the non-9front version as well.
Current intention is to get neoventi good enough to replace venti as
the default implementation in 9front, then patch up the external repo
for non9front _afterwards_. I might just add a 4e branch to the repo
for now, and then make that the default later?

-char *configpath = "/dev/kaladin/arenas";
+char *configpath = "/dev/sdC1/arenas";

There's a `-c` option, as with venti/venti; I'll probably change the
default eventually, but not sure what to. Maybe just don't have one.

> procsetname > procsetname(2):

     DESCRIPTION
          Procsetname overrides the current process arguments by writ-
          ing to its /proc/pid/args file.  The process arguments are
          informational only.

procsetname sets process info that can be viewed easily (looking at a
PID, which neoventi proc is it?); threadsetname is internal to
libthread and AFAIK cannot be accessed from the outside.  I don't see
much point in keeping it at that point.

> archWalk: ...  unimplemented

Interesting, there's likely a few places where I'm hanging up the
connection on error / unknown op where I could be warning and moving
on since I hadn't tested with a continuous write-mode connection, only
vacfs.  Thanks, will look more :)

> Perhaps make neoventi fork to background when it is ready to use
with fossil?

Probably!  Haven't gotten there yet.

Thanks for taking a look, I'll apply the fixes soon :)


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/Te748e03814ed814b-M3891ced3b60cd010d1731427
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] neoventi: call for initial testing

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 1157 bytes --]

It's tiny (the biggest file is a notebook) and tidy, what's not to like?

On Mon, 8 Dec 2025 at 19:34, <noam@pixelhero.dev> wrote:

> My reimplementation of venti[1], presented previously at IWP9, has
> gone from "this will DEFINITELY eat your data in the first five
> seconds" to "this will probably eat your data in the first five
> minutes."
> 
> Any brave souls interested in testing it are invited to do so.  I'll
> definitely look at any stack traces or bugs or corruption you find,
> but I can't promise to be able to recover data.  It'll _probably_ be
> recoverable.  Probably.
> 
> Use at your own risk, yadda yadda, I'm only starting testing it with
> fossil presently, but all the obvious tests I could think of offhand
> are passing.
> 
> That means that all the remaining bugs are _subtle_, which is _more_
> dangerous.
> 
> - Noam Preil
> 
> [1] https://git.sr.ht/~pixelherodev/neoventi
> 

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Mfc18dbe9438846181a408dbc
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 2605 bytes --]

Re: [9fans] neoventi: call for initial testing

[wb.kloke]

[-- Attachment #1: Type: text/plain, Size: 586 bytes --]

I had a look into the neoventi code, and noticed some difference in the index handling vs. the original implementation.

So, I have some questions.

Am I right, that Noam's code uses linear search inside the buckets, where the original did binary search?

This is not necessarily bad, but will make the index files incompatible after writing to them.

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Md2e40caf66727601fd69b3b8
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 1189 bytes --]

Re: [9fans] neoventi: call for initial testing

[noam]

Quoth Charles Forsyth <charles.forsyth@gmail.com>:
> It's tiny (the biggest file is a notebook) and tidy, what's not to like?

Hm, where to start?  Oh, maybe at startup.

- The code for recovering arenas in inconsistent state is disabled for
being buggy.

- There's no compression support.  It can decompress whacked blocks
that were written with old venti, but will never write new ones.

- It still uses SHA1 for hashes.

- The disk format is still really bad.

- The protocol needs work too.

- It hasn't been tested with multiple connections, and there's been
enough bugs in the server handling code that there probably are still
some now.

- No formatter.  Still depends on old venti for that.

- Has not been tested nearly well enough to trust it with data.

- The cache implementation is entirely bespoke.  There may be dragons.
Also, I fully intend to bump it from the current eight entries per CPU
cache line up to at _least_ 10, which is probably going to be some...
questionable decision making.

- A lot of the code was written when my grasp on how venti works was
more rudimentary.  Probably going to need a lot of cleaning.  Then
again, it's still leagues better than old venti's code.

- Also, I wrote it.  That's probably a con, because if you reach out
about problems, history and self-awareness says i'm liable to respond
with "oh, that sucks, lemme look into that," chuck it into $home/todo,
and then forget about it for six months.

- Noam Preil


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Mfca52b5741020b8c08eec793
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] neoventi: call for initial testing

[noam]

> This is not necessarily bad, but will make the index files
incompatible after writing to them.

That's certainly not intentional, I'll take a look - but, also, why
would the method used for lookups affect compatibility of writes?  It
should still be writing to the...  ohhhhh wait, is venti keeping the
buckets _sorted_?  On disk?  But then you'd have to read and sort and
writeback multiple times per...  goddammit it's totally doing that
isn't it.

I'll take a look.  Le sigh.  Disregard previous email's claim about me
possibly taking six months to respond, was mostly joking ☺

Thanks :)


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Me8fcb976cc822836d17b5aef
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] neoventi: call for initial testing

[noam]

neoventi is now working with fossil.

Not fully tested, use at your own risk, but it is now successfully
writing snapshots :3

- Noam Preil

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M21c2b92bf72ac85c9ed306c3
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] neoventi: call for initial testing

[noam]

neoventi is now working with fossil.

Not fully tested, use at your own risk, but it is now successfully
writing snapshots :3

- Noam Preil

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Ma1e9baf1831db1bb30a66ba3
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] neoventi: call for initial testing

[wb.kloke]

[-- Attachment #1: Type: text/plain, Size: 1961 bytes --]

On Friday, 9 January 2026, at 2:38 AM, noam wrote:
> Not fully tested, use at your own risk, but it is now successfully
Some compiler warnings on 9front:

...
6c -FTVw disk.c
warning: server.c:54 auto declared and not used: i
warning: server.c:65 label declared and not used "fail"a
6c -FTVw cache.c
6c -FTVw arena.c
warning: arena.c:7 used and not set: tmp 
warning: arena.c:37 unreachable code AS
warning: arena.c:39 format mismatch d UVLONG, arg 3
warning: arena.c:64 format mismatch d UVLONG, arg 3
warning: arena.c:76 format mismatch d UVLONG, arg 3
warning: arena.c:83 format mismatch d UVLONG, arg 3
warning: arena.c:83 format mismatch d UVLONG, arg 4
warning: arena.c:27 set and not used: arena 
warning: arena.c:94 param declared and not used: uncsize
warning: arena.c:94 param declared and not used: score
warning: arena.c:94 param declared and not used: arena
warning: arena.c:93 set and not used: arena 
warning: disk.c:244 format mismatch d UVLONG, arg 3
warning: disk.c:375 auto declared and not used: blocks
6c -FTVw checkarena.c
...
Starrting neoventi, I get
cpu% 6.neoventi -c venti.conf
Initializing neoventi build 5... 6.neoventi: invalid config, bad tokenize on line 11
cpu% 
The venti.conf is:
cpu% cat venti.conf
index main
isect /home/venti/isect0
arenas /home/venti/arenas
arenas /home/venti/arenas1
arenas /home/venti/arenas2
arenas /home/venti/arenas3
arenas /home/venti/arenas4
addr tcp!*!17134
httpaddr tcp!*!8180

When I get around this message, I will retrieve the same data as in my own mventi, and post the times.
There will still be a difference because  I have no isect for the file /home/vent/newarenas . But otherwise the data reside on the same system.
------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-Mce5bfe83fd6e5d56e4013a33
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 3285 bytes --]

Re: [9fans] neoventi: call for initial testing

[wb.kloke]

[-- Attachment #1: Type: text/plain, Size: 1263 bytes --]

On Friday, 9 January 2026, at 2:24 PM, wb.kloke wrote:
> When I get around this message, I will retrieve the same data as in my own mventi, and post the times.
> There will still be a difference because  I have no isect for the file /home/vent/newarenas . But otherwise the data reside on the same system.
I got  this fixed. Neoventi does  not like empty lines.

Now, I have a real problem going further. Both arenas and isect are readonly. Neoventi refuses this. I could try to add a writable empty partition, but this would still need a new isect partition, so this is not a real option for me.

For comparison, I checked the time on a conventional venti setup with this venti.conf. The real time is better than I expected.
Maybe the reason is the higher performance of the CPUs involved.
[wb@wbk1 ~/neoventi]$ time venti=cirno:17134 vcat ffs:f10fb5797e5ea50d6bc567772f9794706548f568 > /dev/null
1540096 blocks total
582848 blocks in use, 2122946 file reads

real    11m52,432s
user    1m28,386s
sys     0m48,180s

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M6b0a162bf53f21d28f7867dc
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 2155 bytes --]

Re: [9fans] neoventi: call for initial testing

[Noam Preil]

> warnings

yeah I'm going to look at those once the bugs are iroend out

...or impulsively in five minutes, one of the two

> Initializing neoventi build 5... 6.neoventi: invalid config, bad tokenize on line 11

A few things: it doesn't handle empty lines in the config right now, and
your config would be unsupported; it only supports a single arena
partition and a single index section partition at the moment.

(As an aside, the tcpaddr is ignored completely and will never be used
as venti's http server is a design flaw)

That also looks like a venti that has data on it: be very, very careful
with that, don't hook it up to a fossil, and don't try issuing writes.
Tehre's definitely remaining issues, I'd heavily suggest not trying to
use this to write to anything that you care about just yet.


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M073ba9de7135e16e752ed19f
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] neoventi: call for initial testing

[Noam Preil]

> read only, neoventi refuses this

neoventi will not write unless you issue a VtWrite command to it (e.g.
hook it up to fossil). Beyond that I'm not sure what you're expecting?

> benchmark

You keep posting benchmarks without any context to give them meaning.

11 minutes for 1 GiB of data would be awful, 11 minutes for 100GiB of
data would be pretty okay.


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M4b4c51c652e5e21b64630e76
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] neoventi: call for initial testing

[wb.kloke]

[-- Attachment #1: Type: text/plain, Size: 922 bytes --]

On Friday, 9 January 2026, at 5:08 PM, Noam Preil wrote:
> You keep posting benchmarks without any context to give them meaning.

11 minutes for 1 GiB of data would be awful, 11 minutes for 100GiB of
data would be pretty okay.

I gave the context in another thread. The size of the vcat'ed disk is about 50GB. (Multiply 1540096 with the FreeBSD blocksize of 32k)

In the meantime I changed the open in disk.c to OREAD if ORDWR fails (just like plan9 venti does).
This is, what I  got:

cpu% 6.neoventi -c venti.conf
Initializing neoventi build 5... 6.neoventi 2517: suicide: sys: trap: general protection violation pc=0x208b90

I an not going to test neoventi on writable arenas
------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M9a501e2c6854a6053bc4a9cd
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 1803 bytes --]

Re: [9fans] neoventi: call for initial testing

[Calvin via 9fans]

There are many security and privacy holes in venti's design that should be fixed. Anyone with access to a venti system has, in essence, full permissions to all data in every file system stored in the system. The only current practical mitigation is to simply prevent access to the venti in the first place, and only expose a file system on top of it, such as fossil, but this can greatly reduce the benefits of venti's deduplication.

Trying to do security in venti feels like it would be ripe for timing attacks - if we have security+global deduplication

Ex:

User 1 has a block written containing you know, something we want to see if the user has.

User 2 (the bad guy) writes an identical block, and it returns very quickly because no write was performed.

User 2 now knows the block exists on the system prior to them writing it.

I don't see how you can really do security with Venti, it seems like each user does need access to the whole system.

Which I guess is why Fossil did all of the heavy lifting.

Interesting stuff regardless, I am following along for fun. 

Relatedly (not really) is ZFS's deduplication 

https://despairlabs.com/blog/posts/2024-10-27-openzfs-dedup-is-good-dont-use-it/

Thanks
Calvin

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M36ee6e6fa151248a40e2949f
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] neoventi: call for initial testing

[wb.kloke]

[-- Attachment #1: Type: text/plain, Size: 1089 bytes --]

On Friday, 9 January 2026, at 6:14 PM, Calvin wrote:
> There are many security and privacy holes in venti's design that should be fixed. Anyone with access to a venti system has, in essence, full permissions to all data in every file system stored in the system. The only current practical mitigation is to simply prevent access to the venti in the first place, and only expose a file system on top of it, such as fossil, but this can greatly reduce the benefits of venti's deduplication.

Indeed, venti is insecure. But  it doesn't do 9p  and it is only intended as a backing store for real servers. If the real server (eg. a fossil) is on the same machine, address of 127.1 is a solution. Imho also adresses in 168.x.x.x nets are quite safe. This way it is not exposed outside your realm. 

Personally, I would even prefer a protocol similar to aoe.

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M29d8a4c82645a573eab876c3
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 1715 bytes --]

Re: [9fans] neoventi: call for initial testing

[Calvin via 9fans]

[-- Attachment #1: Type: text/plain, Size: 1092 bytes --]

> On Friday, 9 January 2026, at 6:14 PM, Calvin wrote:
>
>> There are many security and privacy holes in venti's design that should be fixed. Anyone with access to a venti system has, in essence, full permissions to all data in every file system stored in the system. The only current practical mitigation is to simply prevent access to the venti in the first place, and only expose a file system on top of it, such as fossil, but this can greatly reduce the benefits of venti's deduplication.
>
> Indeed, venti is insecure. But it doesn't do 9p and it is only intended as a backing store for real servers. If the real server (eg. a fossil) is on the same machine, address of 127.1 is a solution. Imho also adresses in 168.x.x.x nets are quite safe. This way it is not exposed outside your realm.

Sorry i was quoting Noam there, I am interested in proposed solutions.

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M2a5d8457b57f7ac8e8103593
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 1884 bytes --]

Re: [9fans] neoventi: call for initial testing

[Emery]

Regarding security, the ERIS spec describes some of these issues,
but I doubt the ERIS solution is compatible.

https://eris.codeberg.page/spec/#name-security-considerations

There is an ERIS 9P server but it's read-only and stores blocks in
its own database.
https://eris.codeberg.page/erishell.1.html#srv-

Excerpts from Calvin via 9fans's message of January 10, 2026 2:08 am:
>> On Friday, 9 January 2026, at 6:14 PM, Calvin wrote:
>>
>>> There are many security and privacy holes in venti's design that should be fixed. Anyone with access to a venti system has, in essence, full permissions to all data in every file system stored in the system. The only current practical mitigation is to simply prevent access to the venti in the first place, and only expose a file system on top of it, such as fossil, but this can greatly reduce the benefits of venti's deduplication.
>>
>> Indeed, venti is insecure. But it doesn't do 9p and it is only intended as a backing store for real servers. If the real server (eg. a fossil) is on the same machine, address of 127.1 is a solution. Imho also adresses in 168.x.x.x nets are quite safe. This way it is not exposed outside your realm.
> 
> Sorry i was quoting Noam there, I am interested in proposed solutions.

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M8702dae5eeef483f9277a8c1
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] neoventi: call for initial testing

[Noam Preil]

Thanks for that testing - I need to fix that and redesign the handling
of mapping virtual venti addresses to physical ones, it's definitely not
ready for usage just yet.

I really do appreciate the testing, though - now I should be able to fix
that over the weekend :D

- Noam Preil

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M7a5c54618c72c53bf38b2c9b
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] neoventi: call for initial testing

[ori]

Quoth Calvin via 9fans <9fans@9fans.net>:
> > On Friday, 9 January 2026, at 6:14 PM, Calvin wrote:
> >
> >> There are many security and privacy holes in venti's design that should be fixed. Anyone with access to a venti system has, in essence, full permissions to all data in every file system stored in the system. The only current practical mitigation is to simply prevent access to the venti in the first place, and only expose a file system on top of it, such as fossil, but this can greatly reduce the benefits of venti's deduplication.
> >
> > Indeed, venti is insecure. But it doesn't do 9p and it is only intended as a backing store for real servers. If the real server (eg. a fossil) is on the same machine, address of 127.1 is a solution. Imho also adresses in 168.x.x.x nets are quite safe. This way it is not exposed outside your realm.
> 
> Sorry i was quoting Noam there, I am interested in proposed solutions.
> 

What exactly is your threat model?


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T4d8ba4236feb5d92-M118b9d7df500b1487e064382
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription