9front - general discussion about 9front
 help / color / mirror / Atom feed
* [9front] cfs, aan and network recommendations
@ 2021-11-15 22:39 sirjofri
  2021-11-16  4:12 ` Xiao-Yong Jin
  0 siblings, 1 reply; 14+ messages in thread
From: sirjofri @ 2021-11-15 22:39 UTC (permalink / raw)
  To: 9front

Hello all,

(I tried sending this mail earlier, but it seems it never reached the 
mailing list.)

due to a full worm on my cwfs I'm rethinking my network setup.

Currently I have one VPS with limited storage (usually not much, the one 
that's almost full). I also have a larger/more powerful computer at home 
with a standard German internet connection, so the IP address can change 
every day and the connection is not always up.

At the moment both machines are full file servers. The VPS (oat) is also 
mail server, http server, as well as auth and secstore. The other machine 
(leaf) is basically just cpu+fs, I had most of my data there.

Since I moved to another apartment and I had internet connection issues I 
wasn't able to put leaf online, and I also only seem to have IPv6 here 
(from outside), so I was never really able to connect to leaf from 
outside. However, I believe it's possible.

My current thoughts were, I could try to make leaf my only cpu+fs for 
working and configuration etc, and have oat start with leaf as a 
fileserver. Since the connection is likely not reliable and potentially 
slow I thought about combining cfs with aan.

Did anyone try something like that? How reliable is aan? I know it 
handles short interruptions reasonably well (e.g. switching between LTE 
and WiFi), but what if something more happens? How could I handle 
potential rebooting (do I need to reboot oat each time I reboot leaf 
then)?

I was also thinking about just removing leaf from my network and only 
rely on VPS. They are always available and hardware managed, as well as 
cheap enough. I'd then just like to have a good/solid filesystem setup 
(cwfs vs hjfs, or potentially even fossil?). Do you have good 
recommendations?

Ideally I want a dump every day for the last X months, and then free 
older stuff (as in, I care about the data, but not the history). Maybe 
that's a use case for classical backups (daily, weekly, monthly, ...) and 
not so much for a dumping filesystem. Do you have any ideas and 
recommendations?

sirjofri

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [9front] cfs, aan and network recommendations
  2021-11-15 22:39 [9front] cfs, aan and network recommendations sirjofri
@ 2021-11-16  4:12 ` Xiao-Yong Jin
  2021-11-16 18:18   ` Noam Preil
  0 siblings, 1 reply; 14+ messages in thread
From: Xiao-Yong Jin @ 2021-11-16  4:12 UTC (permalink / raw)
  To: 9front


> On Nov 15, 2021, at 4:39 PM, sirjofri <sirjofri+ml-9front@sirjofri.de> wrote:
> 
> Did anyone try something like that? How reliable is aan? I know it handles short interruptions reasonably well (e.g. switching between LTE and WiFi), but what if something more happens? How could I handle potential rebooting (do I need to reboot oat each time I reboot leaf then)?

Aan server and client has to sync up.  If you reboot the server,
you have to reboot the client.  Otherwise aan is stable, though
slow.  Cfs+aan is even slower, though I never understood the reason.
I copy commonly used binaries locally to the client /tmp and bind
them in place, and use disk/mkfs to keep those up to date.  Aan
also does not work with tls boot out of the box.  I put my patch
below (some may not be necessary).  Though perhaps it would just
be easier if you maintain a full local copy of files in the client.

diff a4c1f3cc18df6fddd548f4df9f209695c4eb7263 uncommitted
--- a/rc/bin/rconnect
+++ b/rc/bin/rconnect
@@ -3,8 +3,9 @@
 
 argv0=$0
 aanto=86400 #one day
+aanport=0
 fn usage {
-	echo 'usage:' $argv0 '[-u user] [-k keypattern] [-p] [-t timeout] host remotescript localcommand arg ...' >[1=2]
+	echo 'usage:' $argv0 '[-u user] [-k keypattern] [-p] [-t timeout] [-P aanport] host remotescript localcommand arg ...' >[1=2]
 	exit 'usage'
 }
 
@@ -27,6 +28,7 @@
 		case -k; keyspec=($keyspec $2)
 		case -u; keyspec=($keyspec `{user=$2 pvar user})
 		case -t; aanto=$2
+		case -P; aanport=$2
 		case *; usage
 		}
 		shift
@@ -50,11 +52,12 @@
 	fn aanserver {
 		~ $#netdir 1 || netdir=/net/tcp/clone
 		netdir=`{basename -d $netdir} || exit
-		<>$netdir/clone {
-			netdir=$netdir/`{read} || exit
+		<>[4]$netdir/clone {
+			netdir=$netdir/`{read <[0=4]} || exit
 			>[3] $netdir/ctl {
-				echo -n 'announce *!0' >[1=3]
+				echo -n 'announce *!'$aanport >[1=3] || exit
 				echo `{cat $netdir/local} || exit
+				~ `{read} OK || exit
 				bind '#|' /mnt/aan || exit
 				exec aan -m $aanto $netdir <>/mnt/aan/data1 >[1=0] >[2]/dev/null &
 			}
@@ -63,7 +66,7 @@
 			rfork n
 			fn server {
 				echo -n aanserver $netdir >/proc/$pid/args
-				rm -f /env/^('fn#server' 'fn#aanserver' 'aanto')
+				rm -f /env/^('fn#server' 'fn#aanserver' 'aanto' 'aanport')
 				. <{n=`{read} && ! ~ $#n 0 && read -c $n} >[2=1]
 			}
 			exec tlssrv -A /bin/rc -c server
@@ -72,9 +75,10 @@
 	}
 	fn aanclient {
 		host=`{echo $host | sed 's/(.*)!([^!]+)$/\1/'}^!^`{read | sed 's/(.*)!([^!]+)$/\2/'} || exit
+		echo OK || exit
 		tlsclient -a -o <>{rfork s; aan -m $aanto -c $host >[1=0]} /bin/rc -c 'sendscript $*' $*
 	}
-	exec $argv0 -k $"keyspec $host <{builtin whatis aanserver; pvar aanto; echo aanserver} aanclient $*
+	exec $argv0 -k $"keyspec $host <{builtin whatis aanserver; pvar aanto; pvar aanport; echo aanserver} aanclient $*
 	exit
 }
 
--- a/rc/bin/rcpu
+++ b/rc/bin/rcpu
@@ -11,7 +11,7 @@
 	mount -nc /fd/0 /mnt/term || exit
 	bind -q /mnt/term/dev/cons /dev/cons
 	</mnt/term/fd/10 >/mnt/term/fd/11 >[2]/mnt/term/fd/12 {
-		~ $#dir 0 || cd $"dir || echo 'can''t change directory: '^$"dir >[1=2]
+		~ $#dir 0 || cd $"dir >/dev/null >[2=1]
 		switch($#cmd){
 		case 0;	cmd=(rc -li)
 		case 1; cmd=(rc -lc $cmd)
--- a/rc/bin/rimport
+++ b/rc/bin/rimport
@@ -3,7 +3,7 @@
 
 argv0=$0
 fn usage {
-	echo 'usage:' $argv0 '[-abcCnq] [-s name] [-u user] [-k keypattern] [-p] [-t timeout] host tree [mountpoint]' >[1=2]
+	echo 'usage:' $argv0 '[-abcCnq] [-s name] [-u user] [-k keypattern] [-p] [-t timeout] [-P aanport] host tree [mountpoint]' >[1=2]
 	exit 'usage'
 }
 
@@ -29,7 +29,7 @@
 	case *
 		~ $#* 1 && usage
 		switch($1){
-		case -[ukt]; connect=($connect $1 $2)
+		case -[uktP]; connect=($connect $1 $2)
 		case -s; sname=`{basename $2}
 		case *; usage
 		}
--- a/sys/src/9/boot/bootfs.proto
+++ b/sys/src/9/boot/bootfs.proto
@@ -1,6 +1,7 @@
 $objtype
 	bin
 		9660srv
+		aan
 		awk
 		auth
 			secstore
@@ -32,6 +33,7 @@
 			dnsgetip
 		hjfs
 		rc
+		read
 		rm
 		sed
 		sleep
@@ -44,6 +46,7 @@
 			ether
 			disk
 			kb
+		wc
 		xd
 rc
 	lib
@@ -55,6 +58,7 @@
 		fstype
 		diskparts
 		srvtls
+		srvtlsaan
 		nusbrc 555 sys sys ../boot/nusbrc
 		bootrc 555 sys sys ../boot/bootrc
 tmp
--- a/sys/src/9/boot/net.rc
+++ b/sys/src/9/boot/net.rc
@@ -65,19 +65,25 @@
 }
 
 fn connecttcp{
-	while(! ~ $#fs 0 && ! srv -q tcp!$fs(1)^!564 boot)
+	while(! ~ $#fs 0 && ! srv -q $srvopt tcp!$fs(1)^!564 boot)
 		fs=$fs(2-);
 	fs=$fs(1)
 }
 
 fn connecttls{
-	while(! ~ $#fs 0 && ! srvtls -q tcp!$fs(1)^!17020 boot)
+	while(! ~ $#fs 0 && ! srvtls -q $srvopt tcp!$fs(1)^!17020 boot)
 		fs=$fs(2-);
 	fs=$fs(1)
 }
 
+fn connecttlsaan{
+	while(! ~ $#fs 0 && ! srvtlsaan -q $srvopt tcp!$fs(1)^!17019 boot)
+		fs=$fs(2-);
+	fs=$fs(1)
+}
+
 fn connectil{
-	while(! ~ $#fs 0 && ! srv -q il!$fs(1)^!17008 boot)
+	while(! ~ $#fs 0 && ! srv -q $srvopt il!$fs(1)^!17008 boot)
 		fs=$fs(2-);
 	fs=$fs(1)
 }
@@ -84,6 +90,7 @@
 
 mtcp=(confignet connecttcp)
 mtls=(confignet connecttls)
+mtlsaan=(confignet connecttlsaan)
 mil=(confignet connectil)
 
-mt=(mtcp mtls mil $mt)
+mt=(mtcp mtls mtlsaan mil $mt)
--- a/sys/src/cmd/aan.c
+++ b/sys/src/cmd/aan.c
@@ -357,10 +357,6 @@
 
 			err[0] = '\0';
 			errstr(err, sizeof err);
-			if (strstr(err, "connection refused")) {
-				if(debug) fprint(2, "reconnect; server died...\n");
-				threadexitsall("server died...");
-			}
 			if(debug) fprint(2, "reconnect: dialed %s; %s\n", dialstring, err);
 			sleep(1000);
 		}


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [9front] cfs, aan and network recommendations
  2021-11-16  4:12 ` Xiao-Yong Jin
@ 2021-11-16 18:18   ` Noam Preil
  2021-11-16 19:04     ` Steve Simon
  0 siblings, 1 reply; 14+ messages in thread
From: Noam Preil @ 2021-11-16 18:18 UTC (permalink / raw)
  To: Xiao-Yong Jin

Cfs is slow. Not sure what conditions make it faster than using the 
remote fs directly - I'm sure there are some or it would not exist - but 
I have never found them.

- Noam

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [9front] cfs, aan and network recommendations
  2021-11-16 18:18   ` Noam Preil
@ 2021-11-16 19:04     ` Steve Simon
  2021-11-17  1:51       ` hiro
  0 siblings, 1 reply; 14+ messages in thread
From: Steve Simon @ 2021-11-16 19:04 UTC (permalink / raw)
  To: 9front


cfs solves the problem of very low bandwidth links rather well (think modems) however it needs a low latency link so it can confirm qid.version on every file access.

throughput issues can be mitigated for static filesystems, latency is the killer for 9p.

-Steve


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [9front] cfs, aan and network recommendations
  2021-11-16 19:04     ` Steve Simon
@ 2021-11-17  1:51       ` hiro
  2021-11-17  7:31         ` sirjofri
  0 siblings, 1 reply; 14+ messages in thread
From: hiro @ 2021-11-17  1:51 UTC (permalink / raw)
  To: 9front

thank you steve, this is a great summary and it explains everything
for me, didn't know it would check qid.version on every access!

On 11/16/21, Steve Simon <steve@quintile.net> wrote:
>
> cfs solves the problem of very low bandwidth links rather well (think
> modems) however it needs a low latency link so it can confirm qid.version on
> every file access.
>
> throughput issues can be mitigated for static filesystems, latency is the
> killer for 9p.
>
> -Steve
>
>

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [9front] cfs, aan and network recommendations
  2021-11-17  1:51       ` hiro
@ 2021-11-17  7:31         ` sirjofri
  2021-11-17  9:45           ` hiro
  0 siblings, 1 reply; 14+ messages in thread
From: sirjofri @ 2021-11-17  7:31 UTC (permalink / raw)
  To: hiro


17.11.2021 02:51:21 hiro <23hiro@gmail.com>:

> thank you steve, this is a great summary and it explains everything
> for me, didn't know it would check qid.version on every access!

I believe that's even mentioned in the man page. Well, not directly, but 
the 9p man mentions what version is for, so it makes sense.

sirjofri

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [9front] cfs, aan and network recommendations
  2021-11-17  7:31         ` sirjofri
@ 2021-11-17  9:45           ` hiro
  2021-11-17 10:31             ` sirjofri
  0 siblings, 1 reply; 14+ messages in thread
From: hiro @ 2021-11-17  9:45 UTC (permalink / raw)
  To: 9front

i *assumed* that the version would be a local copy!

On 11/17/21, sirjofri <sirjofri+ml-9front@sirjofri.de> wrote:
>
> 17.11.2021 02:51:21 hiro <23hiro@gmail.com>:
>
>> thank you steve, this is a great summary and it explains everything
>> for me, didn't know it would check qid.version on every access!
>
> I believe that's even mentioned in the man page. Well, not directly, but
> the 9p man mentions what version is for, so it makes sense.
>
> sirjofri
>

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [9front] cfs, aan and network recommendations
  2021-11-17  9:45           ` hiro
@ 2021-11-17 10:31             ` sirjofri
  2021-11-17 15:31               ` hiro
  2021-11-17 21:35               ` sirjofri
  0 siblings, 2 replies; 14+ messages in thread
From: sirjofri @ 2021-11-17 10:31 UTC (permalink / raw)
  To: hiro


17.11.2021 10:45:00 hiro <23hiro@gmail.com>:

> i *assumed* that the version would be a local copy!

The version itself is the local copy, saved on the cache partition. Cfs 
uses the qid.version field to determine if the version is outdated or 
not.

So it still needs to ask the server for the version of the file. If it's 
out of date, it fetches the new version and responds with data from that. 
If the qid.version didn't change, it answers with the data from the file 
directly. That's why latency is important.

However, when reading static files like pdfs and images (big filesize to 
protocol ratio) it's really a benefit.

Imagine a process with many big files where only few of them change, and 
the result of the process is quite small and written back to the file 
server, cfs should be quite good. Only those few changed files have to be 
transferred while the other big files remain the same on the cfs cache 
partition. The process does it magic calculation based on the cached 
data, and writes the small artifact back to the (original) server.

sirjofri

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [9front] cfs, aan and network recommendations
  2021-11-17 10:31             ` sirjofri
@ 2021-11-17 15:31               ` hiro
  2021-11-18 22:05                 ` Steve Simon
  2021-11-17 21:35               ` sirjofri
  1 sibling, 1 reply; 14+ messages in thread
From: hiro @ 2021-11-17 15:31 UTC (permalink / raw)
  To: 9front

> However, when reading static files like pdfs and images (big filesize to
> protocol ratio) it's really a benefit.
>

i doubt that, we have readahead, so unless you're bandwidth limited
the 9p latency shouldn't matter. that is when the mountpoint that
supplies the file is cwfs and it didn't get re-exported by something
else...

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [9front] cfs, aan and network recommendations
  2021-11-17 10:31             ` sirjofri
  2021-11-17 15:31               ` hiro
@ 2021-11-17 21:35               ` sirjofri
  2021-11-17 22:12                 ` hiro
  1 sibling, 1 reply; 14+ messages in thread
From: sirjofri @ 2021-11-17 21:35 UTC (permalink / raw)
  To: hiro

Hiro,

I didn't receive your mail to this address, only to my other address 
(whatever's going wrong with the mailing list).

Readahead or not, transferring gigabytes of data or not is here the 
question. If the version is the same and I don't need to download the 
data cfs gives me the cached file from the local disk.

In theory. I have no measured times and stuff like that.

sirjofri

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [9front] cfs, aan and network recommendations
  2021-11-17 21:35               ` sirjofri
@ 2021-11-17 22:12                 ` hiro
  2021-11-17 22:52                   ` Steve Simon
  0 siblings, 1 reply; 14+ messages in thread
From: hiro @ 2021-11-17 22:12 UTC (permalink / raw)
  To: 9front

yeah, but just like steve simon said, it seems at least in relation
it's much better suited for coping with bandwidth limitations than
high latency...

i.e. it probably would have made sense on some local serial connection
or dialup in the same city...

comparatively it's useless for the internet with those much higher
latencies, and bandwidth has suddenly become quite big, so "why waste
local space" (says the cloud fanboy... not completely without reason).

of course it's more complex than i make it sound, there are still
scenarios where saving bandwidth makes sense. it's just that they are
less, and often this doesn't reduce the end-user waiting times any
more, while formerly it did.

On 11/17/21, sirjofri <sirjofri+ml-9front@sirjofri.de> wrote:
> Hiro,
>
> I didn't receive your mail to this address, only to my other address
> (whatever's going wrong with the mailing list).
>
> Readahead or not, transferring gigabytes of data or not is here the
> question. If the version is the same and I don't need to download the
> data cfs gives me the cached file from the local disk.
>
> In theory. I have no measured times and stuff like that.
>
> sirjofri
>

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [9front] cfs, aan and network recommendations
  2021-11-17 22:12                 ` hiro
@ 2021-11-17 22:52                   ` Steve Simon
  0 siblings, 0 replies; 14+ messages in thread
From: Steve Simon @ 2021-11-17 22:52 UTC (permalink / raw)
  To: 9front


cfs could be modified, you could relax the rules…

for example if you know your files are readonly you only need to do a dirread, no subsequent checks.

if the path matches /*/bin/* assume the version is up to date (bin dires are fairly static)

you could keep a timestamp when the version was last checked if the file has not changed for (say) a week then its version only needs to be validated (say) once a day.

if you have recently fetched the versions on the path /sys/src/9/pc dont check them again until (say) 5 mins have elapsed.

the other approach suggested by nemo was to extend 9p with streaming apis - readfile and writefile. these would allow a complete file to be read with only 1 rtt rather than an rtt per path component, 1 for walk, open, and close, and 1 for each 8k of data.

all of these have disadvantages but in real use they could make life easier woth a remote fs.

-Steve


^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [9front] cfs, aan and network recommendations
  2021-11-17 15:31               ` hiro
@ 2021-11-18 22:05                 ` Steve Simon
  2021-11-18 23:21                   ` hiro
  0 siblings, 1 reply; 14+ messages in thread
From: Steve Simon @ 2021-11-18 22:05 UTC (permalink / raw)
  To: 9front


i am interested, where is the readahead? in cwfs, in the 9p kernel? in page(1)?

-Steve


On 18 Nov 2021, at 10:12 am, hiro <23hiro@gmail.com> wrote:


> 
> However, when reading static files like pdfs and images (big filesize to
> protocol ratio) it's really a benefit.
> 

i doubt that, we have readahead, so unless you're bandwidth limited
the 9p latency shouldn't matter. that is when the mountpoint that
supplies the file is cwfs and it didn't get re-exported by something
else...

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [9front] cfs, aan and network recommendations
  2021-11-18 22:05                 ` Steve Simon
@ 2021-11-18 23:21                   ` hiro
  0 siblings, 0 replies; 14+ messages in thread
From: hiro @ 2021-11-18 23:21 UTC (permalink / raw)
  To: 9front

it is only done with filesystems, with other synthetic files we cannot seek.

http://git.9front.org/plan9front/plan9front/6617c63a374f7211b41252d3957e8a89061b8a49/commit.html

later it got changed to only apply to disk-filesystems, can't find
that commit right now

On 11/18/21, Steve Simon <steve@quintile.net> wrote:
>
> i am interested, where is the readahead? in cwfs, in the 9p kernel? in
> page(1)?
>
> -Steve
>
>
> On 18 Nov 2021, at 10:12 am, hiro <23hiro@gmail.com> wrote:
>
> 
>>
>> However, when reading static files like pdfs and images (big filesize to
>> protocol ratio) it's really a benefit.
>>
>
> i doubt that, we have readahead, so unless you're bandwidth limited
> the 9p latency shouldn't matter. that is when the mountpoint that
> supplies the file is cwfs and it didn't get re-exported by something
> else...
>

^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2021-11-18 23:36 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-15 22:39 [9front] cfs, aan and network recommendations sirjofri
2021-11-16  4:12 ` Xiao-Yong Jin
2021-11-16 18:18   ` Noam Preil
2021-11-16 19:04     ` Steve Simon
2021-11-17  1:51       ` hiro
2021-11-17  7:31         ` sirjofri
2021-11-17  9:45           ` hiro
2021-11-17 10:31             ` sirjofri
2021-11-17 15:31               ` hiro
2021-11-18 22:05                 ` Steve Simon
2021-11-18 23:21                   ` hiro
2021-11-17 21:35               ` sirjofri
2021-11-17 22:12                 ` hiro
2021-11-17 22:52                   ` Steve Simon

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).