* [9front] cfs, aan and network recommendations @ 2021-11-15 22:39 sirjofri 2021-11-16 4:12 ` Xiao-Yong Jin 0 siblings, 1 reply; 14+ messages in thread From: sirjofri @ 2021-11-15 22:39 UTC (permalink / raw) To: 9front Hello all, (I tried sending this mail earlier, but it seems it never reached the mailing list.) due to a full worm on my cwfs I'm rethinking my network setup. Currently I have one VPS with limited storage (usually not much, the one that's almost full). I also have a larger/more powerful computer at home with a standard German internet connection, so the IP address can change every day and the connection is not always up. At the moment both machines are full file servers. The VPS (oat) is also mail server, http server, as well as auth and secstore. The other machine (leaf) is basically just cpu+fs, I had most of my data there. Since I moved to another apartment and I had internet connection issues I wasn't able to put leaf online, and I also only seem to have IPv6 here (from outside), so I was never really able to connect to leaf from outside. However, I believe it's possible. My current thoughts were, I could try to make leaf my only cpu+fs for working and configuration etc, and have oat start with leaf as a fileserver. Since the connection is likely not reliable and potentially slow I thought about combining cfs with aan. Did anyone try something like that? How reliable is aan? I know it handles short interruptions reasonably well (e.g. switching between LTE and WiFi), but what if something more happens? How could I handle potential rebooting (do I need to reboot oat each time I reboot leaf then)? I was also thinking about just removing leaf from my network and only rely on VPS. They are always available and hardware managed, as well as cheap enough. I'd then just like to have a good/solid filesystem setup (cwfs vs hjfs, or potentially even fossil?). Do you have good recommendations? Ideally I want a dump every day for the last X months, and then free older stuff (as in, I care about the data, but not the history). Maybe that's a use case for classical backups (daily, weekly, monthly, ...) and not so much for a dumping filesystem. Do you have any ideas and recommendations? sirjofri ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9front] cfs, aan and network recommendations 2021-11-15 22:39 [9front] cfs, aan and network recommendations sirjofri @ 2021-11-16 4:12 ` Xiao-Yong Jin 2021-11-16 18:18 ` Noam Preil 0 siblings, 1 reply; 14+ messages in thread From: Xiao-Yong Jin @ 2021-11-16 4:12 UTC (permalink / raw) To: 9front > On Nov 15, 2021, at 4:39 PM, sirjofri <sirjofri+ml-9front@sirjofri.de> wrote: > > Did anyone try something like that? How reliable is aan? I know it handles short interruptions reasonably well (e.g. switching between LTE and WiFi), but what if something more happens? How could I handle potential rebooting (do I need to reboot oat each time I reboot leaf then)? Aan server and client has to sync up. If you reboot the server, you have to reboot the client. Otherwise aan is stable, though slow. Cfs+aan is even slower, though I never understood the reason. I copy commonly used binaries locally to the client /tmp and bind them in place, and use disk/mkfs to keep those up to date. Aan also does not work with tls boot out of the box. I put my patch below (some may not be necessary). Though perhaps it would just be easier if you maintain a full local copy of files in the client. diff a4c1f3cc18df6fddd548f4df9f209695c4eb7263 uncommitted --- a/rc/bin/rconnect +++ b/rc/bin/rconnect @@ -3,8 +3,9 @@ argv0=$0 aanto=86400 #one day +aanport=0 fn usage { - echo 'usage:' $argv0 '[-u user] [-k keypattern] [-p] [-t timeout] host remotescript localcommand arg ...' >[1=2] + echo 'usage:' $argv0 '[-u user] [-k keypattern] [-p] [-t timeout] [-P aanport] host remotescript localcommand arg ...' >[1=2] exit 'usage' } @@ -27,6 +28,7 @@ case -k; keyspec=($keyspec $2) case -u; keyspec=($keyspec `{user=$2 pvar user}) case -t; aanto=$2 + case -P; aanport=$2 case *; usage } shift @@ -50,11 +52,12 @@ fn aanserver { ~ $#netdir 1 || netdir=/net/tcp/clone netdir=`{basename -d $netdir} || exit - <>$netdir/clone { - netdir=$netdir/`{read} || exit + <>[4]$netdir/clone { + netdir=$netdir/`{read <[0=4]} || exit >[3] $netdir/ctl { - echo -n 'announce *!0' >[1=3] + echo -n 'announce *!'$aanport >[1=3] || exit echo `{cat $netdir/local} || exit + ~ `{read} OK || exit bind '#|' /mnt/aan || exit exec aan -m $aanto $netdir <>/mnt/aan/data1 >[1=0] >[2]/dev/null & } @@ -63,7 +66,7 @@ rfork n fn server { echo -n aanserver $netdir >/proc/$pid/args - rm -f /env/^('fn#server' 'fn#aanserver' 'aanto') + rm -f /env/^('fn#server' 'fn#aanserver' 'aanto' 'aanport') . <{n=`{read} && ! ~ $#n 0 && read -c $n} >[2=1] } exec tlssrv -A /bin/rc -c server @@ -72,9 +75,10 @@ } fn aanclient { host=`{echo $host | sed 's/(.*)!([^!]+)$/\1/'}^!^`{read | sed 's/(.*)!([^!]+)$/\2/'} || exit + echo OK || exit tlsclient -a -o <>{rfork s; aan -m $aanto -c $host >[1=0]} /bin/rc -c 'sendscript $*' $* } - exec $argv0 -k $"keyspec $host <{builtin whatis aanserver; pvar aanto; echo aanserver} aanclient $* + exec $argv0 -k $"keyspec $host <{builtin whatis aanserver; pvar aanto; pvar aanport; echo aanserver} aanclient $* exit } --- a/rc/bin/rcpu +++ b/rc/bin/rcpu @@ -11,7 +11,7 @@ mount -nc /fd/0 /mnt/term || exit bind -q /mnt/term/dev/cons /dev/cons </mnt/term/fd/10 >/mnt/term/fd/11 >[2]/mnt/term/fd/12 { - ~ $#dir 0 || cd $"dir || echo 'can''t change directory: '^$"dir >[1=2] + ~ $#dir 0 || cd $"dir >/dev/null >[2=1] switch($#cmd){ case 0; cmd=(rc -li) case 1; cmd=(rc -lc $cmd) --- a/rc/bin/rimport +++ b/rc/bin/rimport @@ -3,7 +3,7 @@ argv0=$0 fn usage { - echo 'usage:' $argv0 '[-abcCnq] [-s name] [-u user] [-k keypattern] [-p] [-t timeout] host tree [mountpoint]' >[1=2] + echo 'usage:' $argv0 '[-abcCnq] [-s name] [-u user] [-k keypattern] [-p] [-t timeout] [-P aanport] host tree [mountpoint]' >[1=2] exit 'usage' } @@ -29,7 +29,7 @@ case * ~ $#* 1 && usage switch($1){ - case -[ukt]; connect=($connect $1 $2) + case -[uktP]; connect=($connect $1 $2) case -s; sname=`{basename $2} case *; usage } --- a/sys/src/9/boot/bootfs.proto +++ b/sys/src/9/boot/bootfs.proto @@ -1,6 +1,7 @@ $objtype bin 9660srv + aan awk auth secstore @@ -32,6 +33,7 @@ dnsgetip hjfs rc + read rm sed sleep @@ -44,6 +46,7 @@ ether disk kb + wc xd rc lib @@ -55,6 +58,7 @@ fstype diskparts srvtls + srvtlsaan nusbrc 555 sys sys ../boot/nusbrc bootrc 555 sys sys ../boot/bootrc tmp --- a/sys/src/9/boot/net.rc +++ b/sys/src/9/boot/net.rc @@ -65,19 +65,25 @@ } fn connecttcp{ - while(! ~ $#fs 0 && ! srv -q tcp!$fs(1)^!564 boot) + while(! ~ $#fs 0 && ! srv -q $srvopt tcp!$fs(1)^!564 boot) fs=$fs(2-); fs=$fs(1) } fn connecttls{ - while(! ~ $#fs 0 && ! srvtls -q tcp!$fs(1)^!17020 boot) + while(! ~ $#fs 0 && ! srvtls -q $srvopt tcp!$fs(1)^!17020 boot) fs=$fs(2-); fs=$fs(1) } +fn connecttlsaan{ + while(! ~ $#fs 0 && ! srvtlsaan -q $srvopt tcp!$fs(1)^!17019 boot) + fs=$fs(2-); + fs=$fs(1) +} + fn connectil{ - while(! ~ $#fs 0 && ! srv -q il!$fs(1)^!17008 boot) + while(! ~ $#fs 0 && ! srv -q $srvopt il!$fs(1)^!17008 boot) fs=$fs(2-); fs=$fs(1) } @@ -84,6 +90,7 @@ mtcp=(confignet connecttcp) mtls=(confignet connecttls) +mtlsaan=(confignet connecttlsaan) mil=(confignet connectil) -mt=(mtcp mtls mil $mt) +mt=(mtcp mtls mtlsaan mil $mt) --- a/sys/src/cmd/aan.c +++ b/sys/src/cmd/aan.c @@ -357,10 +357,6 @@ err[0] = '\0'; errstr(err, sizeof err); - if (strstr(err, "connection refused")) { - if(debug) fprint(2, "reconnect; server died...\n"); - threadexitsall("server died..."); - } if(debug) fprint(2, "reconnect: dialed %s; %s\n", dialstring, err); sleep(1000); } ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9front] cfs, aan and network recommendations 2021-11-16 4:12 ` Xiao-Yong Jin @ 2021-11-16 18:18 ` Noam Preil 2021-11-16 19:04 ` Steve Simon 0 siblings, 1 reply; 14+ messages in thread From: Noam Preil @ 2021-11-16 18:18 UTC (permalink / raw) To: Xiao-Yong Jin Cfs is slow. Not sure what conditions make it faster than using the remote fs directly - I'm sure there are some or it would not exist - but I have never found them. - Noam ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9front] cfs, aan and network recommendations 2021-11-16 18:18 ` Noam Preil @ 2021-11-16 19:04 ` Steve Simon 2021-11-17 1:51 ` hiro 0 siblings, 1 reply; 14+ messages in thread From: Steve Simon @ 2021-11-16 19:04 UTC (permalink / raw) To: 9front cfs solves the problem of very low bandwidth links rather well (think modems) however it needs a low latency link so it can confirm qid.version on every file access. throughput issues can be mitigated for static filesystems, latency is the killer for 9p. -Steve ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9front] cfs, aan and network recommendations 2021-11-16 19:04 ` Steve Simon @ 2021-11-17 1:51 ` hiro 2021-11-17 7:31 ` sirjofri 0 siblings, 1 reply; 14+ messages in thread From: hiro @ 2021-11-17 1:51 UTC (permalink / raw) To: 9front thank you steve, this is a great summary and it explains everything for me, didn't know it would check qid.version on every access! On 11/16/21, Steve Simon <steve@quintile.net> wrote: > > cfs solves the problem of very low bandwidth links rather well (think > modems) however it needs a low latency link so it can confirm qid.version on > every file access. > > throughput issues can be mitigated for static filesystems, latency is the > killer for 9p. > > -Steve > > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9front] cfs, aan and network recommendations 2021-11-17 1:51 ` hiro @ 2021-11-17 7:31 ` sirjofri 2021-11-17 9:45 ` hiro 0 siblings, 1 reply; 14+ messages in thread From: sirjofri @ 2021-11-17 7:31 UTC (permalink / raw) To: hiro 17.11.2021 02:51:21 hiro <23hiro@gmail.com>: > thank you steve, this is a great summary and it explains everything > for me, didn't know it would check qid.version on every access! I believe that's even mentioned in the man page. Well, not directly, but the 9p man mentions what version is for, so it makes sense. sirjofri ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9front] cfs, aan and network recommendations 2021-11-17 7:31 ` sirjofri @ 2021-11-17 9:45 ` hiro 2021-11-17 10:31 ` sirjofri 0 siblings, 1 reply; 14+ messages in thread From: hiro @ 2021-11-17 9:45 UTC (permalink / raw) To: 9front i *assumed* that the version would be a local copy! On 11/17/21, sirjofri <sirjofri+ml-9front@sirjofri.de> wrote: > > 17.11.2021 02:51:21 hiro <23hiro@gmail.com>: > >> thank you steve, this is a great summary and it explains everything >> for me, didn't know it would check qid.version on every access! > > I believe that's even mentioned in the man page. Well, not directly, but > the 9p man mentions what version is for, so it makes sense. > > sirjofri > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9front] cfs, aan and network recommendations 2021-11-17 9:45 ` hiro @ 2021-11-17 10:31 ` sirjofri 2021-11-17 15:31 ` hiro 2021-11-17 21:35 ` sirjofri 0 siblings, 2 replies; 14+ messages in thread From: sirjofri @ 2021-11-17 10:31 UTC (permalink / raw) To: hiro 17.11.2021 10:45:00 hiro <23hiro@gmail.com>: > i *assumed* that the version would be a local copy! The version itself is the local copy, saved on the cache partition. Cfs uses the qid.version field to determine if the version is outdated or not. So it still needs to ask the server for the version of the file. If it's out of date, it fetches the new version and responds with data from that. If the qid.version didn't change, it answers with the data from the file directly. That's why latency is important. However, when reading static files like pdfs and images (big filesize to protocol ratio) it's really a benefit. Imagine a process with many big files where only few of them change, and the result of the process is quite small and written back to the file server, cfs should be quite good. Only those few changed files have to be transferred while the other big files remain the same on the cfs cache partition. The process does it magic calculation based on the cached data, and writes the small artifact back to the (original) server. sirjofri ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9front] cfs, aan and network recommendations 2021-11-17 10:31 ` sirjofri @ 2021-11-17 15:31 ` hiro 2021-11-18 22:05 ` Steve Simon 2021-11-17 21:35 ` sirjofri 1 sibling, 1 reply; 14+ messages in thread From: hiro @ 2021-11-17 15:31 UTC (permalink / raw) To: 9front > However, when reading static files like pdfs and images (big filesize to > protocol ratio) it's really a benefit. > i doubt that, we have readahead, so unless you're bandwidth limited the 9p latency shouldn't matter. that is when the mountpoint that supplies the file is cwfs and it didn't get re-exported by something else... ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9front] cfs, aan and network recommendations 2021-11-17 15:31 ` hiro @ 2021-11-18 22:05 ` Steve Simon 2021-11-18 23:21 ` hiro 0 siblings, 1 reply; 14+ messages in thread From: Steve Simon @ 2021-11-18 22:05 UTC (permalink / raw) To: 9front i am interested, where is the readahead? in cwfs, in the 9p kernel? in page(1)? -Steve On 18 Nov 2021, at 10:12 am, hiro <23hiro@gmail.com> wrote: > > However, when reading static files like pdfs and images (big filesize to > protocol ratio) it's really a benefit. > i doubt that, we have readahead, so unless you're bandwidth limited the 9p latency shouldn't matter. that is when the mountpoint that supplies the file is cwfs and it didn't get re-exported by something else... ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9front] cfs, aan and network recommendations 2021-11-18 22:05 ` Steve Simon @ 2021-11-18 23:21 ` hiro 0 siblings, 0 replies; 14+ messages in thread From: hiro @ 2021-11-18 23:21 UTC (permalink / raw) To: 9front it is only done with filesystems, with other synthetic files we cannot seek. http://git.9front.org/plan9front/plan9front/6617c63a374f7211b41252d3957e8a89061b8a49/commit.html later it got changed to only apply to disk-filesystems, can't find that commit right now On 11/18/21, Steve Simon <steve@quintile.net> wrote: > > i am interested, where is the readahead? in cwfs, in the 9p kernel? in > page(1)? > > -Steve > > > On 18 Nov 2021, at 10:12 am, hiro <23hiro@gmail.com> wrote: > > >> >> However, when reading static files like pdfs and images (big filesize to >> protocol ratio) it's really a benefit. >> > > i doubt that, we have readahead, so unless you're bandwidth limited > the 9p latency shouldn't matter. that is when the mountpoint that > supplies the file is cwfs and it didn't get re-exported by something > else... > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9front] cfs, aan and network recommendations 2021-11-17 10:31 ` sirjofri 2021-11-17 15:31 ` hiro @ 2021-11-17 21:35 ` sirjofri 2021-11-17 22:12 ` hiro 1 sibling, 1 reply; 14+ messages in thread From: sirjofri @ 2021-11-17 21:35 UTC (permalink / raw) To: hiro Hiro, I didn't receive your mail to this address, only to my other address (whatever's going wrong with the mailing list). Readahead or not, transferring gigabytes of data or not is here the question. If the version is the same and I don't need to download the data cfs gives me the cached file from the local disk. In theory. I have no measured times and stuff like that. sirjofri ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9front] cfs, aan and network recommendations 2021-11-17 21:35 ` sirjofri @ 2021-11-17 22:12 ` hiro 2021-11-17 22:52 ` Steve Simon 0 siblings, 1 reply; 14+ messages in thread From: hiro @ 2021-11-17 22:12 UTC (permalink / raw) To: 9front yeah, but just like steve simon said, it seems at least in relation it's much better suited for coping with bandwidth limitations than high latency... i.e. it probably would have made sense on some local serial connection or dialup in the same city... comparatively it's useless for the internet with those much higher latencies, and bandwidth has suddenly become quite big, so "why waste local space" (says the cloud fanboy... not completely without reason). of course it's more complex than i make it sound, there are still scenarios where saving bandwidth makes sense. it's just that they are less, and often this doesn't reduce the end-user waiting times any more, while formerly it did. On 11/17/21, sirjofri <sirjofri+ml-9front@sirjofri.de> wrote: > Hiro, > > I didn't receive your mail to this address, only to my other address > (whatever's going wrong with the mailing list). > > Readahead or not, transferring gigabytes of data or not is here the > question. If the version is the same and I don't need to download the > data cfs gives me the cached file from the local disk. > > In theory. I have no measured times and stuff like that. > > sirjofri > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [9front] cfs, aan and network recommendations 2021-11-17 22:12 ` hiro @ 2021-11-17 22:52 ` Steve Simon 0 siblings, 0 replies; 14+ messages in thread From: Steve Simon @ 2021-11-17 22:52 UTC (permalink / raw) To: 9front cfs could be modified, you could relax the rules… for example if you know your files are readonly you only need to do a dirread, no subsequent checks. if the path matches /*/bin/* assume the version is up to date (bin dires are fairly static) you could keep a timestamp when the version was last checked if the file has not changed for (say) a week then its version only needs to be validated (say) once a day. if you have recently fetched the versions on the path /sys/src/9/pc dont check them again until (say) 5 mins have elapsed. the other approach suggested by nemo was to extend 9p with streaming apis - readfile and writefile. these would allow a complete file to be read with only 1 rtt rather than an rtt per path component, 1 for walk, open, and close, and 1 for each 8k of data. all of these have disadvantages but in real use they could make life easier woth a remote fs. -Steve ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2021-11-18 23:36 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-11-15 22:39 [9front] cfs, aan and network recommendations sirjofri 2021-11-16 4:12 ` Xiao-Yong Jin 2021-11-16 18:18 ` Noam Preil 2021-11-16 19:04 ` Steve Simon 2021-11-17 1:51 ` hiro 2021-11-17 7:31 ` sirjofri 2021-11-17 9:45 ` hiro 2021-11-17 10:31 ` sirjofri 2021-11-17 15:31 ` hiro 2021-11-18 22:05 ` Steve Simon 2021-11-18 23:21 ` hiro 2021-11-17 21:35 ` sirjofri 2021-11-17 22:12 ` hiro 2021-11-17 22:52 ` Steve Simon
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).