9front - general discussion about 9front
 help / color / mirror / Atom feed
* [9front] Two file servers sharing an auth server
@ 2022-11-02 19:18 Nathan Zimmen-Myers
  2022-11-02 20:23 ` ori
  0 siblings, 1 reply; 11+ messages in thread
From: Nathan Zimmen-Myers @ 2022-11-02 19:18 UTC (permalink / raw)
  To: 9front

Hello all,

I'm still somewhat new to getting 9 set up in a distributed way, and
am struggling to have two separate fileservers sharing an auth server.
Firstly, I want to confirm that this is feasible and (relatively)
common. If this is uncommon/incorrect, is it preferable to maintain
localized documents on a non-OS drive, while booting from an auth
server?

The initial goal that I had started with, was to have a web server
installation that boots from disk, but obtains authentication
information from an authentication server with which it does not share
a disk.

To illustrate: fs1 hosts auth with its own fileserver, currently
nothing else. web1 hosts a fileserver and will host an httpd. These
VPSs are both hosted in the same Vultr datacenter, so the latency
between them is minimal.

As ori mentioned in IRC, keyfs is used to maintain the authentication
database which can be queried remotely (I believe? this might not be a
correct interpretation), is it correct to simply mount keyfs on web1
from fs1?

The other option which comes to mind, is to boot web1 from fs1, and
use web1's disk for documents to be connected to httpd, which may make
long-term maintenance easier. Would this be a more common
configuration?

My assumption is that web1 should not get all of its web documents
from fs1, as this would duplicate traffic - and I am happy if I am
wrong.

Thanks all!

nzm

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [9front] Two file servers sharing an auth server
  2022-11-02 19:18 [9front] Two file servers sharing an auth server Nathan Zimmen-Myers
@ 2022-11-02 20:23 ` ori
  2022-11-02 23:53   ` william
  0 siblings, 1 reply; 11+ messages in thread
From: ori @ 2022-11-02 20:23 UTC (permalink / raw)
  To: 9front

Quoth Nathan Zimmen-Myers <nathan@nzm.ca>:
> As ori mentioned in IRC, keyfs is used to maintain the authentication
> database which can be queried remotely (I believe? this might not be a
> correct interpretation), is it correct to simply mount keyfs on web1
> from fs1?

No; I thought you wanted to move or replicate the users between the
auth servers.

> The other option which comes to mind, is to boot web1 from fs1, and
> use web1's disk for documents to be connected to httpd, which may make
> long-term maintenance easier. Would this be a more common
> configuration?

Just give them the same auth server:

sys=foo ether=aabbccddeeff ip=1.2.3.4
	auth=1.2.3.4

sys=bar ether=aabbccddeeff ip=2.4.6.8
	auth=1.2.3.4

even better if you can share an fs, and
therefore share the ndb, but you don't
need to; just point at the auth server
you want to use.

you don't even need to *own* that auth
server, though you'll need the owner of
the auth server to add a speaksfor line
if your hostowner is different.


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [9front] Two file servers sharing an auth server
  2022-11-02 20:23 ` ori
@ 2022-11-02 23:53   ` william
  2022-11-03  0:43     ` Nathan Zimmen-Myers
  0 siblings, 1 reply; 11+ messages in thread
From: william @ 2022-11-02 23:53 UTC (permalink / raw)
  To: 9front

On a bigger note. I was curious about the users. Its nice to have a single auth
as mentioned by ori. But what's the bigger goal. Do you plan to have a lot of users. 
Do those users know basic unix commands to navigate around? What type of files
or media to you plan to serve?


Quoth ori@eigenstate.org:
> Quoth Nathan Zimmen-Myers <nathan@nzm.ca>:
> > As ori mentioned in IRC, keyfs is used to maintain the authentication
> > database which can be queried remotely (I believe? this might not be a
> > correct interpretation), is it correct to simply mount keyfs on web1
> > from fs1?
> 
> No; I thought you wanted to move or replicate the users between the
> auth servers.
> 
> > The other option which comes to mind, is to boot web1 from fs1, and
> > use web1's disk for documents to be connected to httpd, which may make
> > long-term maintenance easier. Would this be a more common
> > configuration?
> 
> Just give them the same auth server:
> 
> sys=foo ether=aabbccddeeff ip=1.2.3.4
> 	auth=1.2.3.4
> 
> sys=bar ether=aabbccddeeff ip=2.4.6.8
> 	auth=1.2.3.4
> 
> even better if you can share an fs, and
> therefore share the ndb, but you don't
> need to; just point at the auth server
> you want to use.
> 
> you don't even need to *own* that auth
> server, though you'll need the owner of
> the auth server to add a speaksfor line
> if your hostowner is different.
> 
> 


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [9front] Two file servers sharing an auth server
  2022-11-02 23:53   ` william
@ 2022-11-03  0:43     ` Nathan Zimmen-Myers
  2022-11-03  9:28       ` Steve Simon
  0 siblings, 1 reply; 11+ messages in thread
From: Nathan Zimmen-Myers @ 2022-11-03  0:43 UTC (permalink / raw)
  To: 9front

William's question is important, the reality is that I'm just
separating out which systems have access to what on their primary
fileservers. This is just personal infrastructure and trying to have
multiple accounts might just be a mental holdover from unix. All that
I'm doing now with web1 is serving my personal website, and fs1 will
eventually be used as a backend fs for "compute" nodes - temporary
cloud servers for running compilations and other cpu-heavy tasks.

As per Ori's response, I think I misunderstood a piece of the
documentation and didn't tab the auth=ip on web1 or fs1, which
explains some of the confusing error messages.

I think for now I'll just cache web files on web1's hard drive, with
web1 booting off of fs1. I don't see any reason why that would be too
slow, and while I'm still learning the system I want to cut down the
amount of work I do to keep configuration changes the same across both
systems.

On Wed, Nov 2, 2022 at 7:56 PM <william@thinktankworkspaces.com> wrote:
>
> On a bigger note. I was curious about the users. Its nice to have a single auth
> as mentioned by ori. But what's the bigger goal. Do you plan to have a lot of users.
> Do those users know basic unix commands to navigate around? What type of files
> or media to you plan to serve?
>
>
> Quoth ori@eigenstate.org:
> > Quoth Nathan Zimmen-Myers <nathan@nzm.ca>:
> > > As ori mentioned in IRC, keyfs is used to maintain the authentication
> > > database which can be queried remotely (I believe? this might not be a
> > > correct interpretation), is it correct to simply mount keyfs on web1
> > > from fs1?
> >
> > No; I thought you wanted to move or replicate the users between the
> > auth servers.
> >
> > > The other option which comes to mind, is to boot web1 from fs1, and
> > > use web1's disk for documents to be connected to httpd, which may make
> > > long-term maintenance easier. Would this be a more common
> > > configuration?
> >
> > Just give them the same auth server:
> >
> > sys=foo ether=aabbccddeeff ip=1.2.3.4
> >       auth=1.2.3.4
> >
> > sys=bar ether=aabbccddeeff ip=2.4.6.8
> >       auth=1.2.3.4
> >
> > even better if you can share an fs, and
> > therefore share the ndb, but you don't
> > need to; just point at the auth server
> > you want to use.
> >
> > you don't even need to *own* that auth
> > server, though you'll need the owner of
> > the auth server to add a speaksfor line
> > if your hostowner is different.
> >
> >
>

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [9front] Two file servers sharing an auth server
  2022-11-03  0:43     ` Nathan Zimmen-Myers
@ 2022-11-03  9:28       ` Steve Simon
  2022-11-03 10:33         ` sirjofri
  0 siblings, 1 reply; 11+ messages in thread
From: Steve Simon @ 2022-11-03  9:28 UTC (permalink / raw)
  To: 9front

having the web server boot diskless is very nice, if its content is small that is enough.

for a little extra performance you can copy the web content to a ramfs at boot.

if the content is big you could mount a local spinning/ssd at boot onto /usr/web and serve that, but having a diskless web and cpu servers makes things very clean and easy to maintain.



> On 3 Nov 2022, at 08:32, Nathan Zimmen-Myers <nathan@nzm.ca> wrote:
> 
> William's question is important, the reality is that I'm just
> separating out which systems have access to what on their primary
> fileservers. This is just personal infrastructure and trying to have
> multiple accounts might just be a mental holdover from unix. All that
> I'm doing now with web1 is serving my personal website, and fs1 will
> eventually be used as a backend fs for "compute" nodes - temporary
> cloud servers for running compilations and other cpu-heavy tasks.
> 
> As per Ori's response, I think I misunderstood a piece of the
> documentation and didn't tab the auth=ip on web1 or fs1, which
> explains some of the confusing error messages.
> 
> I think for now I'll just cache web files on web1's hard drive, with
> web1 booting off of fs1. I don't see any reason why that would be too
> slow, and while I'm still learning the system I want to cut down the
> amount of work I do to keep configuration changes the same across both
> systems.
> 
>> On Wed, Nov 2, 2022 at 7:56 PM <william@thinktankworkspaces.com> wrote:
>> 
>> On a bigger note. I was curious about the users. Its nice to have a single auth
>> as mentioned by ori. But what's the bigger goal. Do you plan to have a lot of users.
>> Do those users know basic unix commands to navigate around? What type of files
>> or media to you plan to serve?
>> 
>> 
>> Quoth ori@eigenstate.org:
>>> Quoth Nathan Zimmen-Myers <nathan@nzm.ca>:
>>>> As ori mentioned in IRC, keyfs is used to maintain the authentication
>>>> database which can be queried remotely (I believe? this might not be a
>>>> correct interpretation), is it correct to simply mount keyfs on web1
>>>> from fs1?
>>> 
>>> No; I thought you wanted to move or replicate the users between the
>>> auth servers.
>>> 
>>>> The other option which comes to mind, is to boot web1 from fs1, and
>>>> use web1's disk for documents to be connected to httpd, which may make
>>>> long-term maintenance easier. Would this be a more common
>>>> configuration?
>>> 
>>> Just give them the same auth server:
>>> 
>>> sys=foo ether=aabbccddeeff ip=1.2.3.4
>>>      auth=1.2.3.4
>>> 
>>> sys=bar ether=aabbccddeeff ip=2.4.6.8
>>>      auth=1.2.3.4
>>> 
>>> even better if you can share an fs, and
>>> therefore share the ndb, but you don't
>>> need to; just point at the auth server
>>> you want to use.
>>> 
>>> you don't even need to *own* that auth
>>> server, though you'll need the owner of
>>> the auth server to add a speaksfor line
>>> if your hostowner is different.
>>> 
>>> 
>> 

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [9front] Two file servers sharing an auth server
  2022-11-03  9:28       ` Steve Simon
@ 2022-11-03 10:33         ` sirjofri
  2022-11-03 20:57           ` Steve Simon
  0 siblings, 1 reply; 11+ messages in thread
From: sirjofri @ 2022-11-03 10:33 UTC (permalink / raw)
  To: 9front


03.11.2022 10:30:27 Steve Simon <steve@quintile.net>:
> having the web server boot diskless is very nice, if its content is small that is enough.
>
> for a little extra performance you can copy the web content to a ramfs at boot.
>
> if the content is big you could mount a local spinning/ssd at boot onto /usr/web and serve that, but having a diskless web and cpu servers makes things very clean and easy to maintain.

Not to mention there's also cfs which can be nice for bigger stuff. Afaik it works by caching all accessed files on a local disk (except they're changed on the server) and all writes go to the server directly.

sirjofri

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [9front] Two file servers sharing an auth server
  2022-11-03 10:33         ` sirjofri
@ 2022-11-03 20:57           ` Steve Simon
  2022-11-04  0:00             ` william
  2022-11-04  6:58             ` sirjofri
  0 siblings, 2 replies; 11+ messages in thread
From: Steve Simon @ 2022-11-03 20:57 UTC (permalink / raw)
  To: 9front

i agree with all you say about cfs, however… 

beware, in my experience cfs is a real win where your bandwidth to the server is low.

 it is less helpful where the latency to the server is high (as cfs checks the file version number on every access).

-Steve

> On 3 Nov 2022, at 10:35, sirjofri <sirjofri+ml-9front@sirjofri.de> wrote:
> 
> 
> 03.11.2022 10:30:27 Steve Simon <steve@quintile.net>:
>> having the web server boot diskless is very nice, if its content is small that is enough.
>> 
>> for a little extra performance you can copy the web content to a ramfs at boot.
>> 
>> if the content is big you could mount a local spinning/ssd at boot onto /usr/web and serve that, but having a diskless web and cpu servers makes things very clean and easy to maintain.
> 
> Not to mention there's also cfs which can be nice for bigger stuff. Afaik it works by caching all accessed files on a local disk (except they're changed on the server) and all writes go to the server directly.
> 
> sirjofri

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [9front] Two file servers sharing an auth server
  2022-11-03 20:57           ` Steve Simon
@ 2022-11-04  0:00             ` william
  2022-11-04  6:58             ` sirjofri
  1 sibling, 0 replies; 11+ messages in thread
From: william @ 2022-11-04  0:00 UTC (permalink / raw)
  To: 9front

Very interesting. I would like to hear your progress, mybe through up a page. I have thought many times to expand my network but I'm a bit cheap and the webserver seems to be doing just fine. But I might be interested in isolating werc on another node. 

Quoth Steve Simon <steve@quintile.net>:
> i agree with all you say about cfs, however… 
> 
> beware, in my experience cfs is a real win where your bandwidth to the server is low.
> 
>  it is less helpful where the latency to the server is high (as cfs checks the file version number on every access).
> 
> -Steve
> 
> > On 3 Nov 2022, at 10:35, sirjofri <sirjofri+ml-9front@sirjofri.de> wrote:
> > 
> > 
> > 03.11.2022 10:30:27 Steve Simon <steve@quintile.net>:
> >> having the web server boot diskless is very nice, if its content is small that is enough.
> >> 
> >> for a little extra performance you can copy the web content to a ramfs at boot.
> >> 
> >> if the content is big you could mount a local spinning/ssd at boot onto /usr/web and serve that, but having a diskless web and cpu servers makes things very clean and easy to maintain.
> > 
> > Not to mention there's also cfs which can be nice for bigger stuff. Afaik it works by caching all accessed files on a local disk (except they're changed on the server) and all writes go to the server directly.
> > 
> > sirjofri
> 


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [9front] Two file servers sharing an auth server
  2022-11-03 20:57           ` Steve Simon
  2022-11-04  0:00             ` william
@ 2022-11-04  6:58             ` sirjofri
  2022-11-04  7:09               ` william
  1 sibling, 1 reply; 11+ messages in thread
From: sirjofri @ 2022-11-04  6:58 UTC (permalink / raw)
  To: 9front


03.11.2022 21:59:36 Steve Simon <steve@quintile.net>:

> i agree with all you say about cfs, however…
>
> beware, in my experience cfs is a real win where your bandwidth to the server is low.
>
> it is less helpful where the latency to the server is high (as cfs checks the file version number on every access).

I agree with everything you said. Back on spawngrid I built some web chat application (as an additional frontend for gridchat). It used long polling for chat messages and was fully rc based (rc-httpd). You can imagine how it really was impacting server performance, basically starting processes multiple times per second. Also process IDs rocketed into the sky very fast, I've never seen numbers that high. It felt like a forkbomb.

Then, mycroftiv told me about that and that fileserver load is very high and I had to find another solution. In my startup script I indeed just copied the executables (rc, cat, bc, ...) into a local ramfs and bound that over /bin so I don't have to change my programs. This improved everything.

Especially for script-based applications like that I can really recommend that. It's like preloading the binaries into memory before actually loading them. Works well for static stuff.

sirjofri

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [9front] Two file servers sharing an auth server
  2022-11-04  6:58             ` sirjofri
@ 2022-11-04  7:09               ` william
  2022-11-04  8:29                 ` sirjofri
  0 siblings, 1 reply; 11+ messages in thread
From: william @ 2022-11-04  7:09 UTC (permalink / raw)
  To: 9front

Wow. I'm really interested in learning more about this. I kind of figured performance would be better
local rather than served remotely. I might want to dig into this further and I'm curious about
concurrency. Can multiple rc-httpd services be started to handle concurrency?

Examples greatly welcomed

What about non-script apps. I was really interested in some of the web features golang has and if it will work well on another node

Quoth sirjofri <sirjofri+ml-9front@sirjofri.de>:
> 
> 03.11.2022 21:59:36 Steve Simon <steve@quintile.net>:
> 
> > i agree with all you say about cfs, however…
> >
> > beware, in my experience cfs is a real win where your bandwidth to the server is low.
> >
> > it is less helpful where the latency to the server is high (as cfs checks the file version number on every access).
> 
> I agree with everything you said. Back on spawngrid I built some web chat application (as an additional frontend for gridchat). It used long polling for chat messages and was fully rc based (rc-httpd). You can imagine how it really was impacting server performance, basically starting processes multiple times per second. Also process IDs rocketed into the sky very fast, I've never seen numbers that high. It felt like a forkbomb.
> 
> Then, mycroftiv told me about that and that fileserver load is very high and I had to find another solution. In my startup script I indeed just copied the executables (rc, cat, bc, ...) into a local ramfs and bound that over /bin so I don't have to change my programs. This improved everything.
> 
> Especially for script-based applications like that I can really recommend that. It's like preloading the binaries into memory before actually loading them. Works well for static stuff.
> 
> sirjofri
> 


^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [9front] Two file servers sharing an auth server
  2022-11-04  7:09               ` william
@ 2022-11-04  8:29                 ` sirjofri
  0 siblings, 0 replies; 11+ messages in thread
From: sirjofri @ 2022-11-04  8:29 UTC (permalink / raw)
  To: 9front


04.11.2022 08:11:26 william@thinktankworkspaces.com:

> Wow. I'm really interested in learning more about this. I kind of figured performance would be better
> local rather than served remotely. I might want to dig into this further and I'm curious about
> concurrency. Can multiple rc-httpd services be started to handle concurrency?
>
> Examples greatly welcomed
>
> What about non-script apps. I was really interested in some of the web features golang has and if it will work well on another node

Well, I'd say it depends on the application and what it does. Let's forget about user data for a moment, but I'm sure you can figure that out yourself.

Let's imagine a single-binary application for this. That binary starts once and runs forever, and it uses sed (as a separate application) for some task.

This binary could spawn one sed process and feed it data. In this case it doesn't really make much sense to copy the binaries to ramfs and start from there since it would only load both binaries once from the fileserver, start the processes and leave everything in memory. It doesn't matter if your application streams data to sed once per minute or 100 times in a second. The process is loaded already and no additional disk reads will happen.

Or, the application could spawn a sed process every time it uses it. This would mean, however, that the sed binary has to be loaded each time, which is can be a huge load for the disk. In this scenario it makes sense to copy the sed binary to some ramfs (or local disk if you need the memory otherwise, but sed is small).

It could even be worse if sed is dynamically linked, since the loader has to load all the binaries needed for sed.

rc-httpd is one of the worst examples though. It's totally fine to use it with some scale, but rc-httpd will fork many processes per connection.

Regarding your question about concurrency and rc-httpd: On Plan 9 you usually start your process per connection, so each connection from a client will have their own rc-httpd process (see /rc/bin/service/tcp80 or !tcp80). To keep the same state you need something persistent, like a filesystem or "real" files (whatever this means).

I think it would be possible to rewrite rc-httpd to spawn processes in the beginning and stream data to them, but I guess that would make rc-httpd more complicated and if you wanna do that you might as well use httpd or tcp80 or anything else.

If I wanted to design a scalable application for concurrency with Plan 9 in mind I'd probably create a filesystem that handles all synchronization, provides data for the compute nodes and receives results from them. That filesystem can be imported to all compute machines and then we could start the compute node processes on the compute system. Well, the binary could theoretically actually be part of the filesystem, this would allow us to distribute the task basically automatically.

Something similar to this:
- /mnt/concfs/
  - ctl (every process should read this and announce their presence and what they do)
  - data
  - program

Starting a compute process might be just importing the filesystem from the server, then run the "program" file (which handles everything else, like reading from ctl and doing the actual processing). You can see that it easily fits a very small shell script.

data would be something like the endpoint. For rendering an image it might be some structure for pixels, and the processes can just read scene data they need to process and write the pixel result. The data file could also be abstracted to a full filesystem hierarchy, which sometimes makes sense.

ctl would be the synchronization point. The server just tells the client which pixel to compute and the client just announces when it's done (after writing the pixel value to the data file). But that protocol would be hidden inside the "program" file, so the client machine actually knows nothing about what to compute and how the protocol works. It just has to know what file to execute.

Note that it's still possible to write standard multithreaded programs for stuff like that, using libthread, fork or anything like that. It all depends on the use case and the amount of scale.

It would be actually possible to have the "program" multithreaded, so each compute node would not run just one process but many.

Note that I never built a system like that, it's just some idea that would need refinement.

sirjofri

^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2022-11-04  8:31 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-11-02 19:18 [9front] Two file servers sharing an auth server Nathan Zimmen-Myers
2022-11-02 20:23 ` ori
2022-11-02 23:53   ` william
2022-11-03  0:43     ` Nathan Zimmen-Myers
2022-11-03  9:28       ` Steve Simon
2022-11-03 10:33         ` sirjofri
2022-11-03 20:57           ` Steve Simon
2022-11-04  0:00             ` william
2022-11-04  6:58             ` sirjofri
2022-11-04  7:09               ` william
2022-11-04  8:29                 ` sirjofri

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).