[9fans] A note about new software for Plan 9

[9fans] A note about new software for Plan 9

[mycroftiv]

Hi 9fans.  I think I wrote my Advanced Namespace Tools message in an
unclear way.  Let me try again.

I have some new software for Plan 9.  It is based on manipulating
namespaces.  The 9pcram kernel has a new bootup which lets it control
multiple independent namespaces.  It is the foundation for a
collection of tools which manipulate namespace.  I think it adds
useful new features to Plan 9 and also solves some existing issues.

This project, Advanced Namespace Tools, is an attempt to make
something of high quality which lives up to the Plan 9 standard and
also extends what Plan 9 can do.  I have created bootable VM images,
tutorials for the software, and extensive documentation.  I have been
using and testing it myself every day and I believe it is genuinely
useful.

ants.9gridchan.org is served with 9p, ftp, and http and has the
software itself

antfarm.9gridchan.org/Overview and related pages have tutorials for
pre-installed ANTS vms

9gridchan.org has links to 2 vm images, one tiny (6mb) and one fully
installed (132mb) which can be used to explore the Advanced Namespace
Tools.

I'd really like to discuss these things with other Plan 9 users - I'm
friendly and even though my software is called ANTS, I don't think it
will actually bite you.

-Ben Kidwell "mycroftiv"

Re: [9fans] A note about new software for Plan 9

[erik quanstrom]

> I have some new software for Plan 9.  It is based on manipulating
> namespaces.  The 9pcram kernel has a new bootup which lets it control
> multiple independent namespaces.  It is the foundation for a

this part is confusing.  plan 9 has always maintained 1 independent
namespace per process group.  so i'm sure that there's more to it
than this.  could you go into detail on this?

- erik

Re: [9fans] A note about new software for Plan 9

[mycroftiv]

Bakul Shah wrote:
> The question really is, can we virtualize plan9 machines while
> staying pretty much within the plan9 framework or by extending
> it in a way consistent with its design?
>
> To fully virtualize plan9 you'd have to virtualize kernel
> services as well.  May be you'd've to dream up a ## service
> that mediates access to hosts's #services!

Thanks again for your comments.  From your description of past
projects, I see you have professional experience in related domains,
so as a home user without a professional background, I'm glad to learn
more.  I will talk a little bit about what ANTS does right now in
relation to full virtualization/jailing.

As you point out, there is a big difference between per-process
namespaces and fully virtualized kernel devices.  The ANTS software
makes the choice to focus on what can be done right now with the
existing Plan 9 kernel with minimal changes.  I think there has been
previous discussion in years past of making multiple verions of the
kernel #devices available, and also some ideas about letting the Plan
9 kernel run as a userspace process.  If you had the ability to
synthesize kernel devices on demand and also run multiple kernels as
userspace processes, I think that would be pretty close to the
full-scale technically sophisticated general approach to Plan 9
jails/pseudo VMs.  I think a project to do those things would be
interesting and it has been discussed before, but so far as I know, it
hasn't been implemented in practice.

In relation to that approach, ANTS is very different, and the reason
for this is what you talked about in your first reply - what problems
ANTS is trying to solve.  In my use and testing, I discoved that
solving the problems that were important to me didn't require those
mechanisms.  The problems I was working on turned out to be solvable
just by using the existing Plan 9 namespace framework in a new way.
ANTS is designed to help administer venti/fossil/cpu grids by creating
an independent namespace that doesn't depend on those services.  Doing
this has the additional benefit that you can easily create independent
sub-namespaces that can be used for some of the same things that VMs
or jails can.

Because I am a home user using Plan 9 on a grid of machines in my
basement, the problems that I have been focused on solving are related
to making a home grid easy to create and administer, and letting me
blend different forms of Plan 9 like Labs and 9front and imported unix
resources more freely.  I think that in the larger and more
professional context, the problems and solutions are different.
Profesionally, you are very concerned with making sure that customer
A's data doesn't get mixed up with customer B's data so the solution
is to provide full isolation of the environments.

On my home grid, customer A and customer B are both me, and I want to
have access to everything at once.  I want an independent namespace
that I can use to administer my grid even if I need to reboot some
nodes, and I want to use Labs, 9front, p9p, inferno all in a mixed
environment.  My problem and needed solution is how I create and
manage all these different namespaces but also be able to move data
between them and have as much sharing as I want, and as much
separation as I need.

What I have found - and what ANTS is based on - is that even limited
to nothing but per-process namespaces and the existing Plan 9
framework, you get a huge amount of functionality with no additional
complexity if you work to set up namespace carefully.  The way ANTS
solves its target problems has some features in common with virtual
machines, but ANTS isn't intended as a replacement for VMs.  There are
problems VMs solve (full isolation/security) which are not the target
problems of ANTS.  Of course, there are also many features of the
namespace tools which aren't conceptually similar to VMs at all, such
as automated progressive backup and namespace rewriting, and these
solve other problems.  :)

> While one VM has no access to resources of another VM, the
> parent host or the "hypervisor" does need access to every VM
> to manage and connect up their resources as needed (for
> instance connect A and B's serial devices to each other).
> [Ideally this host is also virtualizable]

What you have described here is a major difference between ANTS and
vms.  Resource sharing between environments on the same machine (or on
different machines) in ANTS is done mostly through a 9pfs called
hubfs, which makes both shell execution and datapipes from other
machines available in the file namespace.  It connects machines in a
way that is a little bit like screen or tmux, but it works directly at
the level of the file descriptors (no TTY) and takes the form of a 9p
fs to integrate naturally into Plan 9.

So when the goal is sharing resources between environments, the extra
layer of the VM is often imposes some additional complexity or
performance penalty, in comparison to having the resources side by
side as processes wtih different namespaces.

The question "if you are trying to share everything, why are you
creating separate enviornments?" might occur to the reader, but the
answer is simple.  Different namespace groups and user environments
serve different purposes.  The main "new namespace" in ANTS is the
early boot ram/paq rooted namespace.  The reason this namespace exists
and has a different structure from the user namespace is that it has a
different purpose.  The service namespace exists to start and stop
services and administer the machine.  This is a different purpose than
the user namespace, which is about running interactive applications,
working with documents, etc.  Because the job of administering the
machine and connecting to the userspace fileserver is a different job
than the job of the userspace processes running in Acme (or another
program), it makes sense for their namespace to be structured in a way
that matches their purpose.

Thanks again for the chance to discuss with someone with a lot of
practical experience working with larger groups of machines.

Ben Kidwell
"mycroftiv"

- A minor note.  There is a tiny example of a jail in ANTS in the form
of the default profile for user none, which unmounts devices and rfork
m to prevent new attaches.  Sometimes I set up a telnet listener for
none using it.  In theory if you have a user profile like this and set
up an entry into this namespace, you can do lightweight "jails" in the
different ANTS namespaces.  This isn't really ANTS specific although
the fact that ANTS lets you create independent namespaces might make
these kind of RFNOMNT jails more useful.  I haven't actually played
with these things too much because I don't need to put myself in a
security jail when I'm the only user on my grid!  If someone is
interested in working on how ANTS could be used as part of a true
jail/security framework I'd certainly be interested in helping.

Re: [9fans] A note about new software for Plan 9

[Bakul Shah]

On Tue, 12 Mar 2013 10:57:53 EDT "Joel C. Salomon" <joelcsalomon@gmail.com> wrote:
> On Mon, Mar 11, 2013 at 11:52 PM, Bakul Shah <bakul@bitblocks.com> wrote:
> > To do something similar you will have to constrain each jail
> > to see a subset of processes, give it its own /dev, /env etc.
> > Not sure how you do this.
>
> So long as processes in the jail use /dev, /env, etc., etc., as
> inherited from/shared with their parent processes, this seems doable,
> if tedious: provide a synthetic file system that shows a limited view
> on /dev, /env, etc.
>
> But the child process can always mount #x for various x, and get out of jail.

The question really is, can we virtualize plan9 machines while
staying pretty much within the plan9 framework or by extending
it in a way consistent with its design?

To fully virtualize plan9 you'd have to virtualize kernel
services as well.  May be you'd've to dream up a ## service
that mediates access to hosts's #services!

While one VM has no access to resources of another VM, the
parent host or the "hypervisor" does need access to every VM
to manage and connect up their resources as needed (for
instance connect A and B's serial devices to each other).
[Ideally this host is also virtualizable]

And it is more than limiting the view; you may have to
fabricate a new view! In the networkign example I previously
mentioned, we had to provide virtual interfaces of various
kinds for each VM where the corresponding physical interfaces
didn't exist on the host.

You can of course do all this in Qemu/Vbox/VMware etc. but the
runtime cost is much higher. And you need to compile code into
these emulators if you want to simulate new devices. With
plan9 this could be much easier.

Re: [9fans] A note about new software for Plan 9

[erik quanstrom]

> But the child process can always mount #x for various x, and get out of jail.

not always.  from fork(2)

          RFNOMNT   If set, subsequent mounts into the new name space
                    and dereferencing of pathnames starting with # are
                    disallowed.

- erik

Re: [9fans] A note about new software for Plan 9

[Joel C. Salomon]

On Mon, Mar 11, 2013 at 11:52 PM, Bakul Shah <bakul@bitblocks.com> wrote:
> To do something similar you will have to constrain each jail
> to see a subset of processes, give it its own /dev, /env etc.
> Not sure how you do this.

So long as processes in the jail use /dev, /env, etc., etc., as
inherited from/shared with their parent processes, this seems doable,
if tedious: provide a synthetic file system that shows a limited view
on /dev, /env, etc.

But the child process can always mount #x for various x, and get out of jail.

—Joel

Re: [9fans] A note about new software for Plan 9

[sl]

I can see a use for this in re-inserting broken mounts
higher up in the proc tree.

-sl

Re: [9fans] A note about new software for Plan 9

[Bakul Shah]

On Tue, 12 Mar 2013 04:20:16 -0000 mycroftiv@sphericalharmony.com wrote:
>
> Do you think it would be helpful or welcome to re-post some of this
> material here on 9fans?

Your explanations are ok.  My guess is people don't have time
or sufficient motivation to dig deeper and read your papers.
Posting more stuff here won't help.

Motivation (for me at least) has to come from attempting to
solve "interesting" (i.e. practical, fun, &/or challenging)
problems.  Here are some examples (your ANTS may/may not be
relevant):

Jails solve specific problems (by isolating servers I reduce
exposure and can manage each service independently). In a
previous job we needed to simulate many independent virtual
hosts on each physical machine to test routing software.  We
used something similar to jail to simulate large IP networks.

Migrating a service to another machine without taking it down
so that a physical machines can be repaired/upgraded etc.

If you could bundle off a few services and move them onto a
new machine that can help with scaling during peak hours.

Controlled namespace sharing (Rangboom did this I think).
These have many uses -- screen sharing, something like google
drive or dropbox, collaborative work, etc.

Instantly bring up a brand new machine with nothing but pxe
(background installation! -- until a local FS is created and
needed files copied, you work off of a `staging' server
provided namespace).

Ability to have the same work environment and no loss of data
or context regardless of where I am in the world (and may be
not even on the same machine).

Ability to move a program closer to the data it operates on
(different pieces may be in different places).

Map N virtual machines on M physical machines -- you can do your
own virtual cluster!

Making clusters resilient against node failures.

Re: [9fans] A note about new software for Plan 9

[mycroftiv]

Bakul Shah wrote:
> Some motivating examples may help.  Perhaps you can show
> 1. Initial state of the world (show with the help of a few ls)
> 2. A sequence of commands to change it
> 3. What the world looks like finally

I have actually documented these things in this manner in the ANTS
paper that can be found at http://ants.9gridchan.org/doc/ants.ps and
it also overlaps with the tutorial material presented at
http//antfarm.9gridchan.org/tutorial and the related pages.

Do you think it would be helpful or welcome to re-post some of this
material here on 9fans?  I have tried to document all of these things
extensively as part of my work.

> And a brief description why this would be desirable.  And how
> it compares with just using chroot. Most people here can
> perhaps read rc code more readily than elaborate explanations!

Part of the comparison is that "chroot" does not exist in Plan 9, so
"rerootwin" is very similar to chroot in a conceptual sense.  The
desirability is simply that if you have multiple independent
namespaces, you need a mechanism to move freely between them.  The
"rerootwin" script and namespace file is really very simple.  After
checking and preparing the namespace, rerootwin does:

oldterm=oldterm.$pid
oldwsys=oldwsys.$pid
srvfs -p 0600 oldterm.$pid /mnt/term
srvfs -p 0600 oldwsys.$pid /mnt/wsys

And then does auth/newns into a custom parameterized namespace file
which has binds like this:

mount -c /srv/$oldterm $pivot/$oldterm
mount -c /srv/$oldwsys $pivot/$oldwsys
bind $pivot/$oldterm/dev/cons /dev/cons
bind $pivot/$oldterm/dev/consctl /dev/consctl

The reason you can't just use auth/newns and the standard namespace
file is that if you do this when you are connected via cpu or
drawterm, the standard namespace file will not re-bind the window
system and so you won't be able to run gui programs in the new
namespace.  Rerootwin "passes the devices through" via srvfs so you
can reroot to a new namespace but still start new GUI applications.

> In *BSD a jail is more than chroot since it can also get access
> to its own interfaces, networking stack, init process, devices
> etc. And it can't look at the state of another jail or the
> host. And another host on the net doesn't even know it is
> talking to a jail.

Yes I absolutely understand this and I have tried to be careful to
emphasize that I am using "jails" as a way of understanding what I am
talking about.  What I am doing is not intended as full isolation, and
is not intended for security - I actually write about this in the faq
on the antfarm site.  Talking about jails/vms is just the only way I
know to express the idea of an independent group of processes which
act to create a user-environment.

In fact, the ANTS design is deliberately pursuing integration between
these seperate namespaces rather than isolation.  My goal is not to
"isolate" each environment - it is to create environments which are
independent, but share data freely and deliberately make use of
resources from the other environments.

So even though the core idea - "independent userspaces" - is similar,
the implementation and purpose is very different.  I am trying to make
the independent namespaces so that each namespace can serve a
different function or role within the network - not impose security
and isolation on users.  The most important separate namespace is the
kernel-device only namespace, which acts as an admin/launching layer
for the other user namespaces.

> To do something similar you will have to constrain each jail
> to see a subset of processes, give it its own /dev, /env etc.
> Not sure how you do this.

As explained above (just to be clear) I am deliberately not doing
this, because it is not the intended purpose.  It would be possible to
adapt the ideas of ANTS into a more jail like framework by using a lot
of RFNOMNT to isolate things and take other steps, but that isn't how
I use these tools.  I want independent namespaces so I can administer
my fossil and my venti "from beneath" so that if the fossil or venti
need to be stopped, I still have all my needed tools available.  I
want independent namespaces so I can have 9front and Bell labs at the
same time and actually share namespace between the two as needed.

In fact one of the major ideas in ANTS is using a 9p fs called "hubfs"
to deliberately create persistent data connections between these
independent namespaces.  In the setup I use, each of my independent
namespaces has access to persistent shells in every other namespace
and every other machine - so the architecture is literally the exact
opposite of "security and isolation" it is fluid data sharing between
independent userspaces.

I hope these answers have been relevant and that my tone is friendly
and informative.  :) Your questions are very appropriate and
insightful, and I have made an attempt to extensively answer them in
the documentation I have created.  I have a full length paper, full
standard Plan 9 manpages, additional textual documentation within the
ants software folder, and five step-by-step tutorials done with
pre-installed VMs which attempt to comprehensively explain how to use
ANTS, how it is built, and what its purposes are.

I really appreciate the interest!  I hope I am responding and
explaining in a clear way

Ben Kidwell
"mycroftiv"

Re: [9fans] A note about new software for Plan 9

[Bakul Shah]

Just a suggestion:

Some motivating examples may help.  Perhaps you can show
1. Initial state of the world (show with the help of a few ls)
2. A sequence of commands to change it
3. What the world looks like finally

And a brief description why this would be desirable.  And how
it compares with just using chroot. Most people here can
perhaps read rc code more readily than elaborate explanations!

In *BSD a jail is more than chroot since it can also get access
to its own interfaces, networking stack, init process, devices
etc. And it can't look at the state of another jail or the
host. And another host on the net doesn't even know it is
talking to a jail.

To do something similar you will have to constrain each jail
to see a subset of processes, give it its own /dev, /env etc.
Not sure how you do this.

Re: [9fans] A note about new software for Plan 9

[mycroftiv]

erik quanstrom wrote:
> this part is confusing.  plan 9 has always maintained 1 independent
> namespace per process group.  so i'm surethat there's more to it
> than this.  could you go into detail on this?

You are right I was using terminology loosely here.  What I mean by
"namespace" in this context is the idea of a group of processes which
share a common namespace, and combine to create a user environment.
In other words, in a normal Plan 9 system, most of the processes are
aligned with the same root file system that the kernel acquired at
boot, and I'm loosely referring to this common root as "the namespace."

When I talk about running "multiple independent namespaces" with ANTS,
what I mean is using per-process namespace to do something similar to
BSD jails or a virtual machine.  The same Plan 9 kernel and machine
can have groups of processes running with different root filesystems.
If you create a group of processes that share a common root
filesystem, you have a user environment.

One thing that ANTS can do is to host multiple "namespace groups" on a
single machine.  Instead of having only one global namespace on the
machine with a single common root, you can host independent user
environments, each with a different root filesystem.

I know you (and others) are busy with plenty of obligations, but the
demos I have prepared with qemu vm images are designed to show in
practice how you can make multiple independent user environments on a
single plan 9 machine, then freely re-root which namespace you are
using.

So this part of ANTS is: running multiple userpsace environments on
the same machine.  You can do this because per-process namespaces
means that you can have a totally different root file system for one
group of processes than another.  To make this really work, the ANTS
9pcram kernel does not attach to a root filesystem itself.  The main
flow of control of the kernel "stands back" from the user's namespace
and does not attach to the same file descriptor that is the root of
the user fs.  This allows the main flow of control of the kernel to
act as a kind of "namespace hypervisor" that can launch multiple
indpendent user environments available at the same time on the same
machine.

To allow the user to actualy move between the different namespaces
freely, I have a script and parameterized namespace file called
"rerootwin" that acts as an instant Plan 9 chroot into a new
filesystem.  It avoids losing connection to the active window system
by passing the devices through a srvfs.  This means that even though
you might have Bell Labs and 9front running on the same box at the
same time, you can move back and forth at will, and enter either one
from a lower-layer kernel-only namespace.

This explanation was already kind of long and elaborate, but as I
mentioned I have virtual machine images and demo walkthroughs so you
can see how it actually works in practice, which is actually simpler
than the verbal description.

I'm not too good at balancing being precise and specific with keeping
things brief, so I hope this was a good answer without too much
unnecessary length.

Ben Kidwell
"mycroftiv"