Re: [9fans] A note about new software for Plan 9

[mycroftiv@sphericalharmony.com]

Bakul Shah wrote:
> The question really is, can we virtualize plan9 machines while
> staying pretty much within the plan9 framework or by extending
> it in a way consistent with its design?
>
> To fully virtualize plan9 you'd have to virtualize kernel
> services as well.  May be you'd've to dream up a ## service
> that mediates access to hosts's #services!

Thanks again for your comments.  From your description of past
projects, I see you have professional experience in related domains,
so as a home user without a professional background, I'm glad to learn
more.  I will talk a little bit about what ANTS does right now in
relation to full virtualization/jailing.

As you point out, there is a big difference between per-process
namespaces and fully virtualized kernel devices.  The ANTS software
makes the choice to focus on what can be done right now with the
existing Plan 9 kernel with minimal changes.  I think there has been
previous discussion in years past of making multiple verions of the
kernel #devices available, and also some ideas about letting the Plan
9 kernel run as a userspace process.  If you had the ability to
synthesize kernel devices on demand and also run multiple kernels as
userspace processes, I think that would be pretty close to the
full-scale technically sophisticated general approach to Plan 9
jails/pseudo VMs.  I think a project to do those things would be
interesting and it has been discussed before, but so far as I know, it
hasn't been implemented in practice.

In relation to that approach, ANTS is very different, and the reason
for this is what you talked about in your first reply - what problems
ANTS is trying to solve.  In my use and testing, I discoved that
solving the problems that were important to me didn't require those
mechanisms.  The problems I was working on turned out to be solvable
just by using the existing Plan 9 namespace framework in a new way.
ANTS is designed to help administer venti/fossil/cpu grids by creating
an independent namespace that doesn't depend on those services.  Doing
this has the additional benefit that you can easily create independent
sub-namespaces that can be used for some of the same things that VMs
or jails can.

Because I am a home user using Plan 9 on a grid of machines in my
basement, the problems that I have been focused on solving are related
to making a home grid easy to create and administer, and letting me
blend different forms of Plan 9 like Labs and 9front and imported unix
resources more freely.  I think that in the larger and more
professional context, the problems and solutions are different.
Profesionally, you are very concerned with making sure that customer
A's data doesn't get mixed up with customer B's data so the solution
is to provide full isolation of the environments.

On my home grid, customer A and customer B are both me, and I want to
have access to everything at once.  I want an independent namespace
that I can use to administer my grid even if I need to reboot some
nodes, and I want to use Labs, 9front, p9p, inferno all in a mixed
environment.  My problem and needed solution is how I create and
manage all these different namespaces but also be able to move data
between them and have as much sharing as I want, and as much
separation as I need.

What I have found - and what ANTS is based on - is that even limited
to nothing but per-process namespaces and the existing Plan 9
framework, you get a huge amount of functionality with no additional
complexity if you work to set up namespace carefully.  The way ANTS
solves its target problems has some features in common with virtual
machines, but ANTS isn't intended as a replacement for VMs.  There are
problems VMs solve (full isolation/security) which are not the target
problems of ANTS.  Of course, there are also many features of the
namespace tools which aren't conceptually similar to VMs at all, such
as automated progressive backup and namespace rewriting, and these
solve other problems.  :)

> While one VM has no access to resources of another VM, the
> parent host or the "hypervisor" does need access to every VM
> to manage and connect up their resources as needed (for
> instance connect A and B's serial devices to each other).
> [Ideally this host is also virtualizable]

What you have described here is a major difference between ANTS and
vms.  Resource sharing between environments on the same machine (or on
different machines) in ANTS is done mostly through a 9pfs called
hubfs, which makes both shell execution and datapipes from other
machines available in the file namespace.  It connects machines in a
way that is a little bit like screen or tmux, but it works directly at
the level of the file descriptors (no TTY) and takes the form of a 9p
fs to integrate naturally into Plan 9.

So when the goal is sharing resources between environments, the extra
layer of the VM is often imposes some additional complexity or
performance penalty, in comparison to having the resources side by
side as processes wtih different namespaces.

The question "if you are trying to share everything, why are you
creating separate enviornments?" might occur to the reader, but the
answer is simple.  Different namespace groups and user environments
serve different purposes.  The main "new namespace" in ANTS is the
early boot ram/paq rooted namespace.  The reason this namespace exists
and has a different structure from the user namespace is that it has a
different purpose.  The service namespace exists to start and stop
services and administer the machine.  This is a different purpose than
the user namespace, which is about running interactive applications,
working with documents, etc.  Because the job of administering the
machine and connecting to the userspace fileserver is a different job
than the job of the userspace processes running in Acme (or another
program), it makes sense for their namespace to be structured in a way
that matches their purpose.

Thanks again for the chance to discuss with someone with a lot of
practical experience working with larger groups of machines.

Ben Kidwell
"mycroftiv"

- A minor note.  There is a tiny example of a jail in ANTS in the form
of the default profile for user none, which unmounts devices and rfork
m to prevent new attaches.  Sometimes I set up a telnet listener for
none using it.  In theory if you have a user profile like this and set
up an entry into this namespace, you can do lightweight "jails" in the
different ANTS namespaces.  This isn't really ANTS specific although
the fact that ANTS lets you create independent namespaces might make
these kind of RFNOMNT jails more useful.  I haven't actually played
with these things too much because I don't need to put myself in a
security jail when I'm the only user on my grid!  If someone is
interested in working on how ANTS could be used as part of a true
jail/security framework I'd certainly be interested in helping.