[9fans] package system for Plan 9: alpha!

[9fans] package system for Plan 9: alpha!

[ron minnich]

Here is a refinement of fgb's fine work on a contrib system. I have
taken his ideas a bit further, based on my use of his tools in an
unreliable environment. I was getting quite frustrated as I had
multiple failures in the midst of an install, and seeing the message
'xyz already installed', when it was only 1/2 installed, was wasting
time. Also, I'm just not patient enough to wait for replica to do its
work.

While I think both replica and the contrib system are quite capable,
each in their own way, I felt they were lacking for my purposes. I
have become very impressed with how tinycore linux does binary
packages. I decided to enumerate what I like about that system, and
based on that, what I'd like to have on Plan 9. Not all goals are met
however.

1. reconstitute root file system on boot, in ram, then mount packages
as file systems, so basic root remains pristine
2. fast -- listing and dependencies should take well under a second;
package install of even big things should be under 3 minutes.
3. easy to list package dependencies quickly
4. auto-install of a package and its dependencies
5. separate package download from install; hence download can proceed
in parallel (not really in tinycore, just possible)
6. know what packages are installed quickly and easily
7. easy to remove a package; just remove one file, reboot, it's gone (see 1)
   *note*: when your system boots in 10 seconds, reboot is not that big a deal
8. no false positive: don't think a package is installed when it is
not or is only partially done (due to failure for example)
9. false negatives are ok: if a package is installed but you don't
think it is, reinstallation should be cheap
10. No need for continuous tinkering of db or other files to keep it
working correctly.
11. Works well in a high latency, even if high bandwidth, network.

Which of these goals does replica meet, e.g. for /sys/src?
>From my point of view, none of them.

Which of these goals does the current contrib system meet?
Much as I like the contrib system, it still depends on replica, so,
from my point of view, none of the goals are met.
I once saw it take four hours to install openssl. That's just not workable.

Which of these goals does the gui-based contrib system meet?
This is the system that downloads .iso files and then runs replica
against them. It meets 3, to some extent, but is still too slow for
me; it sort of meets 6; but, unfortunately, it fails on 8.

Which of these goals does my extension meet?
2 -- can download/install all of hg, including all dependencies, in 3
minutes, 2 of which are hget
3 -- .1 seconds for 'deps hg'; .1 second for list packages; < .1
second for list installed
4 -- it knows the dependencies and will install everything with one command
5 -- get and install are seperate commands
6 -- ls /installed
8 -- yes -- /installed/<name> is only created when the package is
completely installed (but there are bugs still)
9 -- yes
10. there is little in the way of a db file, just a /installed
directory (which you get by a bind -a)
11. It's far faster than existing systems because it uses hget

1. is obviously not yet met. I think it would be worth doing a tiny 9,
just as we have tinycore, for terminals.
7. is still not met. Package removal is still a mess. I had hoped to
just mount the .iso's and run the tools out of them but have not
figured out all the issues yet. A simple rbind failed to do the trick.

Here are some examples.

# available packages
term% time list
4th
8169
82563
9load-e820
9win
X11
abaco

(etc.)
0.00u 0.00s 0.11r 	 list

# what packages does hg need?
term% time deps hg
z
bz2
openssl
python
0.00u 0.00s 0.10r 	 deps hg

#install tiff
term% install tiff
package is tiff
Package z already installed, no need to do it
9660srv 1151: serving /srv/tiff
FINIS

#install tiff again
term% install tiff
package is tiff
Package z already installed, no need to do it
Package tiff already installed, no need to do it
FINIS
term%

Sources to these tools, including the build script, are at
http://9grid.net/magic/webls?dir=/rminnich/src/package-tools

You can try them out -- it's all there. Packages are in
http://9grid.net/magic/webls?dir=/rminnich/src/package

I don't pretend the scripts are very good, they just represent a
starting point. Experience (mine) is that the system work well. For
example, just doing:

get openssh
install openssh

takes very little time and has worked reliably for me on 9vx.

And, since I installed hg earlier, openssh install skipped the openssh
install step. Left to the reader (or me in a bit): don't download iso
when the package is installed! -- but it's so fast I have not
bothered.

I'm able to install packages now without worrying about whether I will
be ready to disconnect my laptop and go home before the install is
done!

Next step, if this system is found to be useful, is to adapt fgb's gui program.

ron

Re: [9fans] package system for Plan 9: alpha!

[Federico G. Benavento]

hmm... this looks good, you are still using the .iso's
from the existing packages.

using hget makes it faster of course, you could
hget from http://plan9.bell-labs.com/sources/contrib
but that would mean the user/package thing...
centralization sometimes makes things easier.

kudos

On Sat, May 15, 2010 at 12:43 PM, ron minnich <rminnich@gmail.com> wrote:
> Here is a refinement of fgb's fine work on a contrib system. I have
> taken his ideas a bit further, based on my use of his tools in an
> unreliable environment. I was getting quite frustrated as I had
> multiple failures in the midst of an install, and seeing the message
> 'xyz already installed', when it was only 1/2 installed, was wasting
> time. Also, I'm just not patient enough to wait for replica to do its
> work.
>
> While I think both replica and the contrib system are quite capable,
> each in their own way, I felt they were lacking for my purposes. I
> have become very impressed with how tinycore linux does binary
> packages. I decided to enumerate what I like about that system, and
> based on that, what I'd like to have on Plan 9. Not all goals are met
> however.
>
> 1. reconstitute root file system on boot, in ram, then mount packages
> as file systems, so basic root remains pristine
> 2. fast -- listing and dependencies should take well under a second;
> package install of even big things should be under 3 minutes.
> 3. easy to list package dependencies quickly
> 4. auto-install of a package and its dependencies
> 5. separate package download from install; hence download can proceed
> in parallel (not really in tinycore, just possible)
> 6. know what packages are installed quickly and easily
> 7. easy to remove a package; just remove one file, reboot, it's gone (see 1)
>   *note*: when your system boots in 10 seconds, reboot is not that big a deal
> 8. no false positive: don't think a package is installed when it is
> not or is only partially done (due to failure for example)
> 9. false negatives are ok: if a package is installed but you don't
> think it is, reinstallation should be cheap
> 10. No need for continuous tinkering of db or other files to keep it
> working correctly.
> 11. Works well in a high latency, even if high bandwidth, network.
>
> Which of these goals does replica meet, e.g. for /sys/src?
> From my point of view, none of them.
>
> Which of these goals does the current contrib system meet?
> Much as I like the contrib system, it still depends on replica, so,
> from my point of view, none of the goals are met.
> I once saw it take four hours to install openssl. That's just not workable.
>
> Which of these goals does the gui-based contrib system meet?
> This is the system that downloads .iso files and then runs replica
> against them. It meets 3, to some extent, but is still too slow for
> me; it sort of meets 6; but, unfortunately, it fails on 8.
>
> Which of these goals does my extension meet?
> 2 -- can download/install all of hg, including all dependencies, in 3
> minutes, 2 of which are hget
> 3 -- .1 seconds for 'deps hg'; .1 second for list packages; < .1
> second for list installed
> 4 -- it knows the dependencies and will install everything with one command
> 5 -- get and install are seperate commands
> 6 -- ls /installed
> 8 -- yes -- /installed/<name> is only created when the package is
> completely installed (but there are bugs still)
> 9 -- yes
> 10. there is little in the way of a db file, just a /installed
> directory (which you get by a bind -a)
> 11. It's far faster than existing systems because it uses hget
>
> 1. is obviously not yet met. I think it would be worth doing a tiny 9,
> just as we have tinycore, for terminals.
> 7. is still not met. Package removal is still a mess. I had hoped to
> just mount the .iso's and run the tools out of them but have not
> figured out all the issues yet. A simple rbind failed to do the trick.
>
> Here are some examples.
>
> # available packages
> term% time list
> 4th
> 8169
> 82563
> 9load-e820
> 9win
> X11
> abaco
>
> (etc.)
> 0.00u 0.00s 0.11r        list
>
> # what packages does hg need?
> term% time deps hg
> z
> bz2
> openssl
> python
> 0.00u 0.00s 0.10r        deps hg
>
> #install tiff
> term% install tiff
> package is tiff
> Package z already installed, no need to do it
> 9660srv 1151: serving /srv/tiff
> FINIS
>
> #install tiff again
> term% install tiff
> package is tiff
> Package z already installed, no need to do it
> Package tiff already installed, no need to do it
> FINIS
> term%
>
> Sources to these tools, including the build script, are at
> http://9grid.net/magic/webls?dir=/rminnich/src/package-tools
>
> You can try them out -- it's all there. Packages are in
> http://9grid.net/magic/webls?dir=/rminnich/src/package
>
> I don't pretend the scripts are very good, they just represent a
> starting point. Experience (mine) is that the system work well. For
> example, just doing:
>
> get openssh
> install openssh
>
> takes very little time and has worked reliably for me on 9vx.
>
> And, since I installed hg earlier, openssh install skipped the openssh
> install step. Left to the reader (or me in a bit): don't download iso
> when the package is installed! -- but it's so fast I have not
> bothered.
>
> I'm able to install packages now without worrying about whether I will
> be ready to disconnect my laptop and go home before the install is
> done!
>
> Next step, if this system is found to be useful, is to adapt fgb's gui program.
>
> ron
>
>



-- 
Federico G. Benavento

Re: [9fans] package system for Plan 9: alpha!

[ron minnich]

On Sat, May 15, 2010 at 7:56 PM, Federico G. Benavento
<benavento@gmail.com> wrote:
> hmm... this looks good, you are still using the .iso's
> from the existing packages.

not quite. I actually recreate all the .iso's from the proto files in
replica/<package>/proto.

The reason is that in some cases, the root/ directories for some
contributors contain multiple packages, and I wanted to have one .iso
per package.

> using hget makes it faster of course, you could
> hget from http://plan9.bell-labs.com/sources/contrib
> but that would mean the user/package thing...
> centralization sometimes makes things easier.

This is kind of a test idea, I decided not to clutter up plan9 site
with it. It may make sense to just leave it at sources however. It's
easy to. In fact it's there now:
contrib/rminnich/package
has it all.

9grid.net is on esnet, which is an OC192, which means it has lots of
bandwidth for users ...

but all the .iso.bz2 in the packages directory originates from sources.

thanks

ron

Re: [9fans] package system for Plan 9: alpha!

[ron minnich]

OK I double-checked and the 'install' script does indeed create
/installed/package-name.

So, if you have bound something onto /installed, or the directory
exists, you should be fine.

fgb has suggested improvements to my BUILD script, which I will put in
this week.

Any other suggestions are most welcome. I'd like to make this useful
to as many people as I can.

One misconception I ought to clear up: I do NOT use a ram-based root
at present. In fact I'm actually using this all on 9vx, though I
expect to start using it on mbgokhale.org soon. All packages get
installed into / just as they do with the existing contrib tool. This
new set of tools just makes it easy and quick to install the contrib
packages, and updating said packages is very easy.

Between this package tool and mercurial for sources, I don't expect to
ever need to run replica again.

Thanks

ron

Re: [9fans] package system for Plan 9: alpha!

[Akshat Kumar]

On 5/16/10, ron minnich <rminnich@gmail.com> wrote:
> Between this package tool and mercurial for sources, I don't expect to
> ever need to run replica again.

How do you plan to keep up with updates
to contrib repos people keep on sources?
Will the contrib -> iso.bz2 conversion take
place on a nightly basis? Some sort of
keeping tabs on modification times?

Perhaps it would be nice if people with
contrib repos could themselves make the
ISOs and upload to 9grid or put it wherever
the central iso.bz2 repo will be.


Best,
ak

Re: [9fans] package system for Plan 9: alpha!

[ron minnich]

On Sun, May 16, 2010 at 6:47 PM, Akshat Kumar
<akumar@mail.nanosouffle.net> wrote:
> On 5/16/10, ron minnich <rminnich@gmail.com> wrote:
>> Between this package tool and mercurial for sources, I don't expect to
>> ever need to run replica again.
>
> How do you plan to keep up with updates
> to contrib repos people keep on sources?

the BUILD tool rebuilds it all on demand.

> Will the contrib -> iso.bz2 conversion take
> place on a nightly basis? Some sort of
> keeping tabs on modification times?

it's actually reasonably fast, rebuilding all the files for all the
packages takes  < 10 minutes, and that's without any parallelism in
the creation ... so it's not a big deal to rerun this every night.

Total size of a tar transfer is 500 mbytes or so, and when I use rx
that also is done quickly.

>
> Perhaps it would be nice if people with
> contrib repos could themselves make the
> ISOs and upload to 9grid or put it wherever
> the central iso.bz2 repo will be.

Yes, if we decide we like this format we can do that.

The build tool does deal with ambiguities and issues:

- contrib names things as name/package or package (well people do),
which I have to handle.
- Also, people mix one or more projects in their root/ directories, so
I use the replica/package/proto files to build the .iso.bz2.

Really, if people do no more than what they do now, that's fine for me
to keep building this set of files. It's really just a script.

So I run a script to keep the bitbucket sysfromiso repo up to date,
usually daily, and I can do that with this script as well.

ron