[9fans] command line tool for storing / reading files on venti

[9fans] command line tool for storing / reading files on venti

[Enrico Weigelt]

Hi folks,


is there any command line tool for reading and writing files
on venti ?

The venti/read and venti/write commands just support single blocks.
Vac seems fine as archive tool, but it only can store - I need some
"unvac" command. (mounting each single archive via vacfs is a bit
too complicated for my project).


cu
--
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [9fans] command line tool for storing / reading files on venti

[Pietro Gagliardi]

9fs dump
command /n/dump/yyyy/mmdd/absolutepathtofile

yyyy -> year
mmdd -> month and date

On Jun 13, 2008, at 3:50 PM, Enrico Weigelt wrote:

>
> Hi folks,
>
>
> is there any command line tool for reading and writing files
> on venti ?
>
> The venti/read and venti/write commands just support single blocks.
> Vac seems fine as archive tool, but it only can store - I need some
> "unvac" command. (mounting each single archive via vacfs is a bit
> too complicated for my project).
>
>
> cu
> --
> ----------------------------------------------------------------------
> Enrico Weigelt, metux IT service -- http://www.metux.de/
>
> cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
> ----------------------------------------------------------------------
> Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
> ----------------------------------------------------------------------
>

Re: [9fans] command line tool for storing / reading files on venti

[a]

Since you mention using vac for storage, I assume you're using venti
directly, not via fossil, in which case the '9fs dump' suggestion will
do nothing for you.

I don't believe there is anything in Plan 9 that does what you want
(certainly the BUGS section in venti(1) implies not), but you might
look at vcat in p9p. It's not quite there - it seems to demand that
you're working with only a single file, rather than directory trees
(possibly containing only a single file) as vac creates - but does the
redirection and may be close to what you want.
Anthony

Re: [9fans] command line tool for storing / reading files on venti

[Enrico Weigelt]

* Pietro Gagliardi <pietro10@mac.com> wrote:
> 9fs dump
> command /n/dump/yyyy/mmdd/absolutepathtofile

but this still requires me to mount each single vac
archive before reading, and I need to create a new one
for each upload, right ?


cu
--
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [9fans] command line tool for storing / reading files on venti

[Pietro Gagliardi]

I don't think so. 9fs dump gives you everything afaik.

On Jun 13, 2008, at 4:29 PM, Enrico Weigelt wrote:

> * Pietro Gagliardi <pietro10@mac.com> wrote:
>> 9fs dump
>> command /n/dump/yyyy/mmdd/absolutepathtofile
>
> but this still requires me to mount each single vac
> archive before reading, and I need to create a new one
> for each upload, right ?
>
>
> cu
> --
> ----------------------------------------------------------------------
> Enrico Weigelt, metux IT service -- http://www.metux.de/
>
> cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
> ----------------------------------------------------------------------
> Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
> ----------------------------------------------------------------------
>

Re: [9fans] command line tool for storing / reading files on venti

[Russ Cox]

> The venti/read and venti/write commands just support single blocks.
> Vac seems fine as archive tool, but it only can store - I need some
> "unvac" command. (mounting each single archive via vacfs is a bit
> too complicated for my project).

I'm hesitant to bother replying, as we learned earlier that
your idea of complicated does not match established norms.

That said, here is an (untested) unvac command:

	#!/bin/rc
	if(! ~ $#* 2){
		echo 'usage: unvac file.vac directory'
		exit 1
	}
	rfork n
	vacfs $1 || exit
	mkdir -p $2 || exit
	dircp /n/vac $2

Russ

Re: [9fans] command line tool for storing / reading files on venti

[Enrico Weigelt]

* a@9srv.net <a@9srv.net> wrote:
> Since you mention using vac for storage, I assume you're using venti
> directly, not via fossil, in which case the '9fs dump' suggestion will
> do nothing for you.

ACK.

> I don't believe there is anything in Plan 9 that does what you want
> (certainly the BUGS section in venti(1) implies not), but you might
> look at vcat in p9p. It's not quite there - it seems to demand that
> you're working with only a single file, rather than directory trees
> (possibly containing only a single file) as vac creates - but does the
> redirection and may be close to what you want.

Yes, that sounds good, but fails with:

    ./vcat: diskopenventi: not a single file

Well, if I now just find the opposite - uploading a file -
I'll be happy :)


cu
--
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [9fans] command line tool for storing / reading files on venti

[Enrico Weigelt]

* Russ Cox <rsc@swtch.com> wrote:
> > The venti/read and venti/write commands just support single blocks.
> > Vac seems fine as archive tool, but it only can store - I need some
> > "unvac" command. (mounting each single archive via vacfs is a bit
> > too complicated for my project).
>
> I'm hesitant to bother replying, as we learned earlier that
> your idea of complicated does not match established norms.
>
> That said, here is an (untested) unvac command:
>
> 	#!/bin/rc
> 	if(! ~ $#* 2){
> 		echo 'usage: unvac file.vac directory'
> 		exit 1
> 	}
> 	rfork n
> 	vacfs $1 || exit
> 	mkdir -p $2 || exit
> 	dircp /n/vac $2

Well, this doesnt seem to run this way on p9p :(
BTW: still leaves the problem that lot's of vacfs servers are
started not terminated after use :(


cu
--
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [9fans] command line tool for storing / reading files on venti

[Russ Cox]

> BTW: still leaves the problem that lot's of vacfs servers are
> started not terminated after use :(

no, it doesn't.  on plan 9, when the shell script exits,
nothing will have vacfs mounted anymore, so vacfs
will get an eof on the 9p connection and exit.
that's why i put an rfork n in the script.

> Well, this doesnt seem to run this way on p9p :(

true.  i thought you were using plan 9, sorry.

if you are using linux and have fuse installed,
you can still run something like the script,
as long as you are willing to run just one at a time.

	vacfs -m /vac file.vac
	cp -R /vac/* .
	unmount /vac

would work, assuming you create a /vac that you
have write permission to.  you can only run one at a time
since there is only one /vac.

russ

Re: [9fans] command line tool for storing / reading files on venti

[Enrico Weigelt]

* Russ Cox <rsc@swtch.com> wrote:

Hi,

> no, it doesn't.  on plan 9, when the shell script exits,
> nothing will have vacfs mounted anymore, so vacfs
> will get an eof on the 9p connection and exit.
> that's why i put an rfork n in the script.

aha, that's probably what the rfork call does ?

> > Well, this doesnt seem to run this way on p9p :(
>
> true.  i thought you were using plan 9, sorry.

is there any way to get the same behaviour on p9p ?
maybe the shell could keep track of its childs and
terminate them on exit.

> if you are using linux and have fuse installed,
> you can still run something like the script,
> as long as you are willing to run just one at a time.

Uh, I'm not sure whether I can use fuse on the target system.
I'd prefer doing all in userland.

Meanwhile I've hacked up vacfs to give it an "autodie" mode.
It means: automatically terminate if the last file has been
closed. Seems to work, but now idea how stable it would be
in production ;-o

The whole gets a bit more complex:

I'm using venti as storage backend for an media library.
People can upload media from one server (via an Joomla-based
frontend) and later fetch them via another download server.
That download server is just an minimal httpd which fetches
by a given key from venti and sends back the result.

So I could some more things to do:

* store some additional metadata (at least mime-type)
* maybe also add some permission information
* when using vac, I have to use always the same filename
  within vac (ls'ing first would be event oo complex ;-o)


Maybe it's better to code a new archiver for this.


cu
--
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [9fans] command line tool for storing / reading files on venti

[Russ Cox]

>> no, it doesn't.  on plan 9, when the shell script exits,
>> nothing will have vacfs mounted anymore, so vacfs
>> will get an eof on the 9p connection and exit.
>> that's why i put an rfork n in the script.
>
> aha, that's probably what the rfork call does ?

rfork n forks the name space, so that any changes
made by the script don't propagate out to the
parent process.  This means that (1) vacfs can
mount on /n/vac without any fear of bothering
some other instance of the script, and (2) when the
script exits, there will be no more references to
the mounted /n/vac and the kernel will unmount it.

If you want simple, you should run Plan 9.

> Meanwhile I've hacked up vacfs to give it an "autodie" mode.
> It means: automatically terminate if the last file has been
> closed. Seems to work, but now idea how stable it would be
> in production ;-o

In the time it took you to do that you could probably
have written unvac instead.

Russ

Re: [9fans] command line tool for storing / reading files on venti

[Enrico Weigelt]

* Russ Cox <rsc@swtch.com> wrote:

> rfork n forks the name space, so that any changes
> made by the script don't propagate out to the
> parent process.  This means that (1) vacfs can
> mount on /n/vac without any fear of bothering
> some other instance of the script, and (2) when the
> script exits, there will be no more references to
> the mounted /n/vac and the kernel will unmount it.

Aha. But this still does not terminate the vacfs, right ?

> > Meanwhile I've hacked up vacfs to give it an "autodie" mode.
> > It means: automatically terminate if the last file has been
> > closed. Seems to work, but now idea how stable it would be
> > in production ;-o
>
> In the time it took you to do that you could probably
> have written unvac instead.

hmm, maybe ;-o

Meanwhile I've written an new tool which stores single files
(no directory stuff at all). Not perfect yet, but already
works quite fine (IMHO):

svn://nibiru.metux.de/public/plan9port/apps/vtstore/

In the next step, I'll add http-alike metadata (mimetype, etc).


cu
--
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [9fans] command line tool for storing / reading files on venti

[Pietro Gagliardi]

On Jun 15, 2008, at 12:02 PM, Enrico Weigelt wrote:

> Aha. But this still does not terminate the vacfs, right ?


Once you kill the script, you also kill the processes it created -
that's what process groups are, remember?

Please read "The Use of Name Spaces in Plan 9" and the rfork man page
and proc file system descriptions and all pages that are referenced
there BEFORE continuing.

Re: [9fans] command line tool for storing / reading files on venti

[Enrico Weigelt]

* Pietro Gagliardi <pietro10@mac.com> wrote:
> On Jun 15, 2008, at 12:02 PM, Enrico Weigelt wrote:
>
> >Aha. But this still does not terminate the vacfs, right ?
>
>
> Once you kill the script, you also kill the processes it created -
> that's what process groups are, remember?

That doesnt seem to work on p9p :(


cu
--
----------------------------------------------------------------------
 Enrico Weigelt, metux IT service -- http://www.metux.de/

 cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
----------------------------------------------------------------------
 Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
----------------------------------------------------------------------

Re: [9fans] command line tool for storing / reading files on venti

[erik quanstrom]

>> Aha. But this still does not terminate the vacfs, right ?
>
>
> Once you kill the script, you also kill the processes it created -
> that's what process groups are, remember?

incorrect.  the script exits, it is not killed.  for example

	; cat script
	#!/bin/rc
	rfork n
	sleep 1492&
	exit ''
	; script
	; ps -a|grep sleep
	quanstro       1519    0:00   0:00        8K Sleep    sleep 1492
	quanstro       1521    0:00   0:00      156K Pread    grep sleep

when the script exits, the channel is closed, since all
the writers are gone.  this means that when vacfs
read(2)'s from its 9p connection, it gets -1.  it is
conventional to exit at this point since no further
work may be done.

	; cat script
	#!/bin/rc
	rfork n
	ramfs
	exit ''
	; script
	; ps -a | grep ramfs
	quanstro       1860    0:00   0:00      156K Pread    grep ramfs

- erik

Re: [9fans] command line tool for storing / reading files on venti

[Russ Cox]

> hmm, maybe ;-o
>
> Meanwhile I've written an new tool which stores single files
> (no directory stuff at all). Not perfect yet, but already
> works quite fine (IMHO):
>
> svn://nibiru.metux.de/public/plan9port/apps/vtstore/
>
> In the next step, I'll add http-alike metadata (mimetype, etc).

I wasn't kidding.  Update your plan9port
and you will find an unvac command.
You'll get more leverage if you stick to vac files.
On the other hand, it won't be as complicated
as rolling your own format.

Russ

Re: [9fans] command line tool for storing / reading files on venti

[Pietro Gagliardi]

Doesn't matter. Process groups are process groups on any Unix clone.
If it's daemons you're dealing with, then it leaves the scope of this
room. If it's Windows, you're out of luck.

On Jun 15, 2008, at 1:13 PM, Enrico Weigelt wrote:

> * Pietro Gagliardi <pietro10@mac.com> wrote:
>> On Jun 15, 2008, at 12:02 PM, Enrico Weigelt wrote:
>>
>>> Aha. But this still does not terminate the vacfs, right ?
>>
>>
>> Once you kill the script, you also kill the processes it created -
>> that's what process groups are, remember?
>
> That doesnt seem to work on p9p :(
>
>
> cu
> --
> ----------------------------------------------------------------------
> Enrico Weigelt, metux IT service -- http://www.metux.de/
>
> cellphone: +49 174 7066481   email: info@metux.de   skype: nekrad666
> ----------------------------------------------------------------------
> Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme
> ----------------------------------------------------------------------
>

Re: [9fans] command line tool for storing / reading files on venti

[Enrico Weigelt]

* Pietro Gagliardi <pietro10@mac.com> wrote:
> Doesn't matter. Process groups are process groups on any Unix clone.
> If it's daemons you're dealing with, then it leaves the scope of this
> room. If it's Windows, you're out of luck.

I'm running p9p on Linux. No idea where the actual problem sits,
but your example left the vacfs-processes running.


cu
--
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: [9fans] command line tool for storing / reading files on venti

[Enrico Weigelt]

* Russ Cox <rsc@swtch.com> wrote:

Hi,

> I wasn't kidding.  Update your plan9port
> and you will find an unvac command.

my last checkout is a about 5 days old, and I couldn't find
some working unvac command. But I'll have a re-checkout.

> You'll get more leverage if you stick to vac files.

Maybe right, if'd use an vac dir per file to archive, I could
easily add more metadata later. But I'm very sure I don't
need that for my current project.

> On the other hand, it won't be as complicated
> as rolling your own format.

It wasn't that complicated, now all done ;-P


cu
--
---------------------------------------------------------------------
 Enrico Weigelt    ==   metux IT service - http://www.metux.de/
---------------------------------------------------------------------
 Please visit the OpenSource QM Taskforce:
 	http://wiki.metux.de/public/OpenSource_QM_Taskforce
 Patches / Fixes for a lot dozens of packages in dozens of versions:
	http://patches.metux.de/
---------------------------------------------------------------------

Re: [9fans] command line tool for storing / reading files on venti

[Venkatesh Srinivas]

Hi,

Inferno's vacfs lets you cd into a score and read files from there
without a mount per-score.

--vs

Re: [9fans] command line tool for storing / reading files on venti

[Steve Simon]

> Doesn't matter. Process groups are process groups on any Unix clone.
> If it's daemons you're dealing with, then it leaves the scope of this
> room. If it's Windows, you're out of luck.

Not true, Windows has the concept of process groups.

See the description of CREATE_NEW_PROCESS_GROUP at
http://msdn.microsoft.com/en-us/library/ms684863(VS.85).aspx

Admittedly this relates only to console applications, however
that is where it is most useful.

-Steve

Re: [9fans] command line tool for storing / reading files on venti

[ron minnich]

On Sun, Jun 15, 2008 at 4:58 PM, Enrico Weigelt <weigelt@metux.de> wrote:
> * Pietro Gagliardi <pietro10@mac.com> wrote:
>> Doesn't matter. Process groups are process groups on any Unix clone.
>> If it's daemons you're dealing with, then it leaves the scope of this
>> room. If it's Windows, you're out of luck.
>
> I'm running p9p on Linux. No idea where the actual problem sits,
> but your example left the vacfs-processes running.
>

yes but I don't understand why your response to what you think is a
problem is to write more code instead of figuring out what's going on.

I realize that's the current open source culture, but I don't know why.

ron