[9fans] Regarding venti block size

[9fans] Regarding venti block size

[anooop.anooop]

Hello All,

I want to store some big files on venti.  size range around 200MB.  Is
there any way to do this?

This is what I get when I try to store files of size around 300KB :
"write: input too big: max block size is 57344"

I was wondering may be if increase the block size in arena partition
this can be done. so I tried to format arena with block size 200M but
it gave the error below.

"anoop@allatoona:/usr/local/plan9/bin$ venti/fmtarenas -a 3800M -b
200M  arenas0. /dev/sda6 &
[1] 32604
anoop@allatoona:/usr/local/plan9/bin$ block size too large, max 65536"

Any help is greatly appreciated.

Thanks.

Regards,
Anoop

Re: [9fans] Regarding venti block size

[matt]

anooop.anooop@gmail.com wrote:
> Hello All,
>
> I want to store some big files on venti.  size range around 200MB.  Is
> there any way to do this?
>
> This is what I get when I try to store files of size around 300KB :
> "write: input too big: max block size is 57344"
>
> I was wondering may be if increase the block size in arena partition
> this can be done. so I tried to format arena with block size 200M but
> it gave the error below.
>
> "anoop@allatoona:/usr/local/plan9/bin$ venti/fmtarenas -a 3800M -b
> 200M  arenas0. /dev/sda6 &
> [1] 32604
> anoop@allatoona:/usr/local/plan9/bin$ block size too large, max 65536"
>

why are you choosing such a big block size ?

Blocks size and file size are related such that ceil(filesize /
blocksize) = numberofblocks

The larger the blocksize the less likely you are of coalescing any (or
that's what my intuition says). Sure, there's a tradeoff but the
research has been done already and < 64k blocks looks to have been found
appropriate.


If your file is 300Kb and your blocksize 200M then every distinct 300kB
file will occupy 200Mb of actual disk.


Also, 200Mb is not a big file!!

Re: [9fans] Regarding venti block size

[erik quanstrom]

> The larger the blocksize the less likely you are of coalescing any (or
> that's what my intuition says). Sure, there's a tradeoff but the
> research has been done already and < 64k blocks looks to have been found
> appropriate.

what's the goal?  the absolute smallest repository
or access speed?  sharing means seeks.  seeks are
deathly slow if you're using disks.

there was a time when there was a compelling
argument for reducing the number of i/o operations
for performance by increasing the block size.
i'm not sure that argument is as strong as it once
was.

- erik

Re: [9fans] Regarding venti block size

[erik quanstrom]

> This is what I get when I try to store files of size around 300KB :
> "write: input too big: max block size is 57344"

this is why god^wken used filesystems.

try vac(1), fossil(4) or if you need something
esoteric you can roll your own filesystem with
venti-file(2).

- erik

Re: [9fans] Regarding venti block size

[anooop.anooop]

On Feb 16, 9:32�am, quans...@quanstro.net (erik quanstrom) wrote:
> > This is what I get when I try to store files of size around 300KB :
> > "write: input too big: max block size is 57344"
>
> this is why god^wken used filesystems.
>
> try vac(1), fossil(4) or if you need something
> esoteric you can roll your own filesystem with
> venti-file(2).
>
> - erik

Thank you so much for the replies.

This is what I am planning to do.

1. take a file, split it into many different parts (each part is
char*)
2. store each part on a venti server and collect the scores
3. when I need the file back, provide the scores to the venti and get
the parts back
4. merge the parts and create the file back

The problem is that I was planning on keeping the number of parts is
fixed. So depending on the file size, the part size will vary. And
therefore sometimes the (part = block) that I store will be more than
the current upper limit of 64k.

It would be great if you could let me know your thoughts on this.

Regards,
Anoop

Re: [9fans] Regarding venti block size

[matt]

>
> This is what I am planning to do.
>
>
>
I have a few wheel designs here in my basement too,

Re: [9fans] Regarding venti block size

[Russ Cox]

> The problem is that I was planning on keeping the number of parts is
> fixed. So depending on the file size, the part size will vary. And
> therefore sometimes the (part  block) that I store will be more than
> the current upper limit of 64k.
>
> It would be great if you could let me know your thoughts on this.

I think you will need to rewrite both venti and the
venti protocol to get around the 64k limit.
I started to do this once and decided it was
too large a change to bother.  If you don't care
about coalescing duplicate blocks (and you
don't seem to), why not use an extent-based
file system instead?

Russ

Re: [9fans] Regarding venti block size

[erik quanstrom]

> I think you will need to rewrite both venti and the
> venti protocol to get around the 64k limit.
> I started to do this once and decided it was
> too large a change to bother.  If you don't care
> about coalescing duplicate blocks (and you
> don't seem to), why not use an extent-based
> file system instead?

i guess i'm not up to speed yet.  what is the advantage
of supporting blocks of different sizes?

- erik

Re: [9fans] Regarding venti block size

[roger peppe]

2009/2/16  <anooop.anooop@gmail.com>:
> I want to store some big files on venti.  size range around 200MB.  Is
> there any way to do this?

maybe i'm being stupid, but isn't that precisely what the
vac format (as documented in http://www.cs.bell-labs.com/sys/doc/fossil.pdf)
and its block tree structure is there for?

Re: [9fans] Regarding venti block size

[anooop.anooop]

On Feb 16, 11:20�am, mattmob...@proweb.co.uk (matt) wrote:

> I have a few wheel designs here in my basement too,

did you mean that I am trying to re-invent the wheel?  :-)
Please do let me know if there is already some functionality to store
and retrieve a single file without breaking up the file and calling
the "write" client for the parts. I am new to plan9 and venti and an
not fully aware of all the available clients for venti.
I was reading up on vac and vbackup but they talk about backing up
file system trees and disks which is not what I am looking for. I just
need to store away one single file or block of data,  get a score and
later use the score to get the file back.

~Anoop

Re: [9fans] Regarding venti block size

[lucio]

> I was reading up on vac and vbackup but they talk about backing up
> file system trees and disks which is not what I am looking for.

People are happy to use zip to compress (and archive, but who cares?)
a single file, it can't be much different to use vac in the same vein?

++L

Re: [9fans] Regarding venti block size

[ron minnich]

>> I was reading up on vac and vbackup but they talk about backing up
>> file system trees and disks which is not what I am looking for.


I don't get this comment. You want to backup a file, right? What about
'file system tree' is incompatible or not what you are looking for?

rno

Re: [9fans] Regarding venti block size

[matt]

I don't get why you're dicking around with block sizes but I guess it's
just learning :)

If these functions don't allow you to create and read/write files in a
venti then my current project is screwed - it compiles but I haven't
started testing it!

venti-file(2)
venti-cache(2)

vtdial()
vtcachealloc()
vtfileopenroot()
vtfilecreateroot()
vtfilecreate()
vtfileopen()
vtfilegetentry()
vtfileread()
vtfilewrite()
vthangup()
vtfreeconn()
>

Re: [9fans] Regarding venti block size

[anooop.anooop]

> i guess i'm not up to speed yet.  what is the advantage
> of supporting blocks of different sizes?
>
> - erik

I am doing a part of a bigger project which involves splitting files
into many pieces and storing them separately for security reasons. I
thought of using each piece as a block of data to be stored on a venti
server. But since the number of parts a file will be split into need
to be constant the block size varies. But may be this is not the right
way to do it. I am now re-considering splitting files into number of
pieces = filesize/block size.

>  If you don't care
> about coalescing duplicate blocks (and you
> don't seem to), why not use an extent-based
> file system instead?
>
> Russ

It is necessary for me to avoid duplicate blocks and hence it doesnt
look like other alternatives will work here.

> maybe i'm being stupid, but isn't that precisely what the
> vac format (as documented inhttp://www.cs.bell-labs.com/sys/doc/fossil.pdf)
> and its block tree structure is there for?

I have to learn more about vac before I say anything about this. I
will get back after catching up. but what I understand is vac backs up
file system trees on the server. I am on linux and using plan9port.And
I am using raw partitions for arena, index and bloom filter and do not
have a fossil fs. If vac can store a single file and retrieve it, then
I will be able to use this to store the parts of the original file.

I was using "venti/write -h 127.0.0.1 < datablock" to store file parts
until I hit the upper limit when I started using "comparatively"
bigger files which gave bigger than 64k parts.

~Anoop

Re: [9fans] Regarding venti block size

[anooop.anooop]

On Feb 16, 2:26�pm, rminn...@gmail.com (ron minnich) wrote:
> >> I was reading up on vac and vbackup but they talk about backing up
> >> file system trees and disks which is not what I am looking for.
>
> I don't get this comment. You want to backup a file, right? What about
> 'file system tree' is incompatible or not what you are looking for?
>
> rno

Vac is what I need. I got confused by the vac manpage into thinking
vac takes snapshots of file sytem.
If it was not for you guys I would have tried to implement my own
version of vac. :-)
Thank you to all.

Regards,
Anoop

Re: [9fans] Regarding venti block size

[erik quanstrom]

> > i guess i'm not up to speed yet.  what is the advantage
> > of supporting blocks of different sizes?
> >
> > - erik
>
> I am doing a part of a bigger project which involves splitting files
> into many pieces and storing them separately for security reasons. I
> thought of using each piece as a block of data to be stored on a venti
> server. But since the number of parts a file will be split into need
> to be constant the block size varies. But may be this is not the right
> way to do it. I am now re-considering splitting files into number of
> pieces = filesize/block size.

i ment that question to be aimed at
venti.

- erik