[9fans] seek() in fossil?

[9fans] seek() in fossil?

[andrey mirtchovski]

this article (linked from slashdot, sorry) on freshmeat discusses problems
with large files in unix, but i think at least one of them may be remotely
relevant to plan9, however small the issue may be, it may warrant some
discussion:

fossil has a maximum stream size (maximum file size, as i understand it) of
2^73 bytes, however the seek() system call accepts only a vlong (2^64) as an
argument.

the article can be found here:

	http://freshmeat.net/articles/view/709/

andrey

Re: [9fans] seek() in fossil?

[Russ Cox]

I think 2^63 is a perfectly fine limit for the
forseeable future.  Do you really have files
(or file systems) that big?

The only point of the sentence about the 2^73
hash tree limit was that the maximum depth
of 7 is more than enough.

Russ

Re: [9fans] seek() in fossil?

[David Presotto]

[-- Attachment #1: Type: text/plain, Size: 302 bytes --]

I don't know.  If I took I wanted to attach a GPS to
every person on earth and delta encode there movements
once a second for their whole lifetime, I could do that
as a single stream on venti (just barely).  On the other
hand, once they died the position data would probably
compress a lot better.

[-- Attachment #2: Type: message/rfc822, Size: 1811 bytes --]

From: "Russ Cox" <rsc@plan9.bell-labs.com>
To: 9fans@cse.psu.edu
Subject: Re: [9fans] seek() in fossil?
Date: Sun, 26 Jan 2003 14:01:15 -0500
Message-ID: <3d1b4126560a1275cfc5b04714fafe12@plan9.bell-labs.com>

I think 2^63 is a perfectly fine limit for the
forseeable future.  Do you really have files
(or file systems) that big?

The only point of the sentence about the 2^73
hash tree limit was that the maximum depth
of 7 is more than enough.

Russ

Re: [9fans] seek() in fossil?

[Russ Cox]

okay, you win.  it wasn't easy, but updated kernels
and binaries are on sources.   enjoy!

g% yesterday -d /sys/include/libc.h
diff /n/dump/2003/0126/sys/include/libc.h /sys/include/libc.h
624c624
< extern	vlong	seek(int, vlong, int);
---
> extern	mpint*	seek(int, mpint*, int);
g% 

Re: [9fans] seek() in fossil?

[rob pike, esq.]

This reminds me of the IPv6 fools, who seemed to think that
if 2^48 wasn't quite enough, they should go for 2^64 but just
to be sure, double it to 2^128.

Do you know how big 2^64 is?  It's not twice as big as a gigabyte.
Let's say we had a gigabit ethernet and we wanted to write a 2^64
byte file.  It would take more than a thousand years.

-rob

Re: [9fans] seek() in fossil?

[Mike Haertel]

>This reminds me of the IPv6 fools, who seemed to think that
>if 2^48 wasn't quite enough, they should go for 2^64 but just
>to be sure, double it to 2^128.

You might argue that 2^128 is insanely wasteful, but it's also
awfully convenient.  It allows all sorts of sparse allocation schemes
for large blocks of addresses, without fear that you're using up
too much of the address space.  At 2^64, some Actual Thought might
be required in doling out large chunks of the address space.  At
2^128 there is definitely no Actual Thought required: you just waste
address space however happens to be most convenient.

Consider this: at 2^64, we couldn't even afford to give each person
their own 2^32 block, since there are already more than 2^32 humans.
While a 2^32 block per person might seem incredibly wasteful, it's
not a huge stretch to imagine technologies that might want such a
thing happening within say the next 30 years (device density is
still increasing exponentially).  I can't imagine what such
technologies might be, but I am sure not going to go on the record
as saying "nope, that's ridiculous, will never happen."

As a programmer, you must have experienced the analagous situation:
often you can make a tradeoff between simple code and larger data
size, or more complex code and smaller data size.  Now that memory
is cheap, the former choice is almost always preferable.  You end
up "wasting" memory, but you save a lot in terms of coding and
debugging effort.  Saving actual thought in global network address
allocation is a similarly worthy tradeoff.

Re: [9fans] seek() in fossil?

[Douglas A. Gwyn]

rob pike, esq. wrote:
> Do you know how big 2^64 is?  It's not twice as big as a gigabyte.
> Let's say we had a gigabit ethernet and we wanted to write a 2^64
> byte file.  It would take more than a thousand years.

But we're talking about addresses; the data could be sparse.

Re: [9fans] seek() in fossil?

[Russ Cox]

> But we're talking about addresses; the data could be sparse.

Sure would make writing a venti server easier if we had 134 bit
file addresses (120 for the SHA1 hash and 16 for the data).

Re: [9fans] seek() in fossil?

[Russ Cox]

And multiply.

Re: [9fans] seek() in fossil?

[Russ Cox]

Sure would help if I could add.