[9front] a novel network protocol of my own design

[9front] a novel network protocol of my own design

[binary cat]

[-- Attachment #1: Type: text/plain, Size: 532 bytes --]

a while ago, frustrated with the complexity of proper http error recovery,
i decided to put together my ideas for a protocol that has less moving
parts that can go wrong.

my protocol is specifically targeted at downloading large immutable files
from a single high-latency source, since that's something http does better
than most alternatives, but only if implemented perfectly.

since the protocol is partially inspired by 9p, i'm curious what you all
think of it.

https://paper.wf/binarycat/reliable-immutable-transfer-protocol

[-- Attachment #2: Type: text/html, Size: 827 bytes --]

Re: [9front] a novel network protocol of my own design

[Jacob Moody]

On 1/9/25 20:45, binary cat wrote:
> a while ago, frustrated with the complexity of proper http error recovery, i decided to put together my ideas for a protocol that has less moving parts that can go wrong.
> 
> my protocol is specifically targeted at downloading large immutable files from a single high-latency source, since that's something http does better than most alternatives, but only if implemented perfectly.
> 
> since the protocol is partially inspired by 9p, i'm curious what you all think of it.
> 
> https://paper.wf/binarycat/reliable-immutable-transfer-protocol <https://paper.wf/binarycat/reliable-immutable-transfer-protocol>
> 

I think your portion of this protocol is fairly sensible, but it seems that
the multihash (and thus uvarint) portions eat up nearly half(if not more)of the complexity of
an implementation, assuming you are starting from scratch. It also seems that
without some sort of negotiation to communicate which hashes are supported on
the server side, you end up with a trial and failure loop trying to address
a given file. I also don't really like the idea of a server having to keep
a decent chunk of different hash algorithms cached for data that it's hosting.

If I was doing something like this I might start with a version negotiation, and then
just tie a good enough™ hash function to a given version. I do realize this muddies
the current nice semantics you have for backwards compatibility, but I think its
worth it. Or you could communicate supported hashes in band through a specific message,
with the server advertising first. Not entirely sure which is better.

I also wanted to mention that for this specific use-case I don't think the issue regarding
high-latency for 9p would manifest if you're somewhat careful. To illustrate this I want
to sketch out what a 9p server implementation that solves a similar problem could do.
The server could serve a single directory where each file is addressed by its hash.
A client would then connect, open a file using it's hash, and then begin reading.
As long as the client is only reading, there is no requirement for the client to
wait for a response of a given Tread before sending another, programs like clone(1)
use this to read a file in parallel, each proc reading different subsections of the
total file.

Given this, a rough idea of the a server<->client communication while keeping tally
of round trips would be something like:

c: Tversion
s: Rversion (1 roundtrip)
c: Tattach
s: Rattach (2 roundtrips)
c: Twalk - Finding the file by hash
s: Rwalk (3 roundtrips)
c: Tstat - Get size of file of interest for calculating offsets later
s: Rstat (4 roundtrips)
c: Topen - Open file of interest for IO
s: Ropen (5 roundtrips)

Now at this point you can submit as many Treads as your hearts content without needing to
wait for any given Rread.

Now sure, 5 round trips is a bit much compared to your protocols 1 before we start getting the
actual data on the wire but it's all front loaded, reusing a mount saves you two (more like one
because for reuse you'd have to clone the root fid before walking to a child). However depending
on your definition of "large" files this may not be a significant portion of the time spent in a transfer.

In a pretty worst case (150ms latency) you're looking at 3 quarters of a second, and for something a bit
more local you're likely looking at ~30-50ms of latency which roughly a quarter of a second. Sure it's not
_great_ but I think I spend more time than this waiting for peer discovery with bit-torrent than I would here.

The bad wrap that 9p gets in regards to high latency comes mostly from directory traversal and writes, if you're
just reading its fairly sensible.

There are some other games you could play, you could perhaps open the root and read that to get the length count
for a whole bunch of files in one go without having to stat each.

I didn't intend for this to become as large of an email as it did, but here we are.


Thanks,
Jacob Moody

Re: [9front] a novel network protocol of my own design

[binary cat]

[-- Attachment #1: Type: text/plain, Size: 5584 bytes --]

i agree with most of your points, and i think multihash is pretty complex,
but i didn't want to make my own format because of the maintenance burden
and requirement for clients to reimplement everything.

there's just two small things i think you overlooked

1. there's no need to check hash support.  both what server has the file
and the fully encoded hash are communicated out-of-band (eg. via a future
url scheme).

also, a particular server implementation can bypass the implementation
overhead of multihash by only supporting a single hash function and
hardcoding the prefix.  for example, a full length sha256 multihash will
always start with the 3 bytes 0x12ff02. (0x12 is the code for sha2-256, and
0xff02 is the varint encoding for 256)

2. latency isn't the only problem with 9p, it also shares two flaws with
http: mutability and lack of a checksum.  there's no way for a general 9p
or http client to reliably know that the file they're downloading hasn't
been modified halfway through. (well, http has a few, but they're all
optional to implement, and Repr-Digest isn't even finalized yet)

also, i don't mind the long response, it was pretty interesting to read :3

On Thu, Jan 9, 2025, 22:24 Jacob Moody <moody@posixcafe.org> wrote:

> On 1/9/25 20:45, binary cat wrote:
> > a while ago, frustrated with the complexity of proper http error
> recovery, i decided to put together my ideas for a protocol that has less
> moving parts that can go wrong.
> >
> > my protocol is specifically targeted at downloading large immutable
> files from a single high-latency source, since that's something http does
> better than most alternatives, but only if implemented perfectly.
> >
> > since the protocol is partially inspired by 9p, i'm curious what you all
> think of it.
> >
> > https://paper.wf/binarycat/reliable-immutable-transfer-protocol <
> https://paper.wf/binarycat/reliable-immutable-transfer-protocol>
> >
>
> I think your portion of this protocol is fairly sensible, but it seems that
> the multihash (and thus uvarint) portions eat up nearly half(if not
> more)of the complexity of
> an implementation, assuming you are starting from scratch. It also seems
> that
> without some sort of negotiation to communicate which hashes are supported
> on
> the server side, you end up with a trial and failure loop trying to address
> a given file. I also don't really like the idea of a server having to keep
> a decent chunk of different hash algorithms cached for data that it's
> hosting.
>
> If I was doing something like this I might start with a version
> negotiation, and then
> just tie a good enough™ hash function to a given version. I do realize
> this muddies
> the current nice semantics you have for backwards compatibility, but I
> think its
> worth it. Or you could communicate supported hashes in band through a
> specific message,
> with the server advertising first. Not entirely sure which is better.
>
> I also wanted to mention that for this specific use-case I don't think the
> issue regarding
> high-latency for 9p would manifest if you're somewhat careful. To
> illustrate this I want
> to sketch out what a 9p server implementation that solves a similar
> problem could do.
> The server could serve a single directory where each file is addressed by
> its hash.
> A client would then connect, open a file using it's hash, and then begin
> reading.
> As long as the client is only reading, there is no requirement for the
> client to
> wait for a response of a given Tread before sending another, programs like
> clone(1)
> use this to read a file in parallel, each proc reading different
> subsections of the
> total file.
>
> Given this, a rough idea of the a server<->client communication while
> keeping tally
> of round trips would be something like:
>
> c: Tversion
> s: Rversion (1 roundtrip)
> c: Tattach
> s: Rattach (2 roundtrips)
> c: Twalk - Finding the file by hash
> s: Rwalk (3 roundtrips)
> c: Tstat - Get size of file of interest for calculating offsets later
> s: Rstat (4 roundtrips)
> c: Topen - Open file of interest for IO
> s: Ropen (5 roundtrips)
>
> Now at this point you can submit as many Treads as your hearts content
> without needing to
> wait for any given Rread.
>
> Now sure, 5 round trips is a bit much compared to your protocols 1 before
> we start getting the
> actual data on the wire but it's all front loaded, reusing a mount saves
> you two (more like one
> because for reuse you'd have to clone the root fid before walking to a
> child). However depending
> on your definition of "large" files this may not be a significant portion
> of the time spent in a transfer.
>
> In a pretty worst case (150ms latency) you're looking at 3 quarters of a
> second, and for something a bit
> more local you're likely looking at ~30-50ms of latency which roughly a
> quarter of a second. Sure it's not
> _great_ but I think I spend more time than this waiting for peer discovery
> with bit-torrent than I would here.
>
> The bad wrap that 9p gets in regards to high latency comes mostly from
> directory traversal and writes, if you're
> just reading its fairly sensible.
>
> There are some other games you could play, you could perhaps open the root
> and read that to get the length count
> for a whole bunch of files in one go without having to stat each.
>
> I didn't intend for this to become as large of an email as it did, but
> here we are.
>
>
> Thanks,
> Jacob Moody
>
>

[-- Attachment #2: Type: text/html, Size: 6385 bytes --]

Re: [9front] a novel network protocol of my own design

[Jacob Moody]

On 1/10/25 14:38, binary cat wrote:
> i agree with most of your points, and i think multihash is pretty complex, but i didn't want to make my own format because of the maintenance burden and requirement for clients to reimplement everything.
> 
> there's just two small things i think you overlooked
> 
> 1. there's no need to check hash support.  both what server has the file and the fully encoded hash are communicated out-of-band (eg. via a future url scheme).
> 
> also, a particular server implementation can bypass the implementation overhead of multihash by only supporting a single hash function and hardcoding the prefix.  for example, a full length sha256 multihash will always start with the 3 bytes 0x12ff02. (0x12 is the code for sha2-256, and 0xff02 is the varint encoding for 256)

Ah okay, yeah communicating which hash to use out-of-band would certainly solve the issue I mentioned. I could be mistaken but I don't think the document you
linked went over this so I was only working with the information I had. In terms of supporting just a single hash, sure that could be hardcoded without an issue
but I think it would be nice to explicitly list a hash that is known to work inside the spec ie "all clients and servers MUST/SHOULD support sha2-256 as a baseline"
It pays to be explicit with expectations when you're putting out specs in to the world for other people to implement :)

> 
> 2. latency isn't the only problem with 9p, it also shares two flaws with http: mutability and lack of a checksum.  there's no way for a general 9p or http client to reliably know that the file they're downloading hasn't been modified halfway through. (well, http has a few, but they're all optional to implement, and Repr-Digest isn't even finalized yet)

I wasn't too explicit with this but I did not imagine this being a general purpose file server, I imagined this being a specific file server for this purpose.
You would have a program(bigtransferfs) that would look at an existing directory of files and serve 9p with their names as the hash of their contents,
bigtransferFS would not allow connecting clients any ability to modify files. Now the place that the fs is using to get the files could allow them to be
modified under itself and you'll have to figure some way of either communicating to the "host" filesystem that you can't do that(or just be careful),
but any ritp server has the same issue. So really the issue of mutated content is not different between bigtransferfs and a ritp server.

There's also no need for a checksum, you are addressing the file with its hash, you can just verify that the content is correct by hashing what you actually
downloaded and comparing it with the name you used to grab it.


Thanks,
moody

Re: [9front] a novel network protocol of my own design

[binary cat]

[-- Attachment #1: Type: text/plain, Size: 3113 bytes --]

yeah, i suppose i should've clarified the whole out of band thing.

also, the sha256 baseline was in a previous version, added it back in an
addendum.

thanks for the feedback :3

On Fri, Jan 10, 2025, 17:16 Jacob Moody <moody@posixcafe.org> wrote:

> On 1/10/25 14:38, binary cat wrote:
> > i agree with most of your points, and i think multihash is pretty
> complex, but i didn't want to make my own format because of the maintenance
> burden and requirement for clients to reimplement everything.
> >
> > there's just two small things i think you overlooked
> >
> > 1. there's no need to check hash support.  both what server has the file
> and the fully encoded hash are communicated out-of-band (eg. via a future
> url scheme).
> >
> > also, a particular server implementation can bypass the implementation
> overhead of multihash by only supporting a single hash function and
> hardcoding the prefix.  for example, a full length sha256 multihash will
> always start with the 3 bytes 0x12ff02. (0x12 is the code for sha2-256, and
> 0xff02 is the varint encoding for 256)
>
> Ah okay, yeah communicating which hash to use out-of-band would certainly
> solve the issue I mentioned. I could be mistaken but I don't think the
> document you
> linked went over this so I was only working with the information I had. In
> terms of supporting just a single hash, sure that could be hardcoded
> without an issue
> but I think it would be nice to explicitly list a hash that is known to
> work inside the spec ie "all clients and servers MUST/SHOULD support
> sha2-256 as a baseline"
> It pays to be explicit with expectations when you're putting out specs in
> to the world for other people to implement :)
>
> >
> > 2. latency isn't the only problem with 9p, it also shares two flaws with
> http: mutability and lack of a checksum.  there's no way for a general 9p
> or http client to reliably know that the file they're downloading hasn't
> been modified halfway through. (well, http has a few, but they're all
> optional to implement, and Repr-Digest isn't even finalized yet)
>
> I wasn't too explicit with this but I did not imagine this being a general
> purpose file server, I imagined this being a specific file server for this
> purpose.
> You would have a program(bigtransferfs) that would look at an existing
> directory of files and serve 9p with their names as the hash of their
> contents,
> bigtransferFS would not allow connecting clients any ability to modify
> files. Now the place that the fs is using to get the files could allow them
> to be
> modified under itself and you'll have to figure some way of either
> communicating to the "host" filesystem that you can't do that(or just be
> careful),
> but any ritp server has the same issue. So really the issue of mutated
> content is not different between bigtransferfs and a ritp server.
>
> There's also no need for a checksum, you are addressing the file with its
> hash, you can just verify that the content is correct by hashing what you
> actually
> downloaded and comparing it with the name you used to grab it.
>
>
> Thanks,
> moody
>
>

[-- Attachment #2: Type: text/html, Size: 3580 bytes --]

Re: [9front] a novel network protocol of my own design

[Ori Bernstein]

Clients already need to reimplement everything.

Don't hedge: require a specific hash. Rev the protocol if the hash gets broken.

On January 10, 2025 2:38:27 PM CST, binary cat <dogedoge61@gmail.com> wrote:
>i agree with most of your points, and i think multihash is pretty complex,
>but i didn't want to make my own format because of the maintenance burden
>and requirement for clients to reimplement everything.
>
>there's just two small things i think you overlooked
>
>1. there's no need to check hash support.  both what server has the file
>and the fully encoded hash are communicated out-of-band (eg. via a future
>url scheme).
>
>also, a particular server implementation can bypass the implementation
>overhead of multihash by only supporting a single hash function and
>hardcoding the prefix.  for example, a full length sha256 multihash will
>always start with the 3 bytes 0x12ff02. (0x12 is the code for sha2-256, and
>0xff02 is the varint encoding for 256)
>
>2. latency isn't the only problem with 9p, it also shares two flaws with
>http: mutability and lack of a checksum.  there's no way for a general 9p
>or http client to reliably know that the file they're downloading hasn't
>been modified halfway through. (well, http has a few, but they're all
>optional to implement, and Repr-Digest isn't even finalized yet)
>
>also, i don't mind the long response, it was pretty interesting to read :3
>
>On Thu, Jan 9, 2025, 22:24 Jacob Moody <moody@posixcafe.org> wrote:
>
>> On 1/9/25 20:45, binary cat wrote:
>> > a while ago, frustrated with the complexity of proper http error
>> recovery, i decided to put together my ideas for a protocol that has less
>> moving parts that can go wrong.
>> >
>> > my protocol is specifically targeted at downloading large immutable
>> files from a single high-latency source, since that's something http does
>> better than most alternatives, but only if implemented perfectly.
>> >
>> > since the protocol is partially inspired by 9p, i'm curious what you all
>> think of it.
>> >
>> > https://paper.wf/binarycat/reliable-immutable-transfer-protocol <
>> https://paper.wf/binarycat/reliable-immutable-transfer-protocol>
>> >
>>
>> I think your portion of this protocol is fairly sensible, but it seems that
>> the multihash (and thus uvarint) portions eat up nearly half(if not
>> more)of the complexity of
>> an implementation, assuming you are starting from scratch. It also seems
>> that
>> without some sort of negotiation to communicate which hashes are supported
>> on
>> the server side, you end up with a trial and failure loop trying to address
>> a given file. I also don't really like the idea of a server having to keep
>> a decent chunk of different hash algorithms cached for data that it's
>> hosting.
>>
>> If I was doing something like this I might start with a version
>> negotiation, and then
>> just tie a good enough™ hash function to a given version. I do realize
>> this muddies
>> the current nice semantics you have for backwards compatibility, but I
>> think its
>> worth it. Or you could communicate supported hashes in band through a
>> specific message,
>> with the server advertising first. Not entirely sure which is better.
>>
>> I also wanted to mention that for this specific use-case I don't think the
>> issue regarding
>> high-latency for 9p would manifest if you're somewhat careful. To
>> illustrate this I want
>> to sketch out what a 9p server implementation that solves a similar
>> problem could do.
>> The server could serve a single directory where each file is addressed by
>> its hash.
>> A client would then connect, open a file using it's hash, and then begin
>> reading.
>> As long as the client is only reading, there is no requirement for the
>> client to
>> wait for a response of a given Tread before sending another, programs like
>> clone(1)
>> use this to read a file in parallel, each proc reading different
>> subsections of the
>> total file.
>>
>> Given this, a rough idea of the a server<->client communication while
>> keeping tally
>> of round trips would be something like:
>>
>> c: Tversion
>> s: Rversion (1 roundtrip)
>> c: Tattach
>> s: Rattach (2 roundtrips)
>> c: Twalk - Finding the file by hash
>> s: Rwalk (3 roundtrips)
>> c: Tstat - Get size of file of interest for calculating offsets later
>> s: Rstat (4 roundtrips)
>> c: Topen - Open file of interest for IO
>> s: Ropen (5 roundtrips)
>>
>> Now at this point you can submit as many Treads as your hearts content
>> without needing to
>> wait for any given Rread.
>>
>> Now sure, 5 round trips is a bit much compared to your protocols 1 before
>> we start getting the
>> actual data on the wire but it's all front loaded, reusing a mount saves
>> you two (more like one
>> because for reuse you'd have to clone the root fid before walking to a
>> child). However depending
>> on your definition of "large" files this may not be a significant portion
>> of the time spent in a transfer.
>>
>> In a pretty worst case (150ms latency) you're looking at 3 quarters of a
>> second, and for something a bit
>> more local you're likely looking at ~30-50ms of latency which roughly a
>> quarter of a second. Sure it's not
>> _great_ but I think I spend more time than this waiting for peer discovery
>> with bit-torrent than I would here.
>>
>> The bad wrap that 9p gets in regards to high latency comes mostly from
>> directory traversal and writes, if you're
>> just reading its fairly sensible.
>>
>> There are some other games you could play, you could perhaps open the root
>> and read that to get the length count
>> for a whole bunch of files in one go without having to stat each.
>>
>> I didn't intend for this to become as large of an email as it did, but
>> here we are.
>>
>>
>> Thanks,
>> Jacob Moody
>>
>>

Re: [9front] a novel network protocol of my own design

[binary cat]

[-- Attachment #1: Type: text/plain, Size: 6795 bytes --]

i have three reasons for not making it a hard requirement:

1. allowing single-hash servers as described above, without requiring any
specific hash
2. extremely simple servers may not do any hashing themselves, requiring
sysadmins to manually enter a hash and a file (presumably through some sort
of unix-y script)
3. extremely simple clients may wish to blindly trust the downloaded file
instead of verifying the hash (perhaps certain embedded clients)

On Fri, Jan 10, 2025, 21:17 Ori Bernstein <ori@eigenstate.org> wrote:

> Clients already need to reimplement everything.
>
> Don't hedge: require a specific hash. Rev the protocol if the hash gets
> broken.
>
> On January 10, 2025 2:38:27 PM CST, binary cat <dogedoge61@gmail.com>
> wrote:
> >i agree with most of your points, and i think multihash is pretty complex,
> >but i didn't want to make my own format because of the maintenance burden
> >and requirement for clients to reimplement everything.
> >
> >there's just two small things i think you overlooked
> >
> >1. there's no need to check hash support.  both what server has the file
> >and the fully encoded hash are communicated out-of-band (eg. via a future
> >url scheme).
> >
> >also, a particular server implementation can bypass the implementation
> >overhead of multihash by only supporting a single hash function and
> >hardcoding the prefix.  for example, a full length sha256 multihash will
> >always start with the 3 bytes 0x12ff02. (0x12 is the code for sha2-256,
> and
> >0xff02 is the varint encoding for 256)
> >
> >2. latency isn't the only problem with 9p, it also shares two flaws with
> >http: mutability and lack of a checksum.  there's no way for a general 9p
> >or http client to reliably know that the file they're downloading hasn't
> >been modified halfway through. (well, http has a few, but they're all
> >optional to implement, and Repr-Digest isn't even finalized yet)
> >
> >also, i don't mind the long response, it was pretty interesting to read :3
> >
> >On Thu, Jan 9, 2025, 22:24 Jacob Moody <moody@posixcafe.org> wrote:
> >
> >> On 1/9/25 20:45, binary cat wrote:
> >> > a while ago, frustrated with the complexity of proper http error
> >> recovery, i decided to put together my ideas for a protocol that has
> less
> >> moving parts that can go wrong.
> >> >
> >> > my protocol is specifically targeted at downloading large immutable
> >> files from a single high-latency source, since that's something http
> does
> >> better than most alternatives, but only if implemented perfectly.
> >> >
> >> > since the protocol is partially inspired by 9p, i'm curious what you
> all
> >> think of it.
> >> >
> >> > https://paper.wf/binarycat/reliable-immutable-transfer-protocol <
> >> https://paper.wf/binarycat/reliable-immutable-transfer-protocol>
> >> >
> >>
> >> I think your portion of this protocol is fairly sensible, but it seems
> that
> >> the multihash (and thus uvarint) portions eat up nearly half(if not
> >> more)of the complexity of
> >> an implementation, assuming you are starting from scratch. It also seems
> >> that
> >> without some sort of negotiation to communicate which hashes are
> supported
> >> on
> >> the server side, you end up with a trial and failure loop trying to
> address
> >> a given file. I also don't really like the idea of a server having to
> keep
> >> a decent chunk of different hash algorithms cached for data that it's
> >> hosting.
> >>
> >> If I was doing something like this I might start with a version
> >> negotiation, and then
> >> just tie a good enough™ hash function to a given version. I do realize
> >> this muddies
> >> the current nice semantics you have for backwards compatibility, but I
> >> think its
> >> worth it. Or you could communicate supported hashes in band through a
> >> specific message,
> >> with the server advertising first. Not entirely sure which is better.
> >>
> >> I also wanted to mention that for this specific use-case I don't think
> the
> >> issue regarding
> >> high-latency for 9p would manifest if you're somewhat careful. To
> >> illustrate this I want
> >> to sketch out what a 9p server implementation that solves a similar
> >> problem could do.
> >> The server could serve a single directory where each file is addressed
> by
> >> its hash.
> >> A client would then connect, open a file using it's hash, and then begin
> >> reading.
> >> As long as the client is only reading, there is no requirement for the
> >> client to
> >> wait for a response of a given Tread before sending another, programs
> like
> >> clone(1)
> >> use this to read a file in parallel, each proc reading different
> >> subsections of the
> >> total file.
> >>
> >> Given this, a rough idea of the a server<->client communication while
> >> keeping tally
> >> of round trips would be something like:
> >>
> >> c: Tversion
> >> s: Rversion (1 roundtrip)
> >> c: Tattach
> >> s: Rattach (2 roundtrips)
> >> c: Twalk - Finding the file by hash
> >> s: Rwalk (3 roundtrips)
> >> c: Tstat - Get size of file of interest for calculating offsets later
> >> s: Rstat (4 roundtrips)
> >> c: Topen - Open file of interest for IO
> >> s: Ropen (5 roundtrips)
> >>
> >> Now at this point you can submit as many Treads as your hearts content
> >> without needing to
> >> wait for any given Rread.
> >>
> >> Now sure, 5 round trips is a bit much compared to your protocols 1
> before
> >> we start getting the
> >> actual data on the wire but it's all front loaded, reusing a mount saves
> >> you two (more like one
> >> because for reuse you'd have to clone the root fid before walking to a
> >> child). However depending
> >> on your definition of "large" files this may not be a significant
> portion
> >> of the time spent in a transfer.
> >>
> >> In a pretty worst case (150ms latency) you're looking at 3 quarters of a
> >> second, and for something a bit
> >> more local you're likely looking at ~30-50ms of latency which roughly a
> >> quarter of a second. Sure it's not
> >> _great_ but I think I spend more time than this waiting for peer
> discovery
> >> with bit-torrent than I would here.
> >>
> >> The bad wrap that 9p gets in regards to high latency comes mostly from
> >> directory traversal and writes, if you're
> >> just reading its fairly sensible.
> >>
> >> There are some other games you could play, you could perhaps open the
> root
> >> and read that to get the length count
> >> for a whole bunch of files in one go without having to stat each.
> >>
> >> I didn't intend for this to become as large of an email as it did, but
> >> here we are.
> >>
> >>
> >> Thanks,
> >> Jacob Moody
> >>
> >>
>

[-- Attachment #2: Type: text/html, Size: 8587 bytes --]