[TUHS] 'Huge' file support removed from PWB1

[TUHS] 'Huge' file support removed from PWB1

[Noel Chiappa]

In PWB1, support for 'huge' files appears to have been removed. If one
compares bmap() in PWB1'S subr.c with V6's, the "'huge' fetch of double
indirect block" code is gone. I guess PWB didn't need very large (> 8*256*512
= 1,048,576 bytes) files? I'm not sure what the _benefits_ of removing it
were, though - unless PWB was generating lots of files of between 7*256*512
and 8*256*512 bytes in length, and they wanted to avoid the overhead of the
double-indirect block? (The savings in code space are derisory - unlike in
LSX/MINI-UNIX.) Anyone know?

		Noel

[TUHS] Re: 'Huge' file support removed from PWB1

[Kenneth Goodwin]

[-- Attachment #1: Type: text/plain, Size: 1162 bytes --]

I have not seen the UNIX kernel source code in quite a while, but as I
recall the double indirect block algorithm did not kick in until the file
exceeded a certain threshold. So it would not make sense to remove the code
for performance reasons.

Perhaps this is more likely due to the use of larger logical block sizes....

Is the code physically removed or IFDEF'd out for conditional compilation?

Perhaps someone decided that programmers would never need to test code on
large files..

On Wed, Mar 8, 2023, 8:10 AM Noel Chiappa <jnc@mercury.lcs.mit.edu> wrote:

> In PWB1, support for 'huge' files appears to have been removed. If one
> compares bmap() in PWB1'S subr.c with V6's, the "'huge' fetch of double
> indirect block" code is gone. I guess PWB didn't need very large (>
> 8*256*512
> = 1,048,576 bytes) files? I'm not sure what the _benefits_ of removing it
> were, though - unless PWB was generating lots of files of between 7*256*512
> and 8*256*512 bytes in length, and they wanted to avoid the overhead of the
> double-indirect block? (The savings in code space are derisory - unlike in
> LSX/MINI-UNIX.) Anyone know?
>
>                 Noel
>

[-- Attachment #2: Type: text/html, Size: 1649 bytes --]

[TUHS] Re: 'Huge' file support removed from PWB1

[Jonathan Gray]

removed in PWB/UNIX 1.0.

ifdef'd out in the Harvard/Radcliffe Student Time-sharing System (HRSTS)
parts found in tuhs/Applications/Usenix_77. h/distrib.note includes:

'4.	"Huge" files are not supported by our modifications.  This is not
necessarily a hard restriction, however we early on decided
we wanted no more than one level of indirection all the way up to 1 megabyte,
and had no need for larger files.  The incompatibilities may be minimal,
but we have not even bothered to seek them out.'

On Thu, Mar 09, 2023 at 03:21:19AM -0500, Kenneth Goodwin wrote:
> I have not seen the UNIX kernel source code in quite a while, but as I
> recall the double indirect block algorithm did not kick in until the file
> exceeded a certain threshold. So it would not make sense to remove the code
> for performance reasons.
> 
> Perhaps this is more likely due to the use of larger logical block sizes....
> 
> Is the code physically removed or IFDEF'd out for conditional compilation?
> 
> Perhaps someone decided that programmers would never need to test code on
> large files..
> 
> On Wed, Mar 8, 2023, 8:10 AM Noel Chiappa <jnc@mercury.lcs.mit.edu> wrote:
> 
> > In PWB1, support for 'huge' files appears to have been removed. If one
> > compares bmap() in PWB1'S subr.c with V6's, the "'huge' fetch of double
> > indirect block" code is gone. I guess PWB didn't need very large (>
> > 8*256*512
> > = 1,048,576 bytes) files? I'm not sure what the _benefits_ of removing it
> > were, though - unless PWB was generating lots of files of between 7*256*512
> > and 8*256*512 bytes in length, and they wanted to avoid the overhead of the
> > double-indirect block? (The savings in code space are derisory - unlike in
> > LSX/MINI-UNIX.) Anyone know?
> >
> >                 Noel
> >

[TUHS] Re: 'Huge' file support removed from PWB1

[Ron Natalie]

[-- Attachment #1: Type: text/plain, Size: 1543 bytes --]

In the version 6 file system, If the file was less than 4K, the four 
blocks in the inode refer directly to the disk blocks containing the 
file data.
After thef ile grows past that, the large bit is set in the inode and 
these blocks contain the single indirect blocks, referencing 256 data 
blocks each,  until you used up seven of these.
The last one was the double indirect block that pointed to other 
indirect blocks.   The file size was stored in 24 bits, and the block 
numbers were 16 bit.

As Noel points out, PWB uses the same file system as V6 but with the 
HUGE (last double indirect) code commented out.  I have no idea why and 
I never noticed it before.   While we used a lot of stuff from the PWB 
distribution (notably SCCS) we had no real interest in using the PWB’s 
kernel because our V6 kernel had diverged sharply from the Bell 
versions.

Version 7 came around with a different filesystem layout.    File sizes 
were 32 bits, and the disk block indexes were 24 bits.   The latter was 
stored as a series of bytes on the disk but expanded into longs in 
memeory.
There was no large bit anymore.   The first 8 blocks addresses in the 
inode were always single indirect, the next two were single indirect, 
and the last one was double indirect.   (If my memory serves me right)

At BRL, we modified our very-hacked V6 kernel to have the ability to 
mount either V6 or V7 filesystems.

We never ran V5, but I think V5 is essentially the V6 file system 
without the HUGE support again.

-Ron

[-- Attachment #2: Type: text/html, Size: 2497 bytes --]

[TUHS] Re: 'Huge' file support removed from PWB1

[Ron Natalie]

[-- Attachment #1: Type: text/plain, Size: 1991 bytes --]

That should read eight blocks in the inode not four in the first 
sentence (and up to eight later).



------ Original Message ------
From "Ron Natalie" <ron@ronnatalie.com>
To "Kenneth Goodwin" <kennethgoodwin56@gmail.com>; "Noel Chiappa" 
<jnc@mercury.lcs.mit.edu>
Cc "The Eunuchs Hysterical Society" <tuhs@tuhs.org>
Date 3/9/2023 10:07:17 AM
Subject [TUHS] Re: 'Huge' file support removed from PWB1

>In the version 6 file system, If the file was less than 4K, the four 
>blocks in the inode refer directly to the disk blocks containing the 
>file data.
>After thef ile grows past that, the large bit is set in the inode and 
>these blocks contain the single indirect blocks, referencing 256 data 
>blocks each,  until you used up seven of these.
>The last one was the double indirect block that pointed to other 
>indirect blocks.   The file size was stored in 24 bits, and the block 
>numbers were 16 bit.
>
>As Noel points out, PWB uses the same file system as V6 but with the 
>HUGE (last double indirect) code commented out.  I have no idea why and 
>I never noticed it before.   While we used a lot of stuff from the PWB 
>distribution (notably SCCS) we had no real interest in using the PWB’s 
>kernel because our V6 kernel had diverged sharply from the Bell 
>versions.
>
>Version 7 came around with a different filesystem layout.    File sizes 
>were 32 bits, and the disk block indexes were 24 bits.   The latter was 
>stored as a series of bytes on the disk but expanded into longs in 
>memeory.
>There was no large bit anymore.   The first 8 blocks addresses in the 
>inode were always single indirect, the next two were single indirect, 
>and the last one was double indirect.   (If my memory serves me right)
>
>At BRL, we modified our very-hacked V6 kernel to have the ability to 
>mount either V6 or V7 filesystems.
>
>We never ran V5, but I think V5 is essentially the V6 file system 
>without the HUGE support again.
>
>-Ron

[-- Attachment #2: Type: text/html, Size: 3853 bytes --]

[TUHS] Re: 'Huge' file support removed from PWB1

[Tom Ivar Helbekkmo via TUHS]

Noel Chiappa <jnc@mercury.lcs.mit.edu> writes:

> In PWB1, support for 'huge' files appears to have been removed.

They did make various changes to the system to protect against resource
exhaustion in what was a genuine production multi-user facility, and
might possibly have made such a change to ensure that a runaway process
couldn't write a file more than a megabyte in size.

It's not mentioned in the PWB article in the special UNIX issue of the
Bell System Technical Journal, though, and that article does describe
various changes they'd made, explaining how and why.

However: PWB grew out of Research, side by side with it, evaluating and
adopting changes along the way.  They aimed to stay mostly in synch,
but chose to keep and maintain a number of differences.  Maybe they
started their work before the "huge" file support was introduced, and
then simply did not include that change?  It's a backward incompatible
change (at least if you have at least one file that already uses the
last indirect block of its inode), and they might have decided to cross
that bridge if and when they ever got to it.

-tih
-- 
Most people who graduate with CS degrees don't understand the significance
of Lisp.  Lisp is the most important idea in computer science.  --Alan Kay

[TUHS] Re: 'Huge' file support removed from PWB1

[arnold]

"Ron Natalie" <ron@ronnatalie.com> wrote:

> Version 7 came around with a different filesystem layout.    File sizes 
> were 32 bits, and the disk block indexes were 24 bits.   The latter was 
> stored as a series of bytes on the disk but expanded into longs in 
> memeory.
> There was no large bit anymore.   The first 8 blocks addresses in the 
> inode were always single indirect, the next two were single indirect, 
> and the last one was double indirect.   (If my memory serves me right)

Wasn't there a triple-indirect block as the very last one?

Arnold

[TUHS] Re: 'Huge' file support removed from PWB1

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 1982 bytes --]

I talked with John Mashey this PM and asked him about the huge file stuff
and PWB1.

First, he did not remember that they had done that, but ...

He then emphasized that the PWB team was running the first (and for a long
time the largest) "computer center" using UNIX in the Bell System in
Piscataway - supporting over 1000 programmers as a front end to the
mainframes. His memory is that they mostly used 88 Mbyte RP04s [remember,
at the time, RK05s were only 2.5M].  Needing to support a megabyte or
larger file would have been relatively rare for them.  He also pointed out
they needed to run their system on systems as small as 48K byte 11/40s to
as large as 1M 11/70s.

As he said, his team tried to track Dennis and Ken's Research
system closely since the PWB team was not primarily in the system
development/enhancement business -* i.e.* they tied to keep what would
become PWB1 and whatever was running at Research as similar as they could
[he told some stories about running to MH to get Dennis' latest compiler
binary because the incremental development scheme dmr used would make
someone like PWB that took snapshots at different times, end up with a
compiler that could not compile itself --#1#]. But the PWB group >> was <<
interested in making the system they were running as reliable as possible
and, more importantly, being as graceful as possible when different
resources were exhausted.

He said it would have made sense for them to remove the "huge file" support
to help to defend against running out of space and the system having issues
[V6 was extremely ungrateful about that condition]. So limiting
file size became a simple quota, if you will. A runaway program was less
likely to take the system out. So the process would die because the file
got too large (or, to use the IBM shop term in those days -- ABEND).


#1# He also regaled a bit about Ken's infamous Turning Award hack and
discovering symbols in the compiler binary they could not understand ;-)

[-- Attachment #2: Type: text/html, Size: 3050 bytes --]