[TUHS] reboot(2) system call

[TUHS] reboot(2) system call

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 290 bytes --]

I've done some research for a friend about when the reboot() system call
was added, and how it related to the sync, sync, sync dance.

https://bsdimp.blogspot.com/2020/07/when-unix-learned-to-reboot2.html

may be of interest. Please do let me know if I've gotten something wrong...

Warner

[-- Attachment #2: Type: text/html, Size: 481 bytes --]

Re: [TUHS] reboot(2) system call

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 2215 bytes --]

Warner - awesome.  I would add a few things that were sort of in the story,
although not directly.

The primary difference between 4.0 and 4.1 was originally a political push
with Stanford trying to get VMS as the official OS for Arpa and Joy
countering with BSD claiming it was within epsilon of VMS and in some cases
faster.  The performance was the biggest of the Stanford complaints which
Bill fixed with a much a carefully placed assembler.  But IIRC, one of the
other issues was "UNIX is unreliable" and file system corrupt was tossed
out as an example of the same [The lack of real/complete Fortran compiler
was never corrected, but I remember that was one of the other reasons
given.  But it was one of the reasons why a couple of UCB compiler folks
worked on improving F77 and building dbx, although it was never in the same
league as VMS's compiler or any of the many DEC debuggers].

Anyway, after 4.0 came out, Goble released the 'Purdue file system fixes'.
 When V6-4.1BSD Unix crashed, the OS was noted for leaving a possible
corrupted file system (this was why tjk wrote fsck of course for V6 years
before).  George really spent time analyzing the order that different
writes were done *i.e.* making sure the data, inodes, superblock were
updated in the right order.  This cut down file system corruption after a
system crash.  He also had a way to force the buffer cache to be cleared
properly and part of that was that George had implemented reboot
independently of Joy and I think before Bill.

Certainly by 4.1B when McKussick's new FS was being folded into the kernel,
Joy had pulled the original file system write order changes from Purdue.
He may have done that as part of 4.1 "FASTVAX" work and before he explored
reboot.

The point is that sync, reboot, and file system block write ordering was
all mixed together.


On Mon, Jul 27, 2020 at 3:01 PM Warner Losh <imp@bsdimp.com> wrote:

> I've done some research for a friend about when the reboot() system call
> was added, and how it related to the sync, sync, sync dance.
>
> https://bsdimp.blogspot.com/2020/07/when-unix-learned-to-reboot2.html
>
> may be of interest. Please do let me know if I've gotten something wrong...
>
> Warner
>

[-- Attachment #2: Type: text/html, Size: 3400 bytes --]

Re: [TUHS] reboot(2) system call

[Dave Horsfall]

[-- Attachment #1: Type: text/plain, Size: 533 bytes --]

On Mon, 27 Jul 2020, Warner Losh wrote:

> I've done some research for a friend about when the reboot() system call 
> was added, and how it related to the sync, sync, sync dance. 
> https://bsdimp.blogspot.com/2020/07/when-unix-learned-to-reboot2.html
> 
> may be of interest. Please do let me know if I've gotten 
> something wrong...

Seems OK to me; I was taught never to use "sync; sync; sync" for precisely 
those reasons (the buffers may not have time to flush etc, as "sync" 
merely schedules the I/O, not cause it.

-- Dave

Re: [TUHS] reboot(2) system call

[Win Treese]

> On Jul 28, 2020, at 12:06 AM, Dave Horsfall <dave@horsfall.org> wrote:
> 
> On Mon, 27 Jul 2020, Warner Losh wrote:
> 
>> I've done some research for a friend about when the reboot() system call was added, and how it related to the sync, sync, sync dance. https://bsdimp.blogspot.com/2020/07/when-unix-learned-to-reboot2.html
>> may be of interest. Please do let me know if I've gotten something wrong...
> 
> Seems OK to me; I was taught never to use "sync; sync; sync" for precisely those reasons (the buffers may not have time to flush etc, as "sync" merely schedules the I/O, not cause it.

This was summarized at MIT’s Project Athena in the mid-80s as:

When thou shuttest down the system, thou shalt sync three times. No more, no less. Three shall be the number of the syncing, and the number of the syncing shall be three. Four times shalt thou not sync, neither sync twice, except that thou proceedest to sync a third time. Five is right out.

- Win

Re: [TUHS] reboot(2) system call

[Jaap Akkerhuis]

> On Jul 27, 2020, at 20:57, Warner Losh <imp@bsdimp.com> wrote:
> 
> I've done some research for a friend about when the reboot() system call was added, and how it related to the sync, sync, sync dance.

I remember that Piet Beertema (CWI) added a reboot command to our
unix V7 on the 11/45.  I just asked him and he said that it he did
it because he was tired to walk to the computer room.  It was way
more work then he at first expected.

Whether this was picked op by the Berkeley people he doesn't know,
but it could be.

	jaap

Re: [TUHS] reboot(2) system call

[Greg A. Woods]

[-- Attachment #1: Type: text/plain, Size: 1084 bytes --]

At Mon, 27 Jul 2020 12:57:59 -0600, Warner Losh <imp@bsdimp.com> wrote:
Subject: [TUHS] reboot(2) system call
>
> I've done some research for a friend about when the reboot() system call
> was added, and how it related to the sync, sync, sync dance.
>
> https://bsdimp.blogspot.com/2020/07/when-unix-learned-to-reboot2.html
>
> may be of interest. Please do let me know if I've gotten something wrong...

I've definitely experienced the second "sync" blocking until the first
was done.  Once upon a time I remembered exactly why that happened, but
I no longer do.

The lore I was told at the time was that you alwasy ran three and that
it didn't matter if they were all on the same line with semicolons or
not because of the very fact that the second one would block.

The third one was just for superstitious good luck, and just in case
there was a process still running and writing to a filesystem.

--
					Greg A. Woods <gwoods@acm.org>

Kelowna, BC     +1 250 762-7675           RoboHack <woods@robohack.ca>
Planix, Inc. <woods@planix.com>     Avoncote Farms <woods@avoncote.ca>

[-- Attachment #2: OpenPGP Digital Signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [TUHS] reboot(2) system call

[Dave Horsfall]

On Tue, 26 Jan 2021, Greg A. Woods wrote:

> The lore I was told at the time was that you alwasy ran three and that 
> it didn't matter if they were all on the same line with semicolons or 
> not because of the very fact that the second one would block.

What I was taught was:

 	% sync
 	% sync
 	% sync

and never:

 	% sync; sync; sync

The theory was that by waiting for the shell prompt each time, it gave the 
buffer pool enough time to be flushed.

> The third one was just for superstitious good luck, and just in case 
> there was a process still running and writing to a filesystem.

Never heard that theory, but naturally you did a "ps ax" first, didn't 
you?

-- Dave

Re: [TUHS] reboot(2) system call

[Larry McVoy]

On Sun, Jan 31, 2021 at 09:27:10AM +1100, Dave Horsfall wrote:
> On Tue, 26 Jan 2021, Greg A. Woods wrote:
> 
> >The lore I was told at the time was that you alwasy ran three and that it
> >didn't matter if they were all on the same line with semicolons or not
> >because of the very fact that the second one would block.
> 
> What I was taught was:
> 
> 	% sync
> 	% sync
> 	% sync
> 
> and never:
> 
> 	% sync; sync; sync
> 
> The theory was that by waiting for the shell prompt each time, it gave the
> buffer pool enough time to be flushed.

I was taught the exact same thing and for the same reasons.

Re: [TUHS] reboot(2) system call

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 1333 bytes --]

On Sat, Jan 30, 2021 at 3:32 PM Larry McVoy <lm@mcvoy.com> wrote:

> On Sun, Jan 31, 2021 at 09:27:10AM +1100, Dave Horsfall wrote:
> > On Tue, 26 Jan 2021, Greg A. Woods wrote:
> >
> > >The lore I was told at the time was that you alwasy ran three and that
> it
> > >didn't matter if they were all on the same line with semicolons or not
> > >because of the very fact that the second one would block.
> >
> > What I was taught was:
> >
> >       % sync
> >       % sync
> >       % sync
> >
> > and never:
> >
> >       % sync; sync; sync
> >
> > The theory was that by waiting for the shell prompt each time, it gave
> the
> > buffer pool enough time to be flushed.
>
> I was taught the exact same thing and for the same reasons.
>

Yes. There's no actual code in the System V or earlier kernels to block any
of the sync calls. At least none I could find in examining all the 'leaked
sources' I could find on the internet.

sync just schedules I/O. There's nothing in the system call that waits for
anything: it just goes through the list of all non-busy buffers calling the
driver's strategy routine and marking the buffer as busy.

Later systems, like SunOS did have a wait, but that was a later addition,
done a half a dozen different ways. BSD had one that waited for a while,
but gave up after some tens of seconds...

Warner

[-- Attachment #2: Type: text/html, Size: 1896 bytes --]

Re: [TUHS] reboot(2) system call

[Greg A. Woods]

[-- Attachment #1: Type: text/plain, Size: 1499 bytes --]

At Sun, 31 Jan 2021 09:27:10 +1100 (EST), Dave Horsfall <dave@horsfall.org> wrote:
Subject: Re: [TUHS] reboot(2) system call
>
> On Tue, 26 Jan 2021, Greg A. Woods wrote:
>
> > The lore I was told at the time was that you alwasy ran three and
> > that it didn't matter if they were all on the same line with
> > semicolons or not because of the very fact that the second one would
> > block.
>
> What I was taught was:
>
> 	% sync
> 	% sync
> 	% sync
>
> and never:
>
> 	% sync; sync; sync
>
> The theory was that by waiting for the shell prompt each time, it gave
> the buffer pool enough time to be flushed.

If waiting was the true reason, then any sane person would have put a
sleep in there instead so as to avoid any variance in typing (and
terminal) speed.

On at least a large number of old systems I've used either the first or
the second invocation did block and not return if there were still any
dirty blocks it made the sync() call.  It was trivial to see that the
system was busy writing while one waited for the shell prompt to
re-appear if one could see the disk activity lights (or hear them) from
the console, as was usually easy to do on desktop systems.

Since many of those old systems I used were Xenix of one flavour or
another, perhaps it was only those that waited for sync I/O to complete.

--
					Greg A. Woods <gwoods@acm.org>

Kelowna, BC     +1 250 762-7675           RoboHack <woods@robohack.ca>
Planix, Inc. <woods@planix.com>     Avoncote Farms <woods@avoncote.ca>

[-- Attachment #2: OpenPGP Digital Signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [TUHS] reboot(2) system call

[Warner Losh]

[-- Attachment #1: Type: text/plain, Size: 1830 bytes --]

On Mon, Feb 1, 2021, 7:22 PM Greg A. Woods <woods@robohack.ca> wrote:

> At Sun, 31 Jan 2021 09:27:10 +1100 (EST), Dave Horsfall <dave@horsfall.org>
> wrote:
> Subject: Re: [TUHS] reboot(2) system call
> >
> > On Tue, 26 Jan 2021, Greg A. Woods wrote:
> >
> > > The lore I was told at the time was that you alwasy ran three and
> > > that it didn't matter if they were all on the same line with
> > > semicolons or not because of the very fact that the second one would
> > > block.
> >
> > What I was taught was:
> >
> >       % sync
> >       % sync
> >       % sync
> >
> > and never:
> >
> >       % sync; sync; sync
> >
> > The theory was that by waiting for the shell prompt each time, it gave
> > the buffer pool enough time to be flushed.
>
> If waiting was the true reason, then any sane person would have put a
> sleep in there instead so as to avoid any variance in typing (and
> terminal) speed.
>
> On at least a large number of old systems I've used either the first or
> the second invocation did block and not return if there were still any
> dirty blocks it made the sync() call.  It was trivial to see that the
> system was busy writing while one waited for the shell prompt to
> re-appear if one could see the disk activity lights (or hear them) from
> the console, as was usually easy to do on desktop systems.
>
> Since many of those old systems I used were Xenix of one flavour or
> another, perhaps it was only those that waited for sync I/O to complete.
>

Would be nice to know which one so I can go check. I've not seen leaked
xenix code though, so it may be possible.

Warner

--
>                                         Greg A. Woods <gwoods@acm.org>
>
> Kelowna, BC     +1 250 762-7675           RoboHack <woods@robohack.ca>
> Planix, Inc. <woods@planix.com>     Avoncote Farms <woods@avoncote.ca>
>

[-- Attachment #2: Type: text/html, Size: 3037 bytes --]

Re: [TUHS] reboot(2) system call

[Greg A. Woods]

[-- Attachment #1: Type: text/plain, Size: 2107 bytes --]

At Mon, 1 Feb 2021 19:30:47 -0700, Warner Losh <imp@bsdimp.com> wrote:
Subject: Re: [TUHS] reboot(2) system call
>
> Would be nice to know which one so I can go check. I've not seen leaked
> xenix code though, so it may be possible.

Well the first versions I used were on 286 machines, and then 386.

I think there may be binaries available in the darker corners (maybe
even on archive.org) if one wanted to test empirically on either
emulation or real hardware.  Maybe someday I could do that, but not
soon.

On a related note, one of the weird things about Unix System V, from the
beginning more or less, and right up to and including the last SysVr4,
the shutdown scripts use "sync; sync; sync" all on one line like that
(on r4 they use the full path to "/sbin/sync", but still three times).
I think on some versions the subsequent call will block briefly while
the first call finishes scheduling buffer writes (i.e. the call will
wait instead of just returning if another process is in the critical
section), they all still just schedule async writes.

If I'm not mistaken though calling "sync" in any way at all shouldn't
normally be necessary during a proper shutdown as all filesystems,
including the root fs, will be unmounted during the process, thus
forcing all dirty buffers to be flushed first, and this is in fact the
case from SysVr3 and on (i.e. after uadmin(2) with A_SHUTDOWN was
introduced).  Also, on SysVr4 the fsflush daemon runs as a kernel daemon
process, so if I understand correctly it isn't killed during shutdown or
in single user mode, thus it'll still be doing the equivalent of sync(2)
right up to the end.

On the other hand in the particular SysVr4-i386 version I have the code
that would actually wait for all async buffer writes during umounts,
i.e. in bdwait(), is commented out and replaced by a "delay(200)".  I
can't imagine why, nor can I imagine trying to debug that without source!

--
					Greg A. Woods <gwoods@acm.org>

Kelowna, BC     +1 250 762-7675           RoboHack <woods@robohack.ca>
Planix, Inc. <woods@planix.com>     Avoncote Farms <woods@avoncote.ca>

[-- Attachment #2: OpenPGP Digital Signature --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [TUHS] reboot(2) system call

[Dave Horsfall]

On Mon, 1 Feb 2021, Greg A. Woods wrote:

> If waiting was the true reason, then any sane person would have put a 
> sleep in there instead so as to avoid any variance in typing (and 
> terminal) speed.

I dunno; that's merely what I was taught in the early Unix days and it got 
ingrained into me.

Nowadays I just issue "shutdown -[r|h] now" (I'm the only user)...

More Unix lore: sleep(1) could return straight away at one time (the 
granularity was 1 second).

-- Dave, who remembers when sleep(3) used to be sleep(2)