Bug: Losing .zsh_history

Bug: Losing .zsh_history

[Dominik Vogt]

Due to some hardware problem with the display adapter my box
sometimes fails to show anything on the screen but just emits some
error beeps while it boots.  Since I cannot log in and stop the
machine without the screen, I can only switch off the power.

It has now happened twice that in the wake of the hard switch-off,
~/.zsh_history got completely erased, replaced with all null
characters, but keeping its size (or at least nor being truncated
to zero size).  This is on Linux 4.x (5.x not being compatible
with the very old hardware) using ext 4 without journal, running
Devuan 4.  Losing history is a real big pain because for me
because it's one huge database of past commands and syntax (2
million lines).

(However, I'm not sure why ~/.zsh_history is modified at all
because it should be touched only when logging in?)

I wonder if this situation could be prevented by changing the code
that writes the history file at exit.  Is truncating the file to
zero size part of the procedure?  When I need to write a file in a
safe manner, the procedure normally looks like this:

 * Create file.new
 * Rename file to file.old
 * Rename file.new to file
 (* Possibly sync filesystem)
 * Erase file.old

--

 $ zsh --version
 zsh 5.8 (x86_64-debian-linux-gnu)

Ciao

Dominik ^_^  ^_^

--

Dominik Vogt

Re: Bug: Losing .zsh_history

[Eric Cook]

On 10/5/22 15:55, Dominik Vogt wrote:
> Due to some hardware problem with the display adapter my box
> sometimes fails to show anything on the screen but just emits some
> error beeps while it boots.  Since I cannot log in and stop the
> machine without the screen, I can only switch off the power.
>
> It has now happened twice that in the wake of the hard switch-off,
> ~/.zsh_history got completely erased, replaced with all null
> characters, but keeping its size (or at least nor being truncated
> to zero size).

Is it actually erased or the last line is full of null bytes?

Re: Bug: Losing .zsh_history

[Dominik Vogt]

On Wed, Oct 05, 2022 at 04:01:01PM -0400, Eric Cook wrote:
> On 10/5/22 15:55, Dominik Vogt wrote:
> > Due to some hardware problem with the display adapter my box
> > sometimes fails to show anything on the screen but just emits some
> > error beeps while it boots.  Since I cannot log in and stop the
> > machine without the screen, I can only switch off the power.
> >
> > It has now happened twice that in the wake of the hard switch-off,
> > ~/.zsh_history got completely erased, replaced with all null
> > characters, but keeping its size (or at least nor being truncated
> > to zero size).
>
> Is it actually erased or the last line is full of null bytes?

I have no way to reproduce that at will and it has only happened
twice in the last ten years.  (Since the first time I make a copy
once in a while.)  The second time is several weeks ago.

So, I can say with certainty that when this happened I used vim to
look at the history file in a 80x24 terminal window and that
showed all "^@"s (i.e. 23 lines * 40 nulls = 920 nulls).  I'm not
sure whether I looked at any other portion of the file, but
normally I'd check the last line too.  I definitely didn't check
what was in between and definitely didn't see anything but nulls.
(The last line may be identical to the first line.)

Ciao

Dominik ^_^  ^_^

--

Dominik Vogt

Re: Bug: Losing .zsh_history

[Dominik Vogt]

On Wed, Oct 05, 2022 at 08:55:26PM +0100, Dominik Vogt wrote:
> Due to some hardware problem with the display adapter my box
> sometimes fails to show anything on the screen but just emits some
> error beeps while it boots.  Since I cannot log in and stop the
> machine without the screen, I can only switch off the power.
>...

> using ext 4 without journal, running

P.S.:  With the "discard" option on an SSD:

-- ext4 manpage --
       discard/nodiscard
              Controls whether ext4 should issue discard/TRIM commands to  the
              underlying  block  device when blocks are freed.  This is useful
              for SSD devices and sparse/thinly-provisioned LUNs,  but  it  is
              off by default until sufficient testing has been done.
--

Ciao

Dominik ^_^  ^_^

--

Dominik Vogt

Re: Bug: Losing .zsh_history

[Wesley]

On 10/5/22 16:20, Dominik Vogt wrote:
> On Wed, Oct 05, 2022 at 04:01:01PM -0400, Eric Cook wrote:
>> On 10/5/22 15:55, Dominik Vogt wrote:
>>> Due to some hardware problem with the display adapter my box
>>> sometimes fails to show anything on the screen but just emits some
>>> error beeps while it boots.  Since I cannot log in and stop the
>>> machine without the screen, I can only switch off the power.
>>>
>
> So, I can say with certainty that when this happened I used vim to
> look at the history file in a 80x24 terminal window and that
> showed all "^@"s (i.e. 23 lines * 40 nulls = 920 nulls).  I'm not
> sure whether I looked at any other portion of the file, but
> normally I'd check the last line too.  I definitely didn't check
> what was in between and definitely didn't see anything but nulls.
> (The last line may be identical to the first line.)

This might be interesting to read:
https://unix.stackexchange.com/a/273869

What are your history options btw?

Cheers,
Wesley

--
Wesley Schwengle

Re: Bug: Losing .zsh_history

[Dominik Vogt]

On Wed, Oct 05, 2022 at 09:05:44PM +0000, Wesley wrote:
> On 10/5/22 16:20, Dominik Vogt wrote:
> > On Wed, Oct 05, 2022 at 04:01:01PM -0400, Eric Cook wrote:
> >> On 10/5/22 15:55, Dominik Vogt wrote:
> >>> Due to some hardware problem with the display adapter my box
> >>> sometimes fails to show anything on the screen but just emits some
> >>> error beeps while it boots.  Since I cannot log in and stop the
> >>> machine without the screen, I can only switch off the power.
> >>>
> >
> > So, I can say with certainty that when this happened I used vim to
> > look at the history file in a 80x24 terminal window and that
> > showed all "^@"s (i.e. 23 lines * 40 nulls = 920 nulls).  I'm not
> > sure whether I looked at any other portion of the file, but
> > normally I'd check the last line too.  I definitely didn't check
> > what was in between and definitely didn't see anything but nulls.
> > (The last line may be identical to the first line.)
>
> This might be interesting to read:
> https://unix.stackexchange.com/a/273869

Interesting.  Actually, my .zsh_history is always "full", so every
time a shell updates the history some of the oldest entries are
removed, i.e. there's normally not a simple append operation.
Appending won't destroy the first line of the file.

> What are your history options btw?

Starting with the default, these are the changes:

  setopt APPEND_HISTORY
  unsetopt BANG_HIST
  setopt HIST_IGNORE_DUPS
  setopt HIST_FIND_NO_DUPS
  setopt HIST_IGNORE_SPACE
  setopt HIST_VERIFY
  setopt HIST_EXPIRE_DUPS_FIRST
  HISTSIZE=100500
  HISTFILE=~/.zsh_history
  SAVEHIST=100000
  setopt HIST_REDUCE_BLANKS

Ciao

Dominik ^_^  ^_^

--

Dominik Vogt

Re: Bug: Losing .zsh_history

[Bart Schaefer]

On Wed, Oct 5, 2022 at 12:55 PM Dominik Vogt <dominik.vogt@gmx.de> wrote:
>
> I wonder if this situation could be prevented by changing the code
> that writes the history file at exit.  Is truncating the file to
> zero size part of the procedure?  When I need to write a file in a
> safe manner, the procedure normally looks like this:
>
>  * Create file.new
>  * Rename file to file.old
>  * Rename file.new to file
>  (* Possibly sync filesystem)
>  * Erase file.old

zsh is not going to sync the filesystem for you.  Other than that,
though, the default behavior should be close to what you just
described, see the HIST_SAVE_BY_COPY option (which you should make
sure has not been unset).

You might also look at the setting of the HIST_FCNTL_LOCK option.

Re: Bug: Losing .zsh_history

[Bart Schaefer]

On Wed, Oct 5, 2022 at 2:24 PM Dominik Vogt <dominik.vogt@gmx.de> wrote:
>
> On Wed, Oct 05, 2022 at 09:05:44PM +0000, Wesley wrote:
> >
> > This might be interesting to read:
> > https://unix.stackexchange.com/a/273869
>
> Interesting.

Somethig like that is the most likely culprit:  The filesystem has
allocated space for the buffered content to be written out, but
cutting the power brings the system down before the buffer is flushed
to disk.  This is most often seen on "soft-mounted" NFS filesystems
that use delayed writes, but it's certainly possible to happen locally
on abrupt loss of power.

If avoiding FS lossage is important enough to you, you might try
getting a battery power supply unit and configure the power daemon to
do an orderly shutdown if the PS switches to battery.  Then instead of
powering off the machine, unplug the PS.

If the problem merely seems to be that a shell you can't see has
started up and is trying to update the history when it shuts down, you
could add something to .zlogin to make a backup copy of the history
when the shell starts.  The circumstances you've described are odd,
though, because I wouldn't expect a shell to have any opportunity to
begin processing its shutdown when power is cut.  If the same problem
that causes the blank display has already killed the shell, there may
be a connection between the failing display and failing disk
operations.

Re: Bug: Losing .zsh_history

[Dominik Vogt]

On Wed, Oct 05, 2022 at 03:16:27PM -0700, Bart Schaefer wrote:
> On Wed, Oct 5, 2022 at 12:55 PM Dominik Vogt <dominik.vogt@gmx.de> wrote:
> >
> > I wonder if this situation could be prevented by changing the code
> > that writes the history file at exit.  Is truncating the file to
> > zero size part of the procedure?  When I need to write a file in a
> > safe manner, the procedure normally looks like this:
> >
> >  * Create file.new
> >  * Rename file to file.old
> >  * Rename file.new to file
> >  (* Possibly sync filesystem)
> >  * Erase file.old
>
> zsh is not going to sync the filesystem for you.

Of course.

> Other than that,
> though, the default behavior should be close to what you just
> described, see the HIST_SAVE_BY_COPY option (which you should make
> sure has not been unset).

> You might also look at the setting of the HIST_FCNTL_LOCK option.

Both at default:

  $ set -o
  nohistsavebycopy      off
  histfcntllock         off

So, the question is how this can happen at all.

Ciao

Dominik ^_^  ^_^

--

Dominik Vogt

Re: Bug: Losing .zsh_history

[Dominik Vogt]

On Wed, Oct 05, 2022 at 03:31:07PM -0700, Bart Schaefer wrote:
> Somethig like that is the most likely culprit:  The filesystem has
> allocated space for the buffered content to be written out, but
> cutting the power brings the system down before the buffer is flushed
> to disk.  This is most often seen on "soft-mounted" NFS filesystems
> that use delayed writes, but it's certainly possible to happen locally
> on abrupt loss of power.

> If avoiding FS lossage is important enough to you ...

Not at all, I just don't want to lose the shell history.

> If the problem merely seems to be that a shell you can't see has
> started up and is trying to update the history when it shuts down, you
> could add something to .zlogin to make a backup copy of the history
> when the shell starts.

Yes, but that would considerably increase SSD wear (see other mail
thread); so I do it manually once in a while.

> The circumstances you've described are odd,
> though, because I wouldn't expect a shell to have any opportunity to
> begin processing its shutdown when power is cut.  If the same problem
> that causes the blank display has already killed the shell, there may
> be a connection between the failing display and failing disk
> operations.

Actually, what's really strange is that the shell should not be
started at all in this situation:  The display only ever fails
when powering on; once it works it keeps working.  In that
situation I don't log in because I can't see anything.  I'm not
even sure getty starts up at all.  So, why does any zsh run at
all if I can't even log in?

Ciao

Dominik ^_^  ^_^

--

Dominik Vogt

Re: Bug: Losing .zsh_history

[Bart Schaefer]

On Wed, Oct 5, 2022 at 4:03 PM Dominik Vogt <dominik.vogt@gmx.de> wrote:
>
> Actually, what's really strange is that the shell should not be
> started at all in this situation:  The display only ever fails
> when powering on; once it works it keeps working.

It's possible that the window manager has started even though it can't
use the display, so you can't see it.  Although if you normally are
presented with a login screen, that's probably not it.

Is it possible that a shell is starting on a different virtual
terminal (console)?

Incidentally, everything I've seen with publication dates in the last
7 years says that it's no longer worthwhile to worry about SSD wear
from repeated writes.  Any recently-manufactured SSD will probably
outlast an HDD.  I suppose you did mention that you have an old
machine, so maybe you're trying to preserve a very old drive?

Re: Bug: Losing .zsh_history

[Peter Stephenson]

Deliberately deleting the previous history in this thread (tee hee)
because I want to get a higher level picture...

We seem to be homing in on cases where the shell is involved in
misbehaviour but doesn't get enough information or warning that
something's up that it can do anything about it.  Is that fair, or is
there some aspect that might give it a head's up that writing history
could be dangerous?

pws

Re: Bug: Losing .zsh_history

[Dominik Vogt]

On Wed, Oct 05, 2022 at 09:34:15PM -0700, Bart Schaefer wrote:
> On Wed, Oct 5, 2022 at 4:03 PM Dominik Vogt <dominik.vogt@gmx.de> wrote:
> > Actually, what's really strange is that the shell should not be
> > started at all in this situation:  The display only ever fails
> > when powering on; once it works it keeps working.
>
> It's possible that the window manager has started even though it can't
> use the display, so you can't see it.

No, the box boots to the console and X is started from the command
line.

> Is it possible that a shell is starting on a different virtual
> terminal (console)?

Can't think of a way to get into an _interactive_ zsh without typing
a password sowmewhere.  The loss must have occured when shutting
down the machine in the previous session.

> Incidentally, everything I've seen with publication dates in the last
> 7 years says that it's no longer worthwhile to worry about SSD wear
> from repeated writes.  Any recently-manufactured SSD will probably
> outlast an HDD.

> I suppose you did mention that you have an old
> machine, so maybe you're trying to preserve a very old drive?

Yeah.  But I've just taken a look at the output of the smartmon
tools and it looks like even the ancient disk will be safe for
many more years.  Leaving 1/3 of the disk space unpartitioned
seems to be good enough for the rest of my life, without having to
worry about additional measures.

Ciao

Dominik ^_^  ^_^

--

Dominik Vogt

Re: Bug: Losing .zsh_history

[Dominik Vogt]

On Wed, Oct 05, 2022 at 03:16:27PM -0700, Bart Schaefer wrote:
> On Wed, Oct 5, 2022 at 12:55 PM Dominik Vogt <dominik.vogt@gmx.de> wrote:
> > I wonder if this situation could be prevented by changing the code
> > that writes the history file at exit.  Is truncating the file to
> > zero size part of the procedure?  When I need to write a file in a
> > safe manner, the procedure normally looks like this:
> >
> >  * Create file.new
> >  * Rename file to file.old
> >  * Rename file.new to file
> >  (* Possibly sync filesystem)
> >  * Erase file.old
>
> zsh is not going to sync the filesystem for you.  Other than that,
> though, the default behavior should be close to what you just
> described, see the HIST_SAVE_BY_COPY option (which you should make
> sure has not been unset).

So, where would one find the backup file if shutdown was
interrupted.  THere was nothing obvious like ~/.zsh_history.bak or
something.

Ciao

Dominik ^_^  ^_^

--

Dominik Vogt

Re: Bug: Losing .zsh_history

[Dominik Vogt]

On Thu, Oct 06, 2022 at 10:40:04AM +0100, Peter Stephenson wrote:
> Deliberately deleting the previous history in this thread (tee hee)
> because I want to get a higher level picture...
>
> We seem to be homing in on cases where the shell is involved in
> misbehaviour but doesn't get enough information or warning that
> something's up that it can do anything about it.  Is that fair, or is
> there some aspect that might give it a head's up that writing history
> could be dangerous?

With the discussion and the available my hypothesis of whats
happening:

On an unjournaled ext4 (on an SSD; with "discard" option)

1) System starts shutdown
2) Shell is told to terminate.
3) Shell writes new history into separate file.
   This is cached by the disk, the disk drivers or the fs layer.
4) Shell moves new history over old one.
   This is cached by the disk, the disk drivers or the fs layer.
5) Because the filesystem is not journaled, the changes from
   the previous step are written to disk in random order.
6) Power loss occurs while the new directory entries have already
   been written to disk, but the new file contents have not.
   (The box is attached to a switchable multi socket adapter.)

This wouldn't be the fault of the shell.  Still, why did this
happen to the history files twice, but not to the browser config
files which are also written at shutdown?  Is there anything the
shell could do to prevent this or make it less likely?  More
fsync()s?

Ciao

Dominik ^_^  ^_^

--

Dominik Vogt

Re: Bug: Losing .zsh_history

[Mikael Magnusson]

On 10/5/22, Dominik Vogt <dominik.vogt@gmx.de> wrote:
> Due to some hardware problem with the display adapter my box
> sometimes fails to show anything on the screen but just emits some
> error beeps while it boots.  Since I cannot log in and stop the
> machine without the screen, I can only switch off the power.
>
> It has now happened twice that in the wake of the hard switch-off,
> ~/.zsh_history got completely erased, replaced with all null
> characters, but keeping its size (or at least nor being truncated
> to zero size).  This is on Linux 4.x

Since this is linux, and assuming the problem is only the display and
you haven't fully locked the kernel, you could enable the sysrq magic
keys[1] if they aren't already, and instead of doing a hard power off,
press altgr(right alt) + printscreen + s u s b (wait a little between
each), the u will remount all mounts readonly, the s issues a sync,
and b will soft-reboot.

[1] See sysrq.txt in the kernel source

-- 
Mikael Magnusson