issues with saving history to file

issues with saving history to file

[Vincent Lefevre]

With zsh 5.9:

cventin:~> zsh -f
cventin% HISTFILE=~/.histfile
cventin% HISTSIZE=1000
cventin% SAVEHIST=1000
cventin% setopt APPEND_HISTORY
cventin% true
cventin% exit
cventin:~> ls ~/.histfile
ls: cannot access '/home/vlefevre/.histfile': No such file or directory

However, if I use INC_APPEND_HISTORY instead of APPEND_HISTORY, the
history is saved.

According to the zsh(1) man page:

  Note also that the RCS option affects the saving of history files,
  i.e. if RCS is unset when the shell exits, no history file will be
  saved.

But the difference between INC_APPEND_HISTORY and APPEND_HISTORY is
not documented concerning this point.

Also, I do not see the point of not saving the history when RCS is set.
IMHO, HISTFILE should just be unset by default; but if the user sets
this parameter, I suppose that the goal is to save the history.

Without INC_APPEND_HISTORY, the man page doesn't document when
precisely the history is saved. For instance, what happens if one
has a "print -s ..." in the .zlogout file?

Moreover, with INC_APPEND_HISTORY, "print -s" does not put the entry
in the history file immediately, only after the next accept-line.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: issues with saving history to file

[Bart Schaefer]

On Wed, Mar 13, 2024 at 8:05 AM Vincent Lefevre <vincent@vinc17.net> wrote:
>
> With zsh 5.9:
>
> cventin:~> zsh -f
[set up history parameters and options]
> cventin% exit
> cventin:~> ls ~/.histfile
> ls: cannot access '/home/vlefevre/.histfile': No such file or directory
>
> However, if I use INC_APPEND_HISTORY instead of APPEND_HISTORY, the
> history is saved.

This is because history is being incrementally saved (as the option
name indicates) rather than being held only in memory until the shell
is preparing to exit.  APPEND_HISTORY just means that if the history
is saved, it will be added to the history file (up to $SAVEHIST)
instead of overwriting it.

> According to the zsh(1) man page:
>
>   Note also that the RCS option affects the saving of history files,
>   i.e. if RCS is unset when the shell exits, no history file will be
>   saved.
>
> But the difference between INC_APPEND_HISTORY and APPEND_HISTORY is
> not documented concerning this point.

Congratulations, you've found another place where documentation of
feature X was not updated upon the introduction of newer feature Y.
The doc for history is littered with these because it's been around
longer than just about anything: HISTFILE is still documented as only
being used at shell exit.

Someone else can patch it this time.

> Also, I do not see the point of not saving the history when RCS is set.

It is saving when RCS is set, and not when RCS is not set (no_RCs).

This is done because of the handling of $SAVEHIST, not to mention the
various duplicate handling options.  When no_RCs, the history has
(presumably) not been read in at startup in the first place, so it
won't have been merged with the interactive state.  If written out
when never read in, the $SAVEHIST size limit could cause contents to
be lost that otherwise would not be.  (Frobbing the RCs option during
later shell operation is effectively undefined behavior, too difficult
to keep track of.)

> IMHO, HISTFILE should just be unset by default; but if the user sets
> this parameter, I suppose that the goal is to save the history.

The goal might be to read the history, not to write it.

In any case we've had several go-rounds on this (and on the "right
way" to set HISTSIZE vs. SAVEHIST) without coming to any consensus
differing from the existing behavior.

> Without INC_APPEND_HISTORY, the man page doesn't document when
> precisely the history is saved. For instance, what happens if one
> has a "print -s ..." in the .zlogout file?

It'd expect it to be lost, because the history is saved before
.zlogout is read.   Try it and let us know.  Maybe it dumps core.

> Moreover, with INC_APPEND_HISTORY, "print -s" does not put the entry
> in the history file immediately, only after the next accept-line.

"print -s" doesn't write to the HISTFILE and isn't documented that
way.  It explicitly says "Place the results in the history list".  The
"history list" is the internal shell history, not the file.  The file
is updated from the internal list either at shell exit or
(incappendhistory) when a line is entered (i.e., conceptually upon
accept-line, though in practice upon entering a complete command since
updating at PS2 would leave an incomplete entry).  You can use the
zshaddhistory hook to confirm this.

Re: issues with saving history to file

[Vincent Lefevre]

On 2024-03-13 21:52:44 -0700, Bart Schaefer wrote:
> On Wed, Mar 13, 2024 at 8:05 AM Vincent Lefevre <vincent@vinc17.net> wrote:
> > IMHO, HISTFILE should just be unset by default; but if the user sets
> > this parameter, I suppose that the goal is to save the history.
> 
> The goal might be to read the history, not to write it.

To read the history? The zshparam(1) man page just says "to save":

  HISTFILE
      The file to save the history in when an interactive shell exits.
      If unset, the history is not saved.

And when I do

cventin:~> zsh -f
cventin% HISTFILE=~/.histfile

I just get the "HISTFILE=~/.histfile" in the history, while
~/.histfile already contains various commands.

> > Without INC_APPEND_HISTORY, the man page doesn't document when
> > precisely the history is saved. For instance, what happens if one
> > has a "print -s ..." in the .zlogout file?
> 
> It'd expect it to be lost, because the history is saved before
> .zlogout is read.   Try it and let us know.  Maybe it dumps core.

It is added to the history, but not saved. With

ls -l .histfile
sleep 1
print -s foo1
print -s foo2
print -s foo3
history
ls -l .histfile

in my .zlogout file, I get

-rw------- 1 vlefevre vlefevre 292 2024-03-15 13:17:11 .histfile
[...]
   27  foo1
   28  foo2
   29  foo3
-rw------- 1 vlefevre vlefevre 292 2024-03-15 13:17:11 .histfile

and indeed, .histfile does not contain these lines.

> > Moreover, with INC_APPEND_HISTORY, "print -s" does not put the entry
> > in the history file immediately, only after the next accept-line.
> 
> "print -s" doesn't write to the HISTFILE and isn't documented that
> way.  It explicitly says "Place the results in the history list".  The
> "history list" is the internal shell history, not the file.  The file
> is updated from the internal list either at shell exit or
> (incappendhistory) when a line is entered (i.e., conceptually upon
> accept-line, though in practice upon entering a complete command since
> updating at PS2 would leave an incomplete entry).

The documentation is not explicit.

> You can use the zshaddhistory hook to confirm this.

The documentation of the zshaddhistory hook is not clear. It says

  Executed when a history line has been read interactively,

But this is the case for any (possibly empty!) command line,
not just lines read from the history.

Then

  but before it is executed.

This is wrong. Empty command lines (which will not be executed) are
also handled by the zshaddhistory hook (contrary to the preexec hook).

And as a consequence, its example

   zshaddhistory() {
     print -sr -- ${1%%$'\n'}
     fc -p .zsh_local_history
   }

has the effect to put empty command lines in the history, which
is not the usual behavior.

Moreover, the "fc -p .zsh_local_history" does not seem to work:
no .zsh_local_history file is created.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: issues with saving history to file

[Bart Schaefer]

On Fri, Mar 15, 2024 at 5:47 AM Vincent Lefevre <vincent@vinc17.net> wrote:
>
> On 2024-03-13 21:52:44 -0700, Bart Schaefer wrote:
> > On Wed, Mar 13, 2024 at 8:05 AM Vincent Lefevre <vincent@vinc17.net> wrote:
> > > IMHO, HISTFILE should just be unset by default; but if the user sets
> > > this parameter, I suppose that the goal is to save the history.
> >
> > The goal might be to read the history, not to write it.
>
> To read the history?  The zshparam(1) man page just says "to save":
>
>   HISTFILE
>       The file to save the history in when an interactive shell exits.

See also the part about "when exits" which is clearly also incorrect.
This entry is outdated.  The shell also initializes the history from
this file at startup.

I was differentiating
  zsh -f
  % setopt RCS
from
  zsh
  % setopt no_RCS

In the former case, the history is not read, but it is written.  In
the latter case it is read, but not written.  The point being that
merely setting HISTFILE does not imply that the goal is to save to it.
Further, setting SAVEHIST=0 will truncate the file in the former case
and leave it alone in the latter case.  There's no single setting that
covers all cases, which is part of the reason we've so far had trouble
agreeing on default behavior.

If anyone wants to have a go at cleaning up all these documentation
inaccuracies, please do.