SSD-friendly history

SSD-friendly history

[Dominik Vogt]

The zsh history file is probably the one file that causes the most
wear of the SSD drive:

  setopt APPEND_HISTORY
  setopt HIST_IGNORE_DUPS
  setopt HIST_EXPIRE_DUPS_FIRST
  HISTSIZE=100500
  SAVEHIST=100000

I.e. every time a zsh writes the history file when it exits, it
writes a new copy of the history file with some lines removed from
the top and some added at the end.  (The history file is about 4
MB big resulting in about 100 MB to 1 GB writes every day.)

I'd really like to have a mechanism that appends to the history
until some maximum size is reached and then truncates it to some
much smaller size, e.g.

  HISTSIZE_MAX=200000
  HISTSIZE_TRUNCATE_TO=100000

So the history would only be rewritten every couple of months.
(This probably conflicts with the DUPS options.)

Ciao

Dominik ^_^  ^_^

--

Dominik Vogt

Re: SSD-friendly history

[Bart Schaefer]

On Wed, Oct 5, 2022 at 2:34 PM Dominik Vogt <dominik.vogt@gmx.de> wrote:
>
> I'd really like to have a mechanism that appends to the history
> until some maximum size is reached and then truncates it to some
> much smaller size

Setting INC_APPEND_HISTORY will cause the file to be rewritten only
when it reaches 20% larger than SAVEHIST.  There's no option to make
that greater than 20% at this point.

However, this would be a reason to make SAVEHIST a lot larger than
HISTSIZE.  If you don't mind using size-in-bytes instead of
size-in-lines, you could then add a fast check to .zlogout to set
SAVEHIST=$HISTSIZE whenever $HISTFILE is large enough.

(( $(zstat +size $HISTFILE) > 2000000 )) && SAVEHIST=$HISTSIZE

or similar.

Re: SSD-friendly history

[Dominik Vogt]

On Wed, Oct 05, 2022 at 03:50:43PM -0700, Bart Schaefer wrote:
> On Wed, Oct 5, 2022 at 2:34 PM Dominik Vogt <dominik.vogt@gmx.de> wrote:
> > I'd really like to have a mechanism that appends to the history
> > until some maximum size is reached and then truncates it to some
> > much smaller size
>
> Setting INC_APPEND_HISTORY will cause the file to be rewritten only
> when it reaches 20% larger than SAVEHIST.  There's no option to make
> that greater than 20% at this point.
>
> However, this would be a reason to make SAVEHIST a lot larger than
> HISTSIZE.

Disk space is not an issue.  So, if I set

  SAVEHIST=120000
  HISTSIZE=100000

Then only 100.000 lines are loaded when the shell starts?

--

But how does this interact with duplicate handling, especially
with HIST_EXPIRE_DUPS_FIRST?

Ciao

Dominik ^_^  ^_^

--

Dominik Vogt

Re: SSD-friendly history

[Bart Schaefer]

On Wed, Oct 5, 2022 at 4:19 PM Dominik Vogt <dominik.vogt@gmx.de> wrote:
>
> Disk space is not an issue.  So, if I set
>
>   SAVEHIST=120000
>   HISTSIZE=100000
>
> Then only 100.000 lines are loaded when the shell starts?

The shell will read all 120000 lines and keep the last (most recent)
100000 of them.  So there's a slight startup speed penalty.

> But how does this interact with duplicate handling, especially
> with HIST_EXPIRE_DUPS_FIRST?

Duplicate handling all happens on the in-memory history.   This means
anything that suppresses duplicates from the HISTFILE (such as
HIST_SAVE_NO_DUPS) will end up reducing it to HISTSIZE lines even if
SAVEHIST is larger.  The doc thus recommends setting HISTSIZE larger
than SAVEHIST in the case of HIST_EXPIRE_DUPS_FIRST so that you don't
immediately begin discarding your saved history as commands are
entered (which might be important if e.g. you use
accept-and-infer-next-history a lot).  However, if you use
INC_APPEND_HISTORY, duplicates are removed from HISTFILE only when
that file is rewritten (at SAVEHIST + 20% while the shell is
incrementally appending, or at shell exit).  So in that case having
SAVEHIST larger than HISTSIZE can be useful.

(These are details that didn't come up in an earlier zsh-workers
thread about default values for SAVEHIST and HISTSIZE and whether/when
it makes sense for them to differ.)