[BUG] APPEND_HISTORY leads to complete reread + rewrite of histfile

[BUG] APPEND_HISTORY leads to complete reread + rewrite of histfile

[Zeshy]

Hi all!

In search of the reason why zsh (5.4.2 and fairly recent git) is
so much slower reading/writing larger history files (13 MB, 330k lines)
than bash (4.4.12), I think I found a bug. If I missed something (like
some option), and this is not a bug, but just misconfiguration, or if
you need more information, please let me know.


When configured to *append* the history on exit with the following
config:

---------------- BEGIN .zshrc -----------------
#!/usr/bin/zsh
HISTFILE="/var/tmp/.histfile"
HISTSIZE=9223372036854775807 # LONG_MAX (limits.h)
SAVEHIST=9223372036854775807 # LONG_MAX (limits.h)
setopt append_history
setopt extended_history
unsetopt hist_save_by_copy
unsetopt inc_append_history_time
---------------- END .zshrc -----------------

zsh (with "-d" or without) seems to do the following:

Step 1) append to history file
Step 2) read complete history file
Step 3) write complete history file

This seems to be invariant to
- option HIST_SAVE_BY_COPY,
- option INC_APPEND_HISTORY_TIME,
- values of HISTSIZE and SAVEHIST (tried with 500000 for both).

With "unsetopt APPEND_HISTORY", only step 3 is executed, as expected.


I interspersed some debug printf's in Src/hist.c and saved the output
of simply starting zsh and hitting Ctrl-D (exit). Absolute and relative
times are in milliseconds.

URLs:

https://pastebin.com/BzJhrr69  .zshrc-appendhistory
https://pastebin.com/ULCt7XcR  .zshrc-noappendhistory
https://pastebin.com/bHTqQu0C  hist.c
https://pastebin.com/QUjZ98Zn  zsh-appendhistory--debug-output.txt
https://pastebin.com/VUpGf6Fa  zsh-noappendhistory--debug-output.txt


Zeshy

Re: [BUG] APPEND_HISTORY leads to complete reread + rewrite of histfile

[Bart Schaefer]

On Wed, Dec 6, 2017 at 11:31 AM, Zeshy <ml.zsh@gmx.es> wrote:
> Hi all!
>
> In search of the reason why zsh (5.4.2 and fairly recent git) is
> so much slower reading/writing larger history files (13 MB, 330k lines)
> than bash (4.4.12), I think I found a bug. If I missed something (like
> some option), and this is not a bug, but just misconfiguration, or if
> you need more information, please let me know.

This happens because of HISTSIZE and the various HIST_*DUP* options.
The only way to do duplicate filtering is to examine the entire past
history, and there's no setting of HISTSIZE that is interpreted as
"unlimited" so rewriting to output only the most recent $HISTSIZE
lines is always performed.

It would probably be possible to optimize for the case of "$HISTSIZE
is larger than the current number of lines and no duplicate removal is
enabled" but right now there is just one rewriting algorithm that
applies (or not) those conditions internal to the loop.

So not technically a bug but not optimized for people who can "afford"
the memory cost to have every shell preserve huge amounts of history.

Re: [BUG] APPEND_HISTORY leads to complete reread + rewrite of histfile

[Bart Schaefer]

On Wed, Dec 6, 2017 at 11:51 AM, Bart Schaefer
<schaefer@brasslantern.com> wrote:
>
> This happens because of HISTSIZE and the various HIST_*DUP* options.

Er,  please read "SAVEHIST" everywhere that I just wrote "HISTSIZE".  Sorry.

Re: [BUG] APPEND_HISTORY leads to complete reread + rewrite of histfile

[Zeshy]

On Wed, 2017-12-06, at 20:51:02 (+0100), Bart Schaefer wrote:
> On Wed, Dec 6, 2017 at 11:31 AM, Zeshy <ml.zsh@gmx.es> wrote:
>
> [Complete write of histfile instead of just append]
> 
> This happens because of HISTSIZE and the various HIST_*DUP* options.
[...]

Thank you very much for your detailled explanation, that at least
reduces my suffering a little! ;-)

I recently involuntarily encountered a Real Life resource sink, so my
half-finished response will have to stay in my postponed-folder for a
while, but I won't forget it.


Zeshy