delay in exitting from zsh shell

delay in exitting from zsh shell

[Eugene Dzhurinsky]

[-- Attachment #1: Type: text/plain, Size: 912 bytes --]

Hello, I'm facing the strange issue - when I run zsh shell and eventually try
to exit from it - it takes some time (3-5 seconds) between I hit Ctrl+D or
type "exit" in shell prompt.

After some experiments I realized that if I remove .histfile - then ZSH exists
immediately, as supposed.

My history file is ~350K of size and settings are:

HISTFILE=~/.histfile
HISTSIZE=10000
SAVEHIST=10000
setopt append_history
setopt inc_append_history
setopt histignorealldups
setopt histignorespace
setopt histreduceblanks
setopt histsavenodups
setopt sharehistory

I checked performance of filesystem where the file is actually stored:

dd if=/dev/zero of=test bs=1m count=10
10485760 bytes transferred in 0.065193 secs (160841962 bytes/sec)

so it operates quite well.

zsh 4.3.12 (amd64-portbld-freebsd9.0)

What may be wrong with ZSH and how to fix it? Thanks!

-- 
Eugene N Dzhurinsky

[-- Attachment #2: Type: application/pgp-signature, Size: 488 bytes --]

Re: delay in exitting from zsh shell

[Rommel Martinez]

You can have something like this:

function bye () {
    if (( ! $#BUFFER )); then
        SAVEHIST=0
        exit
    else
        zle delete-char-or-list
    fi
}

zle -N bye
bindkey "^D" bye


2011/11/10 Eugene Dzhurinsky <jdevelop@gmail.com>:
> Hello, I'm facing the strange issue - when I run zsh shell and eventually try
> to exit from it - it takes some time (3-5 seconds) between I hit Ctrl+D or
> type "exit" in shell prompt.
>
> After some experiments I realized that if I remove .histfile - then ZSH exists
> immediately, as supposed.
>
> My history file is ~350K of size and settings are:
>
> HISTFILE=~/.histfile
> HISTSIZE=10000
> SAVEHIST=10000
> setopt append_history
> setopt inc_append_history
> setopt histignorealldups
> setopt histignorespace
> setopt histreduceblanks
> setopt histsavenodups
> setopt sharehistory
>
> I checked performance of filesystem where the file is actually stored:
>
> dd if=/dev/zero of=test bs=1m count=10
> 10485760 bytes transferred in 0.065193 secs (160841962 bytes/sec)
>
> so it operates quite well.
>
> zsh 4.3.12 (amd64-portbld-freebsd9.0)
>
> What may be wrong with ZSH and how to fix it? Thanks!
>
> --
> Eugene N Dzhurinsky
>



-- 
Rommel M. Martinez
http://rmm.meta.ph

Re: delay in exitting from zsh shell

[Eugene Dzhurinsky]

[-- Attachment #1: Type: text/plain, Size: 417 bytes --]

On Thu, Nov 10, 2011 at 09:14:02AM +0800, Rommel Martinez wrote:
> You can have something like this:
> 
> function bye () {
>     if (( ! $#BUFFER )); then
>         SAVEHIST=0
>         exit
>     else
>         zle delete-char-or-list
>     fi
> }
> 
> zle -N bye
> bindkey "^D" bye

Well, because most time I do have some commands executed - that wouldn't help
much :)

-- 
Eugene N Dzhurinsky

[-- Attachment #2: Type: application/pgp-signature, Size: 488 bytes --]

Re: delay in exitting from zsh shell

[Bart Schaefer]

On Nov 10, 12:20am, Eugene Dzhurinsky wrote:
}
} Hello, I'm facing the strange issue - when I run zsh shell and eventually try
} to exit from it - it takes some time (3-5 seconds) between I hit Ctrl+D or
} type "exit" in shell prompt.
} 
} After some experiments I realized that if I remove .histfile - then
} ZSH exists immediately, as supposed.

This tends to imply that what's taking those 3-5 seconds is searching
your history file for duplicate entries in order to enforce the
hist_save_no_dups option.  Also, inc_append_history may allow the file
to grow to up to 10500 lines for a SAVEHIST of 10000, and those extra
lines will be trimmed at shell exit.

In more detail, when you use inc_append_history and/or share_history
along with hist_save_no_dups, zsh re-reads and de-duplicates the entire
file from disk after it has obtained a lock for it, rather than just
dumping out the history that is already in memory, because it can't
know if some other shell has appended something new to the file before
the lock was obtained.  The speed of the disk is inconsequential to the
CPU expended doing the deduplication.

Re: delay in exitting from zsh shell

[Eugene Dzhurinsky]

[-- Attachment #1: Type: text/plain, Size: 1619 bytes --]

On Wed, Nov 09, 2011 at 07:51:18PM -0800, Bart Schaefer wrote:
> This tends to imply that what's taking those 3-5 seconds is searching
> your history file for duplicate entries in order to enforce the
> hist_save_no_dups option.  Also, inc_append_history may allow the file
> to grow to up to 10500 lines for a SAVEHIST of 10000, and those extra
> lines will be trimmed at shell exit.
> 
> In more detail, when you use inc_append_history and/or share_history
> along with hist_save_no_dups, zsh re-reads and de-duplicates the entire
> file from disk after it has obtained a lock for it, rather than just
> dumping out the history that is already in memory, because it can't
> know if some other shell has appended something new to the file before
> the lock was obtained.  The speed of the disk is inconsequential to the
> CPU expended doing the deduplication.

Okay, I wrote stupid and simple program to de-duplicate lines in history file:

import qualified Data.ByteString.Char8 as C8
import Data.List
import System
import Control.Monad

fixHist = nub . C8.lines

main = do
    (fname:_) <- getArgs
    hl <- liftM fixHist $ C8.readFile fname
    C8.writeFile fname $ C8.unlines hl

function nub is O(n^2) time-complex, and still it takes less than second in
order to read file, optimize it and write it down.

> ./hst .histfile.tst  0.89s user 0.01s system 99% cpu 0.901 total

and I believe that using linked hashtables will improve performance as well.
Am I missing something there if would say that history saving and
de-duplication is suboptimal?

-- 
Eugene N Dzhurinsky

[-- Attachment #2: Type: application/pgp-signature, Size: 488 bytes --]

Re: delay in exitting from zsh shell

[Bart Schaefer]

On Nov 10,  9:19am, Eugene Dzhurinsky wrote:
}
} Okay, I wrote stupid and simple program to de-duplicate lines in
} history file:

You can't compare that to what zsh is doing.  The history file isn't
just duplicate lines, it's duplicate shell parser constructs which
may span multiple lines.  The only way to _correctly_ deduplicate it
is to run it through the parser while loading it; that's where all
the time/CPU is going.

} Am I missing something there if would say that history saving and
} de-duplication is suboptimal?

If a special routine that does nothing but deduplicate the file were
to be added, it could probably be faster, yes.  The history mechanism
performs the deduplication by loading the file back in roughly as if
all the commands were being typed at the command line again, so that
the behavior of deduplication exactly reproduces what the shell would
have done interactively with all the setopts that are in effect.

This is simply not an operation that's done often enough -- nor is
having 10000+ lines of shared deduplicated history a common enough user
configuration -- for anyone to have undertaken to create an efficient
deduplication routine that nevertheless faithfully reproduces the
behavior of re-parsing the file.  Feel free to contribute that ...

However, it might be simpler just to unset the options that cause the
file to be deduped on shell exit, and e.g. allow it to be deduped on
shell startup instead.

Re: delay in exitting from zsh shell

[Eugene Dzhurinsky]

[-- Attachment #1: Type: text/plain, Size: 936 bytes --]

On Thu, Nov 10, 2011 at 02:10:07AM -0800, Bart Schaefer wrote:
> However, it might be simpler just to unset the options that cause the
> file to be deduped on shell exit, and e.g. allow it to be deduped on
> shell startup instead.


Okay, thank you for the explanations - I was sure that I am not seeing the
whole picture :) About the solution - is it possible to, for example,
configure ZSH to have alias for "de-fragmenting" such a large history files on
demand?

And the problem is that even if I comment out the lines:

HISTFILE=~/.histfile
HISTSIZE=10000
SAVEHIST=10000
#setopt append_history
#setopt inc_append_history
#setopt histignorealldups
#setopt histignorespace
#setopt histreduceblanks
#setopt histsavenodups
#setopt sharehistory
#setopt nohup

it still takes the same (as per my feeling) time as with having those lines in
the configuration. What may be wrong here?

-- 
Eugene N Dzhurinsky

[-- Attachment #2: Type: application/pgp-signature, Size: 488 bytes --]

Re: delay in exitting from zsh shell

[Bart Schaefer]

On Nov 10, 12:15pm, Eugene Dzhurinsky wrote:
} 
} And the problem is that even if I comment out the lines:
} 
[...]
} 
} it still takes the same (as per my feeling) time as with having those
} lines in the configuration. What may be wrong here?

At this point I'm not really sure.  Maybe some of those options are
being set elsewhere, e.g., by /etc/z* files, and you need to unsetopt
them rather than just comment them out?

Leaving inc_append_history set but turning off hist_save_no_dups may
also make a difference.

You could also try "setopt hist_save_by_copy".

Re: delay in exitting from zsh shell

[Eugene Dzhurinsky]

[-- Attachment #1: Type: text/plain, Size: 481 bytes --]

On Thu, Nov 10, 2011 at 02:46:47AM -0800, Bart Schaefer wrote:
> At this point I'm not really sure.  Maybe some of those options are
> being set elsewhere, e.g., by /etc/z* files, and you need to unsetopt
> them rather than just comment them out?
> 
> Leaving inc_append_history set but turning off hist_save_no_dups may
> also make a difference.
> 
> You could also try "setopt hist_save_by_copy".

Tried that - but no luck, still the same.

-- 
Eugene N Dzhurinsky

[-- Attachment #2: Type: application/pgp-signature, Size: 488 bytes --]