On Wed, Nov 09, 2011 at 07:51:18PM -0800, Bart Schaefer wrote:
> This tends to imply that what's taking those 3-5 seconds is searching
> your history file for duplicate entries in order to enforce the
> hist_save_no_dups option.  Also, inc_append_history may allow the file
> to grow to up to 10500 lines for a SAVEHIST of 10000, and those extra
> lines will be trimmed at shell exit.
> 
> In more detail, when you use inc_append_history and/or share_history
> along with hist_save_no_dups, zsh re-reads and de-duplicates the entire
> file from disk after it has obtained a lock for it, rather than just
> dumping out the history that is already in memory, because it can't
> know if some other shell has appended something new to the file before
> the lock was obtained.  The speed of the disk is inconsequential to the
> CPU expended doing the deduplication.

Okay, I wrote stupid and simple program to de-duplicate lines in history file:

import qualified Data.ByteString.Char8 as C8
import Data.List
import System
import Control.Monad

fixHist = nub . C8.lines

main = do
    (fname:_) <- getArgs
    hl <- liftM fixHist $ C8.readFile fname
    C8.writeFile fname $ C8.unlines hl

function nub is O(n^2) time-complex, and still it takes less than second in
order to read file, optimize it and write it down.

> ./hst .histfile.tst  0.89s user 0.01s system 99% cpu 0.901 total

and I believe that using linked hashtables will improve performance as well.
Am I missing something there if would say that history saving and
de-duplication is suboptimal?

-- 
Eugene N Dzhurinsky