Re: About de-duplicating the history and history appending

["Lawrence Velázquez" <larryv@zsh.org>]

I participated in the aforementioned IRC discussion.  Here is some
additional information for anyone inclined to chime in:


On Sat, Feb 10, 2024, at 6:00 PM, Max Blue wrote:
> Hello.
>
> I have been trying to mix the features which allows me to de-duplicate 
> and avoid excessive lines in my history, with the 'appendhistory' 
> variable, which somehow seems to be incompatible.
>
>
>
> I wanted to remove all the duplicated entries after the terminal 
> closed, so it would be far easier to search and also to keep at hand 
> the command i use the most.

To be clear, the goals are:

- To deduplicate the entire history file every time a shell saves
  its history.
- To keep concurrent shells from trampling on each other's divergent
  history entries, as is possible without APPEND_HISTORY.


> After a lot of help from the people in #zsh , i encountered a way to do 
> this, 'externally', with a script.
>
>
>
> In my .zshrc i have added a 'trap my_script.sh EXIT' trap, which makes 
> the script to be ran after i close the terminal.
>
>
>
> The script in question has the following inside:
>
> ----
>
> *#!/usr/bin/env bash
>
> #tac ~/.bash_history | awk '!x[$0]++' | tac  > .bash_h && mv .bash_h 
> .bash_history *-->* *This one saves the last occurence of the repeated 
> lines(commented)*
>
> awk '!x[$0]++' ~/.bash_history > .bash_h && mv .bash_h .bash_history    
> * -->* *this is what i am using at the moment.
>
> ----

This was adapted from <https://unix.stackexchange.com/a/78846/2803>.


> So, with the default values for *share_history *and *appendhistory, 
> *and the following line also inside .zshrc:
>
> *setopt HIST_IGNORE_ALL_DUPS HIST_IGNORE_ALL_DUPS HIST_IGNORE_DUPS 
> HIST_SAVE_NO_DUPS HIST_FIND_NO_DUPS*
>
> I have it working.
>
> But an user of the #zsh channel told me that this kind of script would 
> break multi-line lines saved in the history, and i also have noticed 
> that after running this script, some special characters becomes 'weird' 
> into the *.bash_history *viewing. With weird i mean things like *＃º 
> *and* ＃º *,which fortunately are not present in the terminal history 
> (the commands are properly shown in there).

The use of ".bash_history" here is not a mistake.  OP really is
trying to use a single history file for both bash and zsh.


> Is there some way to improve this? to make it to not break multi-line 
> history entries, to remove the 'weird characters' from the text file 
> and also to know if this would break my history in a random moment.
>
>
>
> This is my first message in here so i do not know if the order of 
> exposing it was the correct.
>
>
>
> Bluey


-- 
vq