About de-duplicating the history and history appending

About de-duplicating the history and history appending

[Max Blue]

[-- Attachment #1: Type: text/plain, Size: 1924 bytes --]

Hello.

I have been trying to mix the features which allows me to de-duplicate 
and avoid excessive lines in my history, with the 'appendhistory' 
variable, which somehow seems to be incompatible.


I wanted to remove all the duplicated entries after the terminal closed, 
so it would be far easier to search and also to keep at hand the command 
i use the most.

After a lot of help from the people in #zsh , i encountered a way to do 
this, 'externally', with a script.


In my .zshrc i have added a 'trap my_script.sh EXIT' trap, which makes 
the script to be ran after i close the terminal.


The script in question has the following inside:

----

/#!/usr/bin/env bash

#tac ~/.bash_history | awk '!x[$0]++' | tac  > .bash_h && mv .bash_h 
.bash_history /-->//This one saves the last occurence of the repeated 
lines(commented)/

awk '!x[$0]++' ~/.bash_history > .bash_h && mv .bash_h .bash_history / 
-->//this is what i am using at the moment.

----

So, with the default values for /share_history /and /appendhistory, /and 
the following line also inside .zshrc:

/setopt HIST_IGNORE_ALL_DUPS HIST_IGNORE_ALL_DUPS HIST_IGNORE_DUPS 
HIST_SAVE_NO_DUPS HIST_FIND_NO_DUPS/

I have it working.

But an user of the #zsh channel told me that this kind of script would 
break multi-line lines saved in the history, and i also have noticed 
that after running this script, some special characters becomes 'weird' 
into the /.bash_history /viewing. With weird i mean things like /＃º 
/and/＃º /,which fortunately are not present in the terminal history 
(the commands are properly shown in there).

Is there some way to improve this? to make it to not break multi-line 
history entries, to remove the 'weird characters' from the text file and 
also to know if this would break my history in a random moment.


This is my first message in here so i do not know if the order of 
exposing it was the correct.


Bluey

[-- Attachment #2: Type: text/html, Size: 2623 bytes --]

Re: About de-duplicating the history and history appending

[Lawrence Velázquez]

I participated in the aforementioned IRC discussion.  Here is some
additional information for anyone inclined to chime in:


On Sat, Feb 10, 2024, at 6:00 PM, Max Blue wrote:
> Hello.
>
> I have been trying to mix the features which allows me to de-duplicate 
> and avoid excessive lines in my history, with the 'appendhistory' 
> variable, which somehow seems to be incompatible.
>
>
>
> I wanted to remove all the duplicated entries after the terminal 
> closed, so it would be far easier to search and also to keep at hand 
> the command i use the most.

To be clear, the goals are:

- To deduplicate the entire history file every time a shell saves
  its history.
- To keep concurrent shells from trampling on each other's divergent
  history entries, as is possible without APPEND_HISTORY.


> After a lot of help from the people in #zsh , i encountered a way to do 
> this, 'externally', with a script.
>
>
>
> In my .zshrc i have added a 'trap my_script.sh EXIT' trap, which makes 
> the script to be ran after i close the terminal.
>
>
>
> The script in question has the following inside:
>
> ----
>
> *#!/usr/bin/env bash
>
> #tac ~/.bash_history | awk '!x[$0]++' | tac  > .bash_h && mv .bash_h 
> .bash_history *-->* *This one saves the last occurence of the repeated 
> lines(commented)*
>
> awk '!x[$0]++' ~/.bash_history > .bash_h && mv .bash_h .bash_history    
> * -->* *this is what i am using at the moment.
>
> ----

This was adapted from <https://unix.stackexchange.com/a/78846/2803>.


> So, with the default values for *share_history *and *appendhistory, 
> *and the following line also inside .zshrc:
>
> *setopt HIST_IGNORE_ALL_DUPS HIST_IGNORE_ALL_DUPS HIST_IGNORE_DUPS 
> HIST_SAVE_NO_DUPS HIST_FIND_NO_DUPS*
>
> I have it working.
>
> But an user of the #zsh channel told me that this kind of script would 
> break multi-line lines saved in the history, and i also have noticed 
> that after running this script, some special characters becomes 'weird' 
> into the *.bash_history *viewing. With weird i mean things like *＃º 
> *and* ＃º *,which fortunately are not present in the terminal history 
> (the commands are properly shown in there).

The use of ".bash_history" here is not a mistake.  OP really is
trying to use a single history file for both bash and zsh.


> Is there some way to improve this? to make it to not break multi-line 
> history entries, to remove the 'weird characters' from the text file 
> and also to know if this would break my history in a random moment.
>
>
>
> This is my first message in here so i do not know if the order of 
> exposing it was the correct.
>
>
>
> Bluey


-- 
vq

Re: About de-duplicating the history and history appending

[Bart Schaefer]

On Sat, Feb 10, 2024 at 7:28 PM Lawrence Velázquez <larryv@zsh.org> wrote:
>
> On Sat, Feb 10, 2024, at 6:00 PM, Max Blue wrote:
> >
> > I wanted to remove all the duplicated entries after the terminal
> > closed, so it would be far easier to search and also to keep at hand
> > the command i use the most.
>
> To be clear, the goals are:
>
> - To deduplicate the entire history file every time a shell saves
>   its history.
> - To keep concurrent shells from trampling on each other's divergent
>   history entries, as is possible without APPEND_HISTORY.
[...]
> The use of ".bash_history" here is not a mistake.  OP really is
> trying to use a single history file for both bash and zsh.

These are fundamentally incompatible goals.  Bash and zsh will not
cooperate with each other for purposes of (avoiding) overlapping
writes, the timestamp formats are incompatible, and the zsh history is
stored "metafied" -- it is not a plain text file.  That also explains
this:

> > that after running this script, some special characters becomes 'weird'
> > into the *.bash_history *viewing. With weird i mean things like *＃º
> > *and* ＃º *,which fortunately are not present in the terminal history
> > (the commands are properly shown in there).

Moving along ...

> This was adapted from <https://unix.stackexchange.com/a/78846/2803>.

I'm not going to try to work through all of that ... but I see it does
discuss the timestamp problem.

> > *setopt HIST_IGNORE_ALL_DUPS HIST_IGNORE_ALL_DUPS HIST_IGNORE_DUPS
> > HIST_SAVE_NO_DUPS HIST_FIND_NO_DUPS*

I think you mean for one of those to be HIST_EXPIRE_DUPS_FIRST ?

My approach to this would be to use "fc -p" to point to a new history
file with setopts as appropriate from the above, pull the HISTFILE
into that with "fc -R", write it out again with "fc -W", then move
that to the desired history file with appropriate locking (which you
can probably do by calling "fc -P" to switch back to the original
history and then cycling -R/-W again).

If you write that as a zsh script you can invoke it from bash to clean
up bash's history as well, with the timestamp caveat.  I'm not at my
usual computer right now so not going to test details, I'll chime in
again later if things seem to be going awry.

Re: About de-duplicating the history and history appending

[Max Blue]

[-- Attachment #1: Type: text/plain, Size: 2641 bytes --]

Hello.

Thanks Larry for the clear explanation, this things are hard for me to 
explain.


These are fundamentally incompatible goals.  Bash and zsh will not
cooperate with each other for purposes of (avoiding) overlapping
writes, the timestamp formats are incompatible, and the zsh history is
stored "metafied" -- it is not a plain text file.  That also explains
this:

>> that after running this script, some special characters becomes 'weird'
>> into the *.bash_history *viewing. With weird i mean things like *＃º
>> *and* ＃º *,which fortunately are not present in the terminal history
>> (the commands are properly shown in there).

About the timestamps, i do not use those, so i think that would not be a 
problem. I have seen, when the trying some options which broke the 
history file (truncated it somehow), that there was some kind of 
timestamp in the 2 last commands, but they are not present in the 
.bash_history now. I think that the timestamp is disabled on my side, 
and that's why i do not see it.

It makes sense to me now that sharing the history file for both shells 
does not make sense and that would be causing those 'funny' characters 
into the /.bash_history/ file. Those ones, on the other hand, display 
correctly in the zsh shell, when looking through the history. Besides 
that, should i expect any other kind of trouble if continuing like this? 
i have been a while in bash and almost all my history lines are written 
that way.


    HIST_EXPIRE_DUPS_FIRST

I think this one is not what i want, this will 'prune' the history when 
some kind of limit on the history lines is reached. I just want to keep 
my history file as clean as possible, but i also keep it long( 
/HISTSIZE=2000000/ , because i do not want to lose lines).

    My approach to this would be to use "fc -p" to point to a new history
    file with setopts as appropriate from the above, pull the HISTFILE
    into that with "fc -R", write it out again with "fc -W", then move
    that to the desired history file with appropriate locking (which you
    can probably do by calling "fc -P" to switch back to the original
    history and then cycling -R/-W again).

    If you write that as a zsh script you can invoke it from bash to clean
    up bash's history as well, with the timestamp caveat.  I'm not at my
    usual computer right now so not going to test details, I'll chime in
    again later if things seem to be going awry.


I do not understand how implement this, and what would be the difference 
about the previous idea of the /trap/ command. I will appreciate if you 
can elaborate about this when you can.


Bluey.

[-- Attachment #2: Type: text/html, Size: 3501 bytes --]