Re: About de-duplicating the history and history appending

[Bart Schaefer <schaefer@brasslantern.com>]

On Sat, Feb 10, 2024 at 7:28 PM Lawrence Velázquez <larryv@zsh.org> wrote:
>
> On Sat, Feb 10, 2024, at 6:00 PM, Max Blue wrote:
> >
> > I wanted to remove all the duplicated entries after the terminal
> > closed, so it would be far easier to search and also to keep at hand
> > the command i use the most.
>
> To be clear, the goals are:
>
> - To deduplicate the entire history file every time a shell saves
>   its history.
> - To keep concurrent shells from trampling on each other's divergent
>   history entries, as is possible without APPEND_HISTORY.
[...]
> The use of ".bash_history" here is not a mistake.  OP really is
> trying to use a single history file for both bash and zsh.

These are fundamentally incompatible goals.  Bash and zsh will not
cooperate with each other for purposes of (avoiding) overlapping
writes, the timestamp formats are incompatible, and the zsh history is
stored "metafied" -- it is not a plain text file.  That also explains
this:

> > that after running this script, some special characters becomes 'weird'
> > into the *.bash_history *viewing. With weird i mean things like *＃º
> > *and* ＃º *,which fortunately are not present in the terminal history
> > (the commands are properly shown in there).

Moving along ...

> This was adapted from <https://unix.stackexchange.com/a/78846/2803>.

I'm not going to try to work through all of that ... but I see it does
discuss the timestamp problem.

> > *setopt HIST_IGNORE_ALL_DUPS HIST_IGNORE_ALL_DUPS HIST_IGNORE_DUPS
> > HIST_SAVE_NO_DUPS HIST_FIND_NO_DUPS*

I think you mean for one of those to be HIST_EXPIRE_DUPS_FIRST ?

My approach to this would be to use "fc -p" to point to a new history
file with setopts as appropriate from the above, pull the HISTFILE
into that with "fc -R", write it out again with "fc -W", then move
that to the desired history file with appropriate locking (which you
can probably do by calling "fc -P" to switch back to the original
history and then cycling -R/-W again).

If you write that as a zsh script you can invoke it from bash to clean
up bash's history as well, with the timestamp caveat.  I'm not at my
usual computer right now so not going to test details, I'll chime in
again later if things seem to be going awry.