* About de-duplicating the history and history appending
@ 2024-02-10 23:00 Max Blue
2024-02-11 1:27 ` Lawrence Velázquez
0 siblings, 1 reply; 4+ messages in thread
From: Max Blue @ 2024-02-10 23:00 UTC (permalink / raw)
To: zsh-users
[-- Attachment #1: Type: text/plain, Size: 1924 bytes --]
Hello.
I have been trying to mix the features which allows me to de-duplicate
and avoid excessive lines in my history, with the 'appendhistory'
variable, which somehow seems to be incompatible.
I wanted to remove all the duplicated entries after the terminal closed,
so it would be far easier to search and also to keep at hand the command
i use the most.
After a lot of help from the people in #zsh , i encountered a way to do
this, 'externally', with a script.
In my .zshrc i have added a 'trap my_script.sh EXIT' trap, which makes
the script to be ran after i close the terminal.
The script in question has the following inside:
----
/#!/usr/bin/env bash
#tac ~/.bash_history | awk '!x[$0]++' | tac > .bash_h && mv .bash_h
.bash_history /-->//This one saves the last occurence of the repeated
lines(commented)/
awk '!x[$0]++' ~/.bash_history > .bash_h && mv .bash_h .bash_history /
-->//this is what i am using at the moment.
----
So, with the default values for /share_history /and /appendhistory, /and
the following line also inside .zshrc:
/setopt HIST_IGNORE_ALL_DUPS HIST_IGNORE_ALL_DUPS HIST_IGNORE_DUPS
HIST_SAVE_NO_DUPS HIST_FIND_NO_DUPS/
I have it working.
But an user of the #zsh channel told me that this kind of script would
break multi-line lines saved in the history, and i also have noticed
that after running this script, some special characters becomes 'weird'
into the /.bash_history /viewing. With weird i mean things like /Z
/and/Z /,which fortunately are not present in the terminal history
(the commands are properly shown in there).
Is there some way to improve this? to make it to not break multi-line
history entries, to remove the 'weird characters' from the text file and
also to know if this would break my history in a random moment.
This is my first message in here so i do not know if the order of
exposing it was the correct.
Bluey
[-- Attachment #2: Type: text/html, Size: 2623 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: About de-duplicating the history and history appending
2024-02-10 23:00 About de-duplicating the history and history appending Max Blue
@ 2024-02-11 1:27 ` Lawrence Velázquez
2024-02-11 4:33 ` Bart Schaefer
0 siblings, 1 reply; 4+ messages in thread
From: Lawrence Velázquez @ 2024-02-11 1:27 UTC (permalink / raw)
To: zsh-users; +Cc: Max Blue
I participated in the aforementioned IRC discussion. Here is some
additional information for anyone inclined to chime in:
On Sat, Feb 10, 2024, at 6:00 PM, Max Blue wrote:
> Hello.
>
> I have been trying to mix the features which allows me to de-duplicate
> and avoid excessive lines in my history, with the 'appendhistory'
> variable, which somehow seems to be incompatible.
>
>
>
> I wanted to remove all the duplicated entries after the terminal
> closed, so it would be far easier to search and also to keep at hand
> the command i use the most.
To be clear, the goals are:
- To deduplicate the entire history file every time a shell saves
its history.
- To keep concurrent shells from trampling on each other's divergent
history entries, as is possible without APPEND_HISTORY.
> After a lot of help from the people in #zsh , i encountered a way to do
> this, 'externally', with a script.
>
>
>
> In my .zshrc i have added a 'trap my_script.sh EXIT' trap, which makes
> the script to be ran after i close the terminal.
>
>
>
> The script in question has the following inside:
>
> ----
>
> *#!/usr/bin/env bash
>
> #tac ~/.bash_history | awk '!x[$0]++' | tac > .bash_h && mv .bash_h
> .bash_history *-->* *This one saves the last occurence of the repeated
> lines(commented)*
>
> awk '!x[$0]++' ~/.bash_history > .bash_h && mv .bash_h .bash_history
> * -->* *this is what i am using at the moment.
>
> ----
This was adapted from <https://unix.stackexchange.com/a/78846/2803>.
> So, with the default values for *share_history *and *appendhistory,
> *and the following line also inside .zshrc:
>
> *setopt HIST_IGNORE_ALL_DUPS HIST_IGNORE_ALL_DUPS HIST_IGNORE_DUPS
> HIST_SAVE_NO_DUPS HIST_FIND_NO_DUPS*
>
> I have it working.
>
> But an user of the #zsh channel told me that this kind of script would
> break multi-line lines saved in the history, and i also have noticed
> that after running this script, some special characters becomes 'weird'
> into the *.bash_history *viewing. With weird i mean things like *#º
> *and* #º *,which fortunately are not present in the terminal history
> (the commands are properly shown in there).
The use of ".bash_history" here is not a mistake. OP really is
trying to use a single history file for both bash and zsh.
> Is there some way to improve this? to make it to not break multi-line
> history entries, to remove the 'weird characters' from the text file
> and also to know if this would break my history in a random moment.
>
>
>
> This is my first message in here so i do not know if the order of
> exposing it was the correct.
>
>
>
> Bluey
--
vq
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: About de-duplicating the history and history appending
2024-02-11 1:27 ` Lawrence Velázquez
@ 2024-02-11 4:33 ` Bart Schaefer
2024-02-11 9:33 ` Max Blue
0 siblings, 1 reply; 4+ messages in thread
From: Bart Schaefer @ 2024-02-11 4:33 UTC (permalink / raw)
To: zsh-users; +Cc: Max Blue
On Sat, Feb 10, 2024 at 7:28 PM Lawrence Velázquez <larryv@zsh.org> wrote:
>
> On Sat, Feb 10, 2024, at 6:00 PM, Max Blue wrote:
> >
> > I wanted to remove all the duplicated entries after the terminal
> > closed, so it would be far easier to search and also to keep at hand
> > the command i use the most.
>
> To be clear, the goals are:
>
> - To deduplicate the entire history file every time a shell saves
> its history.
> - To keep concurrent shells from trampling on each other's divergent
> history entries, as is possible without APPEND_HISTORY.
[...]
> The use of ".bash_history" here is not a mistake. OP really is
> trying to use a single history file for both bash and zsh.
These are fundamentally incompatible goals. Bash and zsh will not
cooperate with each other for purposes of (avoiding) overlapping
writes, the timestamp formats are incompatible, and the zsh history is
stored "metafied" -- it is not a plain text file. That also explains
this:
> > that after running this script, some special characters becomes 'weird'
> > into the *.bash_history *viewing. With weird i mean things like *#º
> > *and* #º *,which fortunately are not present in the terminal history
> > (the commands are properly shown in there).
Moving along ...
> This was adapted from <https://unix.stackexchange.com/a/78846/2803>.
I'm not going to try to work through all of that ... but I see it does
discuss the timestamp problem.
> > *setopt HIST_IGNORE_ALL_DUPS HIST_IGNORE_ALL_DUPS HIST_IGNORE_DUPS
> > HIST_SAVE_NO_DUPS HIST_FIND_NO_DUPS*
I think you mean for one of those to be HIST_EXPIRE_DUPS_FIRST ?
My approach to this would be to use "fc -p" to point to a new history
file with setopts as appropriate from the above, pull the HISTFILE
into that with "fc -R", write it out again with "fc -W", then move
that to the desired history file with appropriate locking (which you
can probably do by calling "fc -P" to switch back to the original
history and then cycling -R/-W again).
If you write that as a zsh script you can invoke it from bash to clean
up bash's history as well, with the timestamp caveat. I'm not at my
usual computer right now so not going to test details, I'll chime in
again later if things seem to be going awry.
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: About de-duplicating the history and history appending
2024-02-11 4:33 ` Bart Schaefer
@ 2024-02-11 9:33 ` Max Blue
0 siblings, 0 replies; 4+ messages in thread
From: Max Blue @ 2024-02-11 9:33 UTC (permalink / raw)
To: zsh-users; +Cc: Bart Schaefer
[-- Attachment #1: Type: text/plain, Size: 2641 bytes --]
Hello.
Thanks Larry for the clear explanation, this things are hard for me to
explain.
These are fundamentally incompatible goals. Bash and zsh will not
cooperate with each other for purposes of (avoiding) overlapping
writes, the timestamp formats are incompatible, and the zsh history is
stored "metafied" -- it is not a plain text file. That also explains
this:
>> that after running this script, some special characters becomes 'weird'
>> into the *.bash_history *viewing. With weird i mean things like *#º
>> *and* #º *,which fortunately are not present in the terminal history
>> (the commands are properly shown in there).
About the timestamps, i do not use those, so i think that would not be a
problem. I have seen, when the trying some options which broke the
history file (truncated it somehow), that there was some kind of
timestamp in the 2 last commands, but they are not present in the
.bash_history now. I think that the timestamp is disabled on my side,
and that's why i do not see it.
It makes sense to me now that sharing the history file for both shells
does not make sense and that would be causing those 'funny' characters
into the /.bash_history/ file. Those ones, on the other hand, display
correctly in the zsh shell, when looking through the history. Besides
that, should i expect any other kind of trouble if continuing like this?
i have been a while in bash and almost all my history lines are written
that way.
HIST_EXPIRE_DUPS_FIRST
I think this one is not what i want, this will 'prune' the history when
some kind of limit on the history lines is reached. I just want to keep
my history file as clean as possible, but i also keep it long(
/HISTSIZE=2000000/ , because i do not want to lose lines).
My approach to this would be to use "fc -p" to point to a new history
file with setopts as appropriate from the above, pull the HISTFILE
into that with "fc -R", write it out again with "fc -W", then move
that to the desired history file with appropriate locking (which you
can probably do by calling "fc -P" to switch back to the original
history and then cycling -R/-W again).
If you write that as a zsh script you can invoke it from bash to clean
up bash's history as well, with the timestamp caveat. I'm not at my
usual computer right now so not going to test details, I'll chime in
again later if things seem to be going awry.
I do not understand how implement this, and what would be the difference
about the previous idea of the /trap/ command. I will appreciate if you
can elaborate about this when you can.
Bluey.
[-- Attachment #2: Type: text/html, Size: 3501 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2024-02-11 9:34 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2024-02-10 23:00 About de-duplicating the history and history appending Max Blue
2024-02-11 1:27 ` Lawrence Velázquez
2024-02-11 4:33 ` Bart Schaefer
2024-02-11 9:33 ` Max Blue
Code repositories for project(s) associated with this public inbox
https://git.vuxu.org/mirror/zsh/
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).