Foreign Commands Not Appearing in 'history' Array When 'sharehistory' Option is Enabled Until a Command is Entered

Foreign Commands Not Appearing in 'history' Array When 'sharehistory' Option is Enabled Until a Command is Entered

[Langbart]

[-- Attachment #1: Type: text/plain, Size: 1659 bytes --]

Guten Tag,

is it a bug that when the 'sharehistory' option is enabled, commands from other shells ('foreign') aren't included in the 'history' array until the user enters a command?

Consider the following widget and settings in my '.zshrc':

```sh
test-history() {
BUFFER=""

print
print -- "=== HISTORY TEST ==="
print -- "history array:"
zmodload -F zsh/parameter p:history || return 1
printf "%s\t%s\n" "${(kv)history[@]}" | head -5

print -- "--------------------"

print -- "fc command:"
print -r -- "$(fc -rl -5)"

zle accept-line
}

zle -N test-history
bindkey '^R' test-history

setopt SHARE_HISTORY
```

I open two terminals, type 'echo Test 1' in Terminal 1, switch to Terminal 2, and press 'Enter' without entering a command. Pressing 'Ctrl-R' shows that the 'history' array doesn't include the 'echo Test 1' entry.

```txt
=== HISTORY TEST ===
history array:

--------------------
fc command:
1* echo Test 1
```

After entering 'echo Test 2' in Terminal 2 and pressing 'Ctrl-R' again in Terminal 2, the output is:

```txt
=== HISTORY TEST ===
history array:
2 echo Test 2
1 echo Test 1
--------------------
fc command:
2 echo Test 2
1* echo Test 1
```

Prepending 'fc -RI' before accessing the 'history' array seems to resolve the issue. Is this the recommended workaround to ensure that the 'history' array lists its resuls like 'fc -rl ...' does ?

```sh
...
[[ "${options[sharehistory]}" == "on" ]] && fc -RI

zmodload -F zsh/parameter p:history || return 1
printf "%s\t%s\n" "${(kv)history[@]}" | head -5

...
```

Related:

- [https://github.com/junegunn/fzf/issues/4061](https://github.com/junegunn/fzf/issues/4061#issuecomment-2436756709)

[-- Attachment #2: Type: text/html, Size: 4986 bytes --]

Re: Foreign Commands Not Appearing in 'history' Array When 'sharehistory' Option is Enabled Until a Command is Entered

[Bart Schaefer]

[-- Attachment #1: Type: text/plain, Size: 964 bytes --]

Short answers:

On Mon, Oct 28, 2024 at 8:06 AM Langbart <Langbart@protonmail.com> wrote:

>
> is it a bug that when the 'sharehistory' option is enabled, commands from
> other shells ('foreign') aren't included in the 'history' array until the
> user enters a command?
>

No.


> Prepending 'fc -RI' before accessing the 'history' array seems to resolve
> the issue. Is this the recommended workaround to ensure that the 'history'
> array lists its resuls like 'fc -rl ...'  does ?
>

Yes.

Longer answer:

When you're at a prompt, two things are going on.  First, a history
position (the "next" number) has already been allocated for expansion of
the %h prompt escape, and loading from the shared history would have to
somehow fit "above" that.  Second, the line editor is blocked on the
terminal driver waiting for input, it's not burning CPU in the background
polling the history file, etc., so you have to explicitly wake it up.

[-- Attachment #2: Type: text/html, Size: 1662 bytes --]

Re: Foreign Commands Not Appearing in 'history' Array When 'sharehistory' Option is Enabled Until a Command is Entered

[Roman Perepelitsa]

On Mon, Oct 28, 2024 at 5:43 PM Bart Schaefer <schaefer@brasslantern.com> wrote:
>
> Short answers:
>
> On Mon, Oct 28, 2024 at 8:06 AM Langbart <Langbart@protonmail.com> wrote:
>>
>>
>> is it a bug that when the 'sharehistory' option is enabled, commands from other shells ('foreign') aren't included in the 'history' array until the user enters a command?
>
> No.

I believe the OP is saying that after you execute a command in one
terminal and press ENTER in another terminal, within that second
terminal all widgets that operate on history, such as
history-incremental-search-backward, will see the command, but the
`history` array won't contain it. You would need to execute a
non-empty command in the second terminal for `history` to get updated.

This sounds quite strange and unexpected to me. Is this really
intended? Why is an empty command enough to for all history related
widgets to pick up commands executed in another terminal, but for
`history` to update you need to execute a non-empty command?

Roman

P.S.

I haven't actually tried it but this is my understanding based on what
the OP wrote here and on github.

Re: Foreign Commands Not Appearing in 'history' Array When 'sharehistory' Option is Enabled Until a Command is Entered

[Bart Schaefer]

On Mon, Oct 28, 2024 at 10:35 AM Roman Perepelitsa
<roman.perepelitsa@gmail.com> wrote:
>
> I believe the OP is saying that after you execute a command in one
> terminal and press ENTER in another terminal, within that second
> terminal all widgets that operate on history, such as
> history-incremental-search-backward, will see the command, but the
> `history` array won't contain it.

That would seem to be pretty nearly impossible, because $history is
populated by calling the same methods that the widgets use, it's not
pre-generated/cached.

However, I notice that the OP is doing this:
  printf "%s\t%s\n" "${(kv)history[@]}" | head -5

The left hand side of that pipe is going to be forked off and might
not see the same history list as the parent shell.  However, also note
from my "longer answer":

> First, a history position (the "next" number) has already been allocated for expansion of the %h prompt escape, and loading from the shared history would have to somehow fit "above" that.

The history number isn't incremented on an empty command, so this
situation remains.  I just re-read the OP and I don't find the
"widgets that operate on history ... will see the command" part?  What
have I missed?

Re: Foreign Commands Not Appearing in 'history' Array When 'sharehistory' Option is Enabled Until a Command is Entered

[Roman Perepelitsa]

On Mon, Oct 28, 2024 at 7:15 PM Bart Schaefer <schaefer@brasslantern.com> wrote:
>
> I just re-read the OP and I don't find the
> "widgets that operate on history ... will see the command" part?  What
> have I missed?

I inferred it from this prior interaction:
https://github.com/romkatv/zsh4humans/issues/329#issuecomment-2439657976.
This comment and the ones below it. I really need to try this for
myself and not speculate.

Roman

Re: Foreign Commands Not Appearing in 'history' Array When 'sharehistory' Option is Enabled Until a Command is Entered

[Roman Perepelitsa]

On Mon, Oct 28, 2024 at 7:22 PM Roman Perepelitsa
<roman.perepelitsa@gmail.com> wrote:
>
> On Mon, Oct 28, 2024 at 7:15 PM Bart Schaefer <schaefer@brasslantern.com> wrote:
> >
> > I just re-read the OP and I don't find the
> > "widgets that operate on history ... will see the command" part?  What
> > have I missed?
>
> I inferred it from this prior interaction:
> https://github.com/romkatv/zsh4humans/issues/329#issuecomment-2439657976.
> This comment and the ones below it. I really need to try this for
> myself and not speculate.

I've tried it myself and can confirm this issue: the history
associative array updates only when you execute a non-empty command;
it does not update when you simply press ENTER, even when
share_history is set.

To reproduce:

1. Run `zsh -f` in two terminals. I'll refer to them as Terminal 1 and
Terminal 2 below.
2. Paste this code into each of the two terminals.

    setopt share_history
    HISTSIZE=1000
    SAVEHIST=1000
    HISTFILE=/tmp/hist

    zmodload zsh/parameter

    function test-history() {
      zle up-history
      zle -M -- "${history[${#history}]}"
    }

    zle -N test-history
    bindkey '^T' test-history

3. Run `: 1` in Terminal 1.
4. Run `: 2` in Terminal 2.
5. Press ENTER in Terminal 1 without entering a command.
6. Press Ctrl-T in Terminal 1.

Expected:

    adam% : 2
    : 2

Actual:

    adam% : 2
    : 1

We can see that `zle up-history` sees the last history event, which
was written by Terminal 2, but `history` does not have it.

Roman

Re: Foreign Commands Not Appearing in 'history' Array When 'sharehistory' Option is Enabled Until a Command is Entered

[Bart Schaefer]

On Sat, Nov 23, 2024 at 11:41 PM Roman Perepelitsa
<roman.perepelitsa@gmail.com> wrote:
>
> Expected:
>
>     adam% : 2
>     : 2
>
> Actual:
>
>     adam% : 2
>     : 1
>
> We can see that `zle up-history` sees the last history event, which
> was written by Terminal 2, but `history` does not have it.

Change the prompt to display %h (history event number) and change the
test function to

  function test-history() {
      zle up-history
      zle -M -- "$HISTNO $#history ${history[${#history}]}"
  }

This will demonstrate what's happening:  The event shared from
Terminal 2 is being placed in the "current event" slot ($HISTNO) which
is not committed to the full history until a non-empty command is
executed.  I believe this in turn is because the test-history widget
is (by program logic) being executed "during editing of the current
event" and consequently that edit is considered unfinished.

I don't have availability to chase this any further any time soon, nor
any clues as to how difficult it would be to alter this behavior in
the face of shared history.  The logic about when to commit the
"current event" is pretty deeply ingrained.

Re: Foreign Commands Not Appearing in 'history' Array When 'sharehistory' Option is Enabled Until a Command is Entered

[Mikael Magnusson]

On Mon, Nov 25, 2024 at 9:12 PM Bart Schaefer <schaefer@brasslantern.com> wrote:
>
> On Sat, Nov 23, 2024 at 11:41 PM Roman Perepelitsa
> <roman.perepelitsa@gmail.com> wrote:
> >
> > Expected:
> >
> >     adam% : 2
> >     : 2
> >
> > Actual:
> >
> >     adam% : 2
> >     : 1
> >
> > We can see that `zle up-history` sees the last history event, which
> > was written by Terminal 2, but `history` does not have it.
>
> Change the prompt to display %h (history event number) and change the
> test function to
>
>   function test-history() {
>       zle up-history
>       zle -M -- "$HISTNO $#history ${history[${#history}]}"
>   }
>
> This will demonstrate what's happening:  The event shared from
> Terminal 2 is being placed in the "current event" slot ($HISTNO) which
> is not committed to the full history until a non-empty command is
> executed.  I believe this in turn is because the test-history widget
> is (by program logic) being executed "during editing of the current
> event" and consequently that edit is considered unfinished.
>
> I don't have availability to chase this any further any time soon, nor
> any clues as to how difficult it would be to alter this behavior in
> the face of shared history.  The logic about when to commit the
> "current event" is pretty deeply ingrained.

Another consequence of this is
https://www.zsh.org/mla/workers/2021/msg01087.html

-- 
Mikael Magnusson

Re: Foreign Commands Not Appearing in 'history' Array When 'sharehistory' Option is Enabled Until a Command is Entered

[Langbart]

On Monday, October 28th, 2024 at 6:14 PM, Bart Schaefer <schaefer@brasslantern.com> wrote:

> On Mon, Oct 28, 2024 at 10:35 AM Roman Perepelitsa
> roman.perepelitsa@gmail.com wrote:
>
> > I believe the OP is saying that after you execute a command in one
> > terminal and press ENTER in another terminal, within that second
> > terminal all widgets that operate on history, such as
> > history-incremental-search-backward, will see the command, but the
> > `history` array won't contain it.
>
>
> That would seem to be pretty nearly impossible, because $history is
> populated by calling the same methods that the widgets use, it's not
> pre-generated/cached.

Yes, Roman described my issue accurately.


> However, I notice that the OP is doing this:
> printf "%s\t%s\n" "${(kv)history[@]}" | head -5

I have also tested it without the extra pipe, the issue is the same.

> The left hand side of that pipe is going to be forked off and might
> not see the same history list as the parent shell. However, also note
> from my "longer answer":
>
> > First, a history position (the "next" number) has already been allocated for expansion of the %h prompt escape, and loading from the shared history would have to somehow fit "above" that.
>
>
> The history number isn't incremented on an empty command, so this
> situation remains. I just re-read the OP and I don't find the
> "widgets that operate on history ... will see the command" part? What
> have I missed?

Thanks for the answers.

I have also follow-up question about the usage of 'fc -RI'. When I run this command in my widget, I notice that I can't retrieve the recent history entry by pressing the 'Up' arrow after exiting the widget.

to reproduce:

1. Add the widget below to your setup
```sh
test-widget() {
  fc -RI
  zle -M "History updated"
}
zle -N test-widget

bindkey '^p' test-widget
bindkey '^[[A' up-line-or-history
```

2. Press 'Control-P'

3. Press the 'Up' key notice nothing happens

PS: The issue doesn't occur in an environment where the 'HISTFILE' is undefined or the assigned file is empty. Using 'fc -R' lets me recall a recent command to the command line buffer when pressing the 'Up' key. Would that be a valid bug?