An example of writing a custom history file?

An example of writing a custom history file?

[Rocky Bernstein]

[-- Attachment #1: Type: text/plain, Size: 294 bytes --]

Hi again -

I the zsh debugger I'd like to add a history mechanism to the command
processor.
I'm using zle to read lines. Of course, I don't want this history mechanism
to interfere with the normal zsh history mechanim.

Is there an example somewhere that I can follow for this?

Thanks again.

Re: An example of writing a custom history file?

[Bart Schaefer]

On Dec 14,  8:38pm, Rocky Bernstein wrote:
}
} Is there an example somewhere that I can follow for this?

The only one I can suggest is Functions/Misc/sticky-note which comes
with zsh.  Look at uses of "fc -ap".

Re: An example of writing a custom history file?

[Rocky Bernstein]

[-- Attachment #1: Type: text/plain, Size: 1097 bytes --]

Thanks for the pointer. That program is too complicated and there is too
much going on for me to understand which parts add to the history file. I'm
guessing it has something to do with the
zle .push-line.

Here is the 138 line program boiled down to less than 10,  the parts I'd
like to focus on:

    #!/usr/bin/zsh
    fc -ap /tmp/example_history 1000
    # Read lines and add them to history
    local sticky
    while vared -h -p "hey: "  sticky
    do
        print $sticky
        sticky=''
    done

In the above, if I add lines to history file /tmp/example_history, I see
them available in vared. However in the the above doesn't *add* lines to
the history.

Additionally what I would like to do in the body of the loop decide whether
or not to add this to the history.

Thanks


On Sun, Dec 14, 2014 at 9:53 PM, Bart Schaefer <schaefer@brasslantern.com>
wrote:

> On Dec 14,  8:38pm, Rocky Bernstein wrote:
> }
> } Is there an example somewhere that I can follow for this?
>
> The only one I can suggest is Functions/Misc/sticky-note which comes
> with zsh.  Look at uses of "fc -ap".
>

Re: An example of writing a custom history file?

[Bart Schaefer]

On Dec 15,  2:09am, Rocky Bernstein wrote:
}
} Thanks for the pointer. That program is too complicated and there is too
} much going on for me to understand which parts add to the history file. I'm
} guessing it has something to do with the
} zle .push-line.

Well, no, it doesn't have anything to do with that at all.

} Here is the 138 line program boiled down to less than 10,  the parts I'd
} like to focus on:
} 
}     #!/usr/bin/zsh
}     fc -ap /tmp/example_history 1000
}     # Read lines and add them to history
}     local sticky
}     while vared -h -p "hey: "  sticky
}     do
}         print $sticky
}         sticky=''
}     done

You've missed the important bit, which is the -s option to print.  THAT
is what adds lines to the history.

I probably should have pointed you to the example zshaddhistory hook in
the documentation.

} Additionally what I would like to do in the body of the loop decide whether
} or not to add this to the history.

    # Pseudo-code
    if this_line should be added to the history
    then print -sR "$this_line"
    fi

Re: An example of writing a custom history file?

[Rocky Bernstein]

[-- Attachment #1: Type: text/plain, Size: 1921 bytes --]

Almost there! The only remaining problem is now how write the accumulated
history. For that, I assume I use fc -W *filename* or fc -A *filename*. But
this isn't working. Here is the corrected simple program:

    #!/usr/bin/zsh
    fc -ap /tmp/example_history 1000

    local line
    # Read lines and add them to history
    while vared -h -p "hey: " line
    do
        [[ $line == 'quit' ]] && exit 0
        fc -p /tmp/example_history 1000
        # The -s option below adds the line to the history
        print -s $line
        line=''
    done
    fc -W /tmp/example_history

I have also tried putting "fc -A" as well and tried putting the write
command inside the loop.

Again, thank you.




On Mon, Dec 15, 2014 at 3:37 AM, Bart Schaefer <schaefer@brasslantern.com>
wrote:

> On Dec 15,  2:09am, Rocky Bernstein wrote:
> }
> } Thanks for the pointer. That program is too complicated and there is too
> } much going on for me to understand which parts add to the history file.
> I'm
> } guessing it has something to do with the
> } zle .push-line.
>
> Well, no, it doesn't have anything to do with that at all.
>
> } Here is the 138 line program boiled down to less than 10,  the parts I'd
> } like to focus on:
> }
> }     #!/usr/bin/zsh
> }     fc -ap /tmp/example_history 1000
> }     # Read lines and add them to history
> }     local sticky
> }     while vared -h -p "hey: "  sticky
> }     do
> }         print $sticky
> }         sticky=''
> }     done
>
> You've missed the important bit, which is the -s option to print.  THAT
> is what adds lines to the history.
>
> I probably should have pointed you to the example zshaddhistory hook in
> the documentation.
>
> } Additionally what I would like to do in the body of the loop decide
> whether
> } or not to add this to the history.
>
>     # Pseudo-code
>     if this_line should be added to the history
>     then print -sR "$this_line"
>     fi
>
>

Re: An example of writing a custom history file?

[Bart Schaefer]

On Dec 15,  9:26am, Rocky Bernstein wrote:
} 
} Almost there! The only remaining problem is now how write the accumulated
} history. For that, I assume I use fc -W *filename* or fc -A *filename*.

I don't really mind helping you out here, but if you were to actually
read the documentation for the "fc" command or even look more closely
at the way it's used in sticky-note you should be able to figure this
out for yourself.

}     #!/usr/bin/zsh
}     fc -ap /tmp/example_history 1000

You need to supply another argument to "fc -ap" telling it what the
maximum size of the saved file is.  Then that number of lines will
be saved automatically when the current function goes out of scope.

Re: An example of writing a custom history file?

[Rocky Bernstein]

[-- Attachment #1: Type: text/plain, Size: 2664 bytes --]

Making the change suggested, adding 1000 doesn't change the behavior - no
file is written. Again, here is the entire 12-line program:

    #!/usr/bin/zsh
    fc -ap /tmp/example_history 1000

    local line
    # Read lines and add them to history
    while vared -h -p "hey: " line
    do
        [[ $line == 'quit' ]] && exit 0
        # The -s option below adds the line to the history
        print -s $line
        line=''
    done

As to reading docs, I've gone over
http://zsh.sourceforge.net/Doc/Release/Shell-Builtin-Commands.html and
http://zsh.sourceforge.net/Doc/Release/Completion-System.html#Completion-System-Configuration
and https://github.com/johan/zsh/blob/master/Functions/Misc/sticky-note way
more times than I want or normally have to do to get corresponding
information in programming systems. And still without success.

In the first doc for built-in commands, 1000, the histsize parameter is
optional and is taken from HISTSIZE and SAVEHIST if those parameters are
not given. My HISTSIZE and SAVEHIST are 1000.

What is sorely missing, and I've also looked at
http://zsh.sourceforge.net/Guide/zshguide.html, are some *SIMPLE* examples
like the program above. In fact it would be great if a corrected version of
the above could be included in the User Guide.

When the basic history mechanism isn't working, it doesn't help to pour
over a 138-line program that performs several functions in one program,
adds key bindings, beeps at the terminal, has color themes, and uses some
sort of "{ } aways { }" construct that probably is a Zsh extension and not
POSIX shell construct.

That program is a great as an example of a finished zsh program.  But for
learning purposes, even variable interpolation as in

fc -ap $stickyfile $stickysize $stickysize

instead of:

 fc -ap /tmp/example_history 1000

is a distraction.

is a distraction.


On Mon, Dec 15, 2014 at 11:14 AM, Bart Schaefer <schaefer@brasslantern.com>
wrote:

> On Dec 15,  9:26am, Rocky Bernstein wrote:
> }
> } Almost there! The only remaining problem is now how write the accumulated
> } history. For that, I assume I use fc -W *filename* or fc -A *filename*.
>
> I don't really mind helping you out here, but if you were to actually
> read the documentation for the "fc" command or even look more closely
> at the way it's used in sticky-note you should be able to figure this
> out for yourself.
>
> }     #!/usr/bin/zsh
> }     fc -ap /tmp/example_history 1000
>
> You need to supply another argument to "fc -ap" telling it what the
> maximum size of the saved file is.  Then that number of lines will
> be saved automatically when the current function goes out of scope.
>

Re: An example of writing a custom history file?

[Peter Stephenson]

On Mon, 15 Dec 2014 12:05:30 -0500
Rocky Bernstein <rockyb@rubyforge.org> wrote:
> Making the change suggested, adding 1000 doesn't change the behavior - no
> file is written. Again, here is the entire 12-line program:
> 
>     #!/usr/bin/zsh
>     fc -ap /tmp/example_history 1000
> 
>     local line
>     # Read lines and add them to history
>     while vared -h -p "hey: " line
>     do
>         [[ $line == 'quit' ]] && exit 0
>         # The -s option below adds the line to the history
>         print -s $line
>         line=''
>     done

I'm not entirely following everything you're attempting to do, but the
feature you're using is for temporarily switching a history file during
a shell session.  So if you're running this as a script (rather than a
shell function), as the #! and the word "program" would imply, (i)
you're not really trying out that aspect of the feature --- though
there's no reason the basic feature shouldn't work (ii) you don't get
the "autopop" behaviour at the end of function scope that the -a
additional option to fc -p implies.

Try using "fc -P" at the end to pop by hand, as if this actually were a
function (but with the save line number argument suggested by Bart).

pws

Re: An example of writing a custom history file?

[Mikael Magnusson]

On Mon, Dec 15, 2014 at 6:05 PM, Rocky Bernstein <rockyb@rubyforge.org> wrote:
> Making the change suggested, adding 1000 doesn't change the behavior - no
> file is written. Again, here is the entire 12-line program:

Please don't top post.

I was able to get your example to work as-is by adding -i to the
hashbang line. None of the other suggestions in this thread made any
difference.

-- 
Mikael Magnusson

Re: An example of writing a custom history file?

[Rocky Bernstein]

[-- Attachment #1: Type: text/plain, Size: 1531 bytes --]

On Mon, Dec 15, 2014 at 1:22 PM, Mikael Magnusson <mikachu@gmail.com> wrote:

> On Mon, Dec 15, 2014 at 6:05 PM, Rocky Bernstein <rockyb@rubyforge.org>
> wrote:
> > Making the change suggested, adding 1000 doesn't change the behavior - no
> > file is written. Again, here is the entire 12-line program:
>
> Please don't top post.
>
> I was able to get your example to work as-is by adding -i to the
> hashbang line. None of the other suggestions in this thread made any
> difference.
>

Thanks - that worked. And as you report, none of the other suggestions did.

The following problem I can work around, but I mention because it is weird.

When I go back and add -i or --interactive to zshdb, the completion
menu comes in the line before rather than after, but what is more
troublesome is that the entered text after
the prompt isn't cleared.

So, for input:

    zshdb<1> set hi<tab>

instead of:

    highlight history
    zshdb<1> set highight

I get:

   zshdb<1> set hiset highlight
   highlight history

Some more info which may be relevant.

I tried to reproduce this in that little 12 line program by adding the
relevant code that I am using:

    completer() {
        ((2 == CURRENT)) && compadd -- fee fie foo
    }
    zle -C test_complete menu-expand-or-complete completer
    bindkey '^i' test_complete


That works fine.  It seems to have to do with the fact that in order to
make sure in zshdb that I have an interactive terminal I open a new one.
So I'm guessing that has something to do with the behavior above.

Re: An example of writing a custom history file?

[Bart Schaefer]

On Dec 15, 12:05pm, Rocky Bernstein wrote:
}
} Making the change suggested, adding 1000 doesn't change the behavior - no
} file is written. Again, here is the entire 12-line program:
} 
}     #!/usr/bin/zsh
}     fc -ap /tmp/example_history 1000
} 
}     local line
}     # Read lines and add them to history
}     while vared -h -p "hey: " line
}     do
}         [[ $line == 'quit' ]] && exit 0
}         # The -s option below adds the line to the history
}         print -s $line
}         line=''
}     done

This worked for me exactly as written; I got lines saved in the
/tmp/example_history file.  However, I was working with an interactive
shell, which normally saves history at exit.  If you are trying to run
this as a stand-alone script, you probably don't want "exit 0" there.
More likely you just want to "break" and let the loop finish.  Also
if running standalone, there's no "function scope" so I suspect that
foils the usual action of "fc -a".

Try it like this:

    fc -ap /tmp/example_history 1000 1000

    local line
    # Read lines and add them to history
    while vared -h -p "hey: " line
    do
        [[ $line == 'quit' ]] && break
        # The -s option below adds the line to the history
        # The -R option avoids other "print" processing
        print -sR $line
        line=''
    done
    # Save/pop the pushed history
    fc -P

You might also try adding

    setopt localoptions incappendhistory

so that the lines are written to the file as soon as "print -s" adds
them, rather than waiting for the "fc -P".

} When the basic history mechanism isn't working, it doesn't help to pour
} over a 138-line program that performs several functions in one program,
} adds key bindings, beeps at the terminal, has color themes, and uses some
} sort of "{ } aways { }" construct that probably is a Zsh extension and not
} POSIX shell construct.

You asked to be pointed at an example; I can't point to something that
doesn't exist, so I pointed to something that does.

However, if you're writing a zsh debugger, presumably you need to know
about and potentially use zsh extensions?

Re: An example of writing a custom history file?

[Mikael Magnusson]

On Mon, Dec 15, 2014 at 7:26 PM, Bart Schaefer
<schaefer@brasslantern.com> wrote:
> On Dec 15, 12:05pm, Rocky Bernstein wrote:
> }
> } Making the change suggested, adding 1000 doesn't change the behavior - no
> } file is written. Again, here is the entire 12-line program:
> }
> }     #!/usr/bin/zsh
> }     fc -ap /tmp/example_history 1000
> }
> }     local line
> }     # Read lines and add them to history
> }     while vared -h -p "hey: " line
> }     do
> }         [[ $line == 'quit' ]] && exit 0
> }         # The -s option below adds the line to the history
> }         print -s $line
> }         line=''
> }     done
>
> This worked for me exactly as written; I got lines saved in the
> /tmp/example_history file.  However, I was working with an interactive
> shell, which normally saves history at exit.  If you are trying to run
> this as a stand-alone script, you probably don't want "exit 0" there.
> More likely you just want to "break" and let the loop finish.  Also
> if running standalone, there's no "function scope" so I suspect that
> foils the usual action of "fc -a".
[...]

I tried the script with and without -a and a -P at the end, and
nesting the body inside a () { } anon function, nothing "helps" in the
case of a non-interactive shell, and with an interactive shell, none
of it is needed.

I found only one place in the manpage that mentions history only being
active in an interactive shell, perhaps it should be clearer about it?
I also noticed that explicit fc -l and fc -W etc do nothing but also
don't print a warning when invoked non-interactively.

HISTFILE
  The file to save the history in when an interactive shell
  exits. If unset, the history is not saved.

I think before we added print -s, there was no way to get history
non-interactively (I suppose fc -R?) and likely nobody thought to
mention the restriction in the fc docs when adding it?

-- 
Mikael Magnusson

Re: An example of writing a custom history file?

[Peter Stephenson]

On Mon, 15 Dec 2014 19:34:52 +0100
Mikael Magnusson <mikachu@gmail.com> wrote:
> I found only one place in the manpage that mentions history only being
> active in an interactive shell, perhaps it should be clearer about it?
> I also noticed that explicit fc -l and fc -W etc do nothing but also
> don't print a warning when invoked non-interactively.

Yes, this is definitely wrong.  (Further discussion on this can
go on zsh-workers but I wanted to respond initially here.)

Here are a couple of places where it could be mentioned.  The roadmap
entry deliberately does not go into the gory subject of turning
on the interactive option for scripts.  (Yes, it's not a roadmap,
it's an overview, in current corporate jargon.)

By the way, I'm entirely open to making fc read and write history in
non-interactive shells if we can manage that, but obviously it would
need some work.

diff --git a/Doc/Zsh/builtins.yo b/Doc/Zsh/builtins.yo
index 4463123..38788d3 100644
--- a/Doc/Zsh/builtins.yo
+++ b/Doc/Zsh/builtins.yo
@@ -657,6 +657,12 @@ xitem(tt(      )[ var(old)tt(=)var(new) ... ] [ var(first) [ var(last) ] ])
 xitem(tt(fc) tt(-p) [ tt(-a) ] [ var(filename) [ var(histsize) [ var(savehistsize) ] ] ])
 xitem(tt(fc) tt(-P))
 item(tt(fc) tt(-ARWI) [ var(filename) ])(
+The tt(fc) command controls the interactive history mechanism.  Note
+that reading and writing of history options is only performed if the
+shell is interactive.  Usually this is detected automatically, but
+it can be forced by setting the tt(interactive) option when starting the
+shell.
+
 Select a range of commands from var(first) to var(last) from the
 history list.
 The arguments var(first) and var(last) may be specified as a
diff --git a/Doc/Zsh/roadmap.yo b/Doc/Zsh/roadmap.yo
index ba598e5..6778489 100644
--- a/Doc/Zsh/roadmap.yo
+++ b/Doc/Zsh/roadmap.yo
@@ -41,7 +41,9 @@ set appropriate variables, and the number of history lines retained by
 default is quite small (30 lines).  See the description of the shell
 variables (referred to in the documentation as parameters) tt(HISTFILE),
 tt(HISTSIZE) and tt(SAVEHIST) in ifzman(zmanref(zshparam))\
-ifnzman(noderef(Parameters Used By The Shell)).
+ifnzman(noderef(Parameters Used By The Shell)).  Note that it's
+currently only possible to read and write files saving history
+when the shell is interactive, i.e. it does not work from scripts.
 
 The shell now supports the UTF-8 character set (and also others if
 supported by the operating system).  This is (mostly) handled transparently

pws

Re: An example of writing a custom history file?

[Rocky Bernstein]

[-- Attachment #1: Type: text/plain, Size: 3651 bytes --]

On Mon, Dec 15, 2014 at 1:26 PM, Bart Schaefer <schaefer@brasslantern.com>
wrote:

> On Dec 15, 12:05pm, Rocky Bernstein wrote:
> }
> } Making the change suggested, adding 1000 doesn't change the behavior - no
> } file is written. Again, here is the entire 12-line program:
> }
> }     #!/usr/bin/zsh
> }     fc -ap /tmp/example_history 1000
> }
> }     local line
> }     # Read lines and add them to history
> }     while vared -h -p "hey: " line
> }     do
> }         [[ $line == 'quit' ]] && exit 0
> }         # The -s option below adds the line to the history
> }         print -s $line
> }         line=''
> }     done
>
> This worked for me exactly as written; I got lines saved in the
> /tmp/example_history file.  However, I was working with an interactive
> shell, which normally saves history at exit.  If you are trying to run
> this as a stand-alone script, you probably don't want "exit 0" there.
> More likely you just want to "break" and let the loop finish.  Also
> if running standalone, there's no "function scope" so I suspect that
> foils the usual action of "fc -a".
>
> Try it like this:
>
>     fc -ap /tmp/example_history 1000 1000
>
>     local line
>     # Read lines and add them to history
>     while vared -h -p "hey: " line
>     do
>         [[ $line == 'quit' ]] && break
>         # The -s option below adds the line to the history
>         # The -R option avoids other "print" processing
>         print -sR $line
>         line=''
>     done
>     # Save/pop the pushed history
>     fc -P
>
> You might also try adding
>
>     setopt localoptions incappendhistory
>
> so that the lines are written to the file as soon as "print -s" adds
> them, rather than waiting for the "fc -P".
>

Tried that. Several times, several places, several ways. It didn't work,
and I
am at my frustration level where I don't want to pursue it. If someone else
does, great.  I appreciate the suggestion and your help though.


} When the basic history mechanism isn't working, it doesn't help to pour
> } over a 138-line program that performs several functions in one program,
> } adds key bindings, beeps at the terminal, has color themes, and uses some
> } sort of "{ } aways { }" construct that probably is a Zsh extension and
> not
> } POSIX shell construct.
>
> You asked to be pointed at an example; I can't point to something that
> doesn't exist, so I pointed to something that does.
>

Sure, that's better than nothing. My larger point is that although I
realize and appreciate
the enormous effort that has been put into documenting zsh, the bottom line
is that too often for me it doesn't work all that well. One reason is a
myriad
of complicated details often specific to zsh rather than some simple basic
examples to start with before giving pointers to where you can get more
detailed
information.

I see there is http://zshwiki.org. Perhaps I'll put this little example on
that.



>
> However, if you're writing a zsh debugger, presumably you need to know
> about and potentially use zsh extensions?
>

Again this misses the point. I could roughly understand what's going there.
As I mentioned with variable interpolation, it's just a distraction if when
suggested
for the pedagogical purpose of understanding how command history works.

The debugger itself, yes, it does have to be aware of zsh extensions. To a
good
extent the zsh interpreter handles that. It will step into blocks and report
where it is and so on.

But as for the coding, I have written 3 POSIX shell debuggers. So the more
I can
just stick to POSIX the more code can be shared among the 3 debuggers.

But in closing, again, thanks for the help.