Two different zsh sessions handle $* differently

Two different zsh sessions handle $* differently

[Mikael Puhakka]

Hi zsh-workers,

Recently I noticed that my journal script has started to lose data. Then
I noticed that this is somehow in how zsh handles my little helper
function. This peculiarly affects one zsh session and not many others
that I can see.

This is what happens in a freshly launched session:

    (~) echo $ZSH_VERSION
    5.2

    (~) whence -f j
    j () {
        noglob journal.py "$*"
    }

    (~) j Test. ABC this is a test.
    argv ['/home/progo/pika/journal.py', 'Test. ABC this is a test.']


And this is what the faulty session does:


    (~) echo $ZSH_VERSION
    5.2

    (~) whence -f j
    j () {
        noglob journal.py "$*"
    }

    (~) j Test. ABC this is a test.
    argv ['/home/progo/pika/journal.py', 'Test.']


The python script is modified to be trivial:

    (~) head -n5 /home/progo/pika/journal.py
    #!/usr/bin/env python3

    import sys
    print ('argv', sys.argv)
    sys.exit (0)


The faulty session might be a longer running one, if there's a way to
find out zsh-uptime, let me know. But I have several terminals open with
at least as old zsh sessions running, they behave correctly.

Unless X crashes or there's a long electricity break, I'll have these
sessions open if anyone comes up with something I can run to help with
this issue.

BR,
-- Mikael

Re: Two different zsh sessions handle $* differently

[Eitan Adler]

On 22 August 2017 at 13:22, Mikael Puhakka <mr.progo@gmail.com> wrote:
> Hi zsh-workers,
>
> Recently I noticed that my journal script has started to lose data. Then
> I noticed that this is somehow in how zsh handles my little helper
> function. This peculiarly affects one zsh session and not many others
> that I can see.

for key val in ${(kv)options}; do echo "$key -> $val"; done
in each shell please


-- 
Eitan Adler

Re: Two different zsh sessions handle $* differently

[Daniel Shahaf]

Eitan Adler wrote on Tue, 22 Aug 2017 14:40 -0400:
> for key val in ${(kv)options}; do echo "$key -> $val"; done
> in each shell please

For future reference, just 'set -o' does the same with less typing.

Re: Two different zsh sessions handle $* differently

[Mikael Puhakka]

They are the same in both:


    posixargzero -> off
    autolist -> on
    printexitvalue -> off
    kshzerosubscript -> off
    histreduceblanks -> off
    chaselinks -> off
    extendedglob -> off
    localloops -> off
    histnostore -> off
    automenu -> on
    globcomplete -> off
    unset -> on
    listpacked -> off
    promptsubst -> on
    shwordsplit -> off
    pathdirs -> off
    kshtypeset -> off
    listtypes -> on
    warncreateglobal -> off
    promptbang -> off
    dotglob -> off
    braceexpand -> on
    listbeep -> on
    correctall -> off
    privileged -> off
    numericglobsort -> off
    histverify -> off
    trackall -> on
    histsubstpattern -> off
    globstarshort -> off
    cbases -> off
    rcquotes -> off
    posixaliases -> off
    histfindnodups -> on
    bashautolist -> off
    sharehistory -> off
    overstrike -> off
    kshautoload -> off
    incappendhistory -> on
    appendcreate -> off
    promptcr -> on
    mailwarn -> off
    pushdignoredups -> off
    interactive -> on
    ignoreeof -> off
    globsubst -> off
    rematchpcre -> off
    monitor -> on
    histsavebycopy -> on
    histbeep -> on
    debugbeforecmd -> on
    magicequalsubst -> off
    rmstarsilent -> off
    posixjobs -> off
    hashcmds -> on
    posixtraps -> off
    extendedhistory -> off
    notify -> on
    kshoptionprint -> off
    histexpiredupsfirst -> off
    glob -> on
    posixcd -> off
    braceccl -> off
    badpattern -> on
    longlistjobs -> off
    banghist -> on
    dvorak -> off
    alwaystoend -> off
    hashall -> on
    globalexport -> on
    ksharrays -> off
    correct -> off
    autonamedirs -> off
    histexpand -> on
    typesetsilent -> off
    rmstarwait -> off
    histnofunctions -> off
    autoparamslash -> on
    trapsasync -> off
    sunkeyboardhack -> off
    promptsp -> on
    histsavenodups -> off
    autocd -> off
    allexport -> off
    posixidentifiers -> off
    cshjunkiehistory -> off
    autopushd -> off
    completeinword -> off
    completealiases -> off
    aliases -> on
    autocontinue -> off
    appendhistory -> on
    singlelinezle -> off
    hashlistall -> on
    ignoreclosebraces -> off
    recexact -> off
    localoptions -> off
    interactivecomments -> off
    errexit -> off
    cshjunkiequotes -> off
    markdirs -> off
    hashdirs -> on
    cdablevars -> off
    rcexpandparam -> off
    vi -> off
    printeightbit -> off
    multifuncdef -> on
    xtrace -> off
    login -> off
    cshjunkieloops -> off
    histappend -> on
    histignorespace -> off
    evallineno -> on
    shfileexpansion -> off
    rcs -> on
    functionargzero -> on
    errreturn -> off
    combiningchars -> off
    histignoredups -> on
    histfcntllock -> off
    beep -> on
    autoremoveslash -> on
    hup -> on
    globdots -> off
    autoparamkeys -> on
    shnullcmd -> off
    multibyte -> on
    zle -> on
    promptpercent -> on
    flowcontrol -> on
    continueonerror -> off
    incappendhistorytime -> off
    autoresume -> off
    globassign -> off
    caseglob -> on
    shortloops -> on
    bsdecho -> off
    cprecedences -> off
    log -> on
    transientrprompt -> off
    verbose -> off
    localpatterns -> off
    ignorebraces -> off
    pipefail -> off
    equals -> on
    menucomplete -> off
    cshnullglob -> off
    casematch -> on
    promptvars -> on
    histallowclobber -> off
    bareglobqual -> on
    shinstdin -> on
    restricted -> off
    pushdminus -> off
    nullglob -> off
    chasedots -> off
    mailwarning -> off
    listambiguous -> on
    cshnullcmd -> off
    bashrematch -> off
    octalzeroes -> off
    forcefloat -> off
    exec -> on
    multios -> on
    emacs -> off
    nomatch -> on
    pathscript -> off
    localtraps -> off
    stdin -> on
    onecmd -> off
    kshglob -> off
    clobber -> on
    posixbuiltins -> off
    alwayslastprompt -> on
    pushdtohome -> off
    histignorealldups -> off
    hashexecutablesonly -> off
    pushdsilent -> off
    shoptionletters -> off
    physical -> off
    sourcetrace -> off
    histlexwords -> off
    bgnice -> on
    globalrcs -> on
    posixstrings -> off
    checkjobs -> on
    shglob -> off
    singlecommand -> off
    listrowsfirst -> off

BR,
-- Mikael


On 8/22/17, Eitan Adler <lists@eitanadler.com> wrote:
> On 22 August 2017 at 13:22, Mikael Puhakka <mr.progo@gmail.com> wrote:
>> Hi zsh-workers,
>>
>> Recently I noticed that my journal script has started to lose data. Then
>> I noticed that this is somehow in how zsh handles my little helper
>> function. This peculiarly affects one zsh session and not many others
>> that I can see.
>
> for key val in ${(kv)options}; do echo "$key -> $val"; done
> in each shell please
>
>
> --
> Eitan Adler
>

Re: Two different zsh sessions handle $* differently

[Eitan Adler]

On 22 August 2017 at 15:04, Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
> Eitan Adler wrote on Tue, 22 Aug 2017 14:40 -0400:
>> for key val in ${(kv)options}; do echo "$key -> $val"; done
>> in each shell please
>
> For future reference, just 'set -o' does the same with less typing.

Good to know - thanks! I knew about "setopt" and "unsetopt" but not "set -o" :)


-- 
Eitan Adler

Re: Two different zsh sessions handle $* differently

[Phil Pennock]

On 2017-08-22 at 20:22 +0300, Mikael Puhakka wrote:
>     (~) whence -f j
>     j () {
>         noglob journal.py "$*"
>     }

This is broken.  If you invoke >> j f* << then the argument list passed
to the function is already expanded.  In this case, you want an alias
instead.  Then you can either have `journal.py` do
  ' '.join(sys.argv[1:])
for you, or still use a function to join things and use an
alias/function combination.  In your shoes, I'd have Python join the
parameters and _just_ use an alias.

Aliases happen before glob expansion and dispatch.  Functions are
invoked after determining their parameter list, too late for `noglob`
inside the function to help.

    # python script looks in all sys.argv[1:] :
    alias j='command journal.py'
or:
    # python script just looks at argv[1] :
    function f_j { journal.py "$*"; }
    alias j='noglob f_j'

>     (~) head -n5 /home/progo/pika/journal.py
>     #!/usr/bin/env python3

Which python3 is used?  Is it the same one in both environments, or is
pyenv or equivalent using shell environment variables or .pyenv-version
files to dispatch to a different interpreter, with one having sys.argv
getting mangled in the start-up files?

Is os.environ[PYTHONSTARTUP] defined, causing logic to happen before
your code sees sys.argv?

Do you have any suffix or global aliases defined in the older zsh
instance?  `alias -s`, `alias -g`?

    % j() { print -l :${^*} }
    % j Test. ABC this is a test.
    :Test.
    :ABC
    :this
    :is
    :a
    :test.
    % alias -g ABC='#'
    % j Test. ABC this is a test.
    :Test.
    %

You can use `setopt xtrace` (or `set -x`) to have zsh emit diagnostics
of what it's doing, so if you do that before invoking `j` then you might
see something different in the two environments.

-Phil

Re: Two different zsh sessions handle $* differently

[Mikael Puhakka]

On 8/23/17, Phil Pennock <zsh-workers+phil.pennock@spodhuis.org> wrote:
> This is broken.  If you invoke >> j f* << then the argument list passed
> to the function is already expanded.  In this case, you want an alias
> instead.

Thanks for pointing this out. I knew it didn't work as intended and I
was pondering why's that the case -- I had it as an alias originally
when things worked and then started to functionize things. Wasn't using
the alias much at the time so the regression went largely unnoticed.

> Which python3 is used?  Is it the same one in both environments, or is
> pyenv or equivalent using shell environment variables or .pyenv-version
> files to dispatch to a different interpreter, with one having sys.argv
> getting mangled in the start-up files?

Both examples run on home directory, the env points to the same python3
version 3.4.5, no virtualenvs.

> Is os.environ[PYTHONSTARTUP] defined, causing logic to happen before
> your code sees sys.argv?

Nothing defined there.

> Do you have any suffix or global aliases defined in the older zsh
> instance?  `alias -s`, `alias -g`?

No, nothing.

> You can use `setopt xtrace` (or `set -x`) to have zsh emit diagnostics
> of what it's doing, so if you do that before invoking `j` then you might
> see something different in the two environments.

This shows something alright:

    (~) j Test. ABC ABC ABC.
    +preexec:0> print -Pn '\e]2;zsh: %~\a'
    +zsh:1165> j Test. ABC ABC ABC.
(*) +j:0> journal.py Test.ABCABCABC.
    argv ['Test.']
    +precmd:0> print -Pn '\e]2;zsh: %~\a'
    +zsh:1166> parse_git_branch
    +parse_git_branch:1> git branch --no-color
    +parse_git_branch:2> sed -e '/^[^*]/d' -e 's/* \(.*\)/(\1)/'

The starred line doesn't look correct. With a fresh session the same
line goes like this:

    +j:0> journal.py 'Test. ABC ABC ABC.'

Now I'm not sure what happens. The faulty one strips whitespace and
concatenates, (there doesn't seem to be nulls or anything separating the
arguments) yet python sees only a part of the argument.

Hope this helps.

BR,
-- Mikael

Re: Two different zsh sessions handle $* differently

[Daniel Shahaf]

Mikael Puhakka wrote on Thu, 24 Aug 2017 09:05 +0300:
> On 8/23/17, Phil Pennock <zsh-workers+phil.pennock@spodhuis.org> wrote:
> > You can use `setopt xtrace` (or `set -x`) to have zsh emit diagnostics
> > of what it's doing, so if you do that before invoking `j` then you might
> > see something different in the two environments.
> 
> This shows something alright:
> 
>     (~) j Test. ABC ABC ABC.
>     +preexec:0> print -Pn '\e]2;zsh: %~\a'
>     +zsh:1165> j Test. ABC ABC ABC.
> (*) +j:0> journal.py Test.ABCABCABC.
>     argv ['Test.']
>     +precmd:0> print -Pn '\e]2;zsh: %~\a'
>     +zsh:1166> parse_git_branch
>     +parse_git_branch:1> git branch --no-color
>     +parse_git_branch:2> sed -e '/^[^*]/d' -e 's/* \(.*\)/(\1)/'
> 
> The starred line doesn't look correct. With a fresh session the same
> line goes like this:
> 
>     +j:0> journal.py 'Test. ABC ABC ABC.'
> 
> Now I'm not sure what happens. The faulty one strips whitespace and
> concatenates, (there doesn't seem to be nulls or anything separating the
> arguments) yet python sees only a part of the argument.

Have you set $IFS to $'\0'?  Check with 'typeset -p IFS'.

Re: Two different zsh sessions handle $* differently

[Mikael Puhakka]

On 8/24/17, Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
> Have you set $IFS to $'\0'?  Check with 'typeset -p IFS'.
>

That's it!

    (~) typeset -p IFS
    typeset IFS=' 	
    '

vs. the faulty one:

    (~) typeset -p IFS
    typeset IFS=

Now, this might have happened accidentally at some point but there's no
history for it, sadly.

Big thanks to everyone!

BR,
-- Mikael