compinit causes completion to fail?

compinit causes completion to fail?

[TJ Luoma]

I split my .zshenv and .zshrc into several smaller files which are sourced separately, and now I've broken completion :-/

If I include these two lines in my zshenv file[1]:

autoload -U compinit
compinit



then completion fails with this message:

_complete:117: command not found: _normal

and a ~/.zcompdump file is created [2]

but if I comment out those two lines:

#autoload -U compinit
#compinit



then completion seems to work OK.

HOWEVER, if I remove my ~/.zshenv and all related ~/.z* files, and then use 'autoload -U compinit' and 'compinit' then it works fine, and the 'zcompdump' file looks better [4], which further solidifies my belief that the problem here is something I've done in my .zshenv files… but what?


Did I fail to load something before turning on compinit, or did I set something too early? Bindkeys, perhaps?


If you look at the 'zcompdump' file [2], specifically at line 117, it says:

bindkey '^[g" get-lin'   

that looks wrong to me. In fact, ALL of the bindkey lines look like they have lost the last character.

This started immediately after I split my ~/.zshenv file into several different files. You can see them all here [3]. This is using 4.3.11 on Mac OS X 10.7.3.  

I have no idea what I've done here, but I'd appreciate any help.


TjL


[1]  
https://www.dropbox.com/sh/7t1jagj5clkz0ci/_nzmp1eO9b/interactive/completion.sh





[2]
http://dl.dropbox.com/u/18414/tmp/zcompdump.txt

[3]
https://www.dropbox.com/sh/7t1jagj5clkz0ci/28xgQ85Mye

[4]
http://dl.dropbox.com/u/18414/tmp/zcompdump2.txt

Re: compinit causes completion to fail?

[TJ Luoma]

To answer my own question:

IFS='
'

is what was causing the problem.

I'm now learning about using the f expansion flag instead of mucking with IFS for splitting arrays on newlines

TjL

  

On Sunday, April 29, 2012 at 1:21 AM, TJ Luoma wrote:

>  
> I split my .zshenv and .zshrc into several smaller files which are sourced separately, and now I've broken completion :-/
>  
> If I include these two lines in my zshenv file[1]:
>  
> autoload -U compinit
> compinit
>  
>  
>  
> then completion fails with this message:
>  
> _complete:117: command not found: _normal
>  
> and a ~/.zcompdump file is created [2]
>  
> but if I comment out those two lines:
>  
> #autoload -U compinit
> #compinit
>  
>  
>  
> then completion seems to work OK.
>  
> HOWEVER, if I remove my ~/.zshenv and all related ~/.z* files, and then use 'autoload -U compinit' and 'compinit' then it works fine, and the 'zcompdump' file looks better [4], which further solidifies my belief that the problem here is something I've done in my .zshenv files… but what?
>  
>  
> Did I fail to load something before turning on compinit, or did I set something too early? Bindkeys, perhaps?
>  
>  
> If you look at the 'zcompdump' file [2], specifically at line 117, it says:
>  
> bindkey '^[g" get-lin'  
>  
> that looks wrong to me. In fact, ALL of the bindkey lines look like they have lost the last character.
>  
> This started immediately after I split my ~/.zshenv file into several different files. You can see them all here [3]. This is using 4.3.11 on Mac OS X 10.7.3.  
>  
> I have no idea what I've done here, but I'd appreciate any help.
>  
>  
> TjL
>  
>  
> [1]  
> https://www.dropbox.com/sh/7t1jagj5clkz0ci/_nzmp1eO9b/interactive/completion.sh
>  
>  
>  
>  
>  
> [2]
> http://dl.dropbox.com/u/18414/tmp/zcompdump.txt
>  
> [3]
> https://www.dropbox.com/sh/7t1jagj5clkz0ci/28xgQ85Mye
>  
> [4]
> http://dl.dropbox.com/u/18414/tmp/zcompdump2.txt

Re: compinit causes completion to fail?

[Peter Stephenson]

On Sun, 29 Apr 2012 02:57:56 -0400
TJ Luoma <luomat@gmail.com> wrote:
> To answer my own question:
> 
> IFS='
> '
> 
> is what was causing the problem.
> 
> I'm now learning about using the f expansion flag instead of mucking
> with IFS for splitting arrays on newlines

I'd certainly recommend parameter expansion for this kind of thing,
despite its rococo aspects, but note that you should be able to set
IFS as a prefix to a command line and have it restored after the
command line is executed, otherwise it's a bug.  However, that has its
own quirks; in particular, the IFS isn't set until after the command
line is fully expanded.  So I think you're going the right way.

-- 
Peter Stephenson <p.w.stephenson@ntlworld.com>
Web page now at http://homepage.ntlworld.com/p.w.stephenson/

Re: compinit causes completion to fail?

[Bart Schaefer]

On Apr 29,  2:57am, TJ Luoma wrote:
} 
} IFS='
} '
} 
} is what was causing the problem.

Could you say a bit more about exactly how this caused a problem?  In
normal circumstances the value of IFS shouldn't affect parsing of
scripts etc., it should only affect "read" and parameter substitution.
So I'm puzzled as to how you got the command-not-found symptom.

Re: compinit causes completion to fail?

[TJ Luoma]

On Apr 29, 2012, at 8:26 PM, Bart Schaefer <schaefer@brasslantern.com> wrote:

On Apr 29,  2:57am, TJ Luoma wrote:
}
} IFS='
} '
}
} is what was causing the problem.

Could you say a bit more about exactly how this caused a problem?  In
normal circumstances the value of IFS shouldn't affect parsing of
scripts etc., it should only affect "read" and parameter substitution.
So I'm puzzled as to how you got the command-not-found symptom.

Well, I don't know exactly either.  Here's what I did:

I had split my zshenv and zshrc files into several files. The
~/.zshenv was skeleton which sourced the other files as appropriate
(aside: it does seem faster this way).

But when I did that, completion broke with that "_normal not found" error.

So I removed the ~/.zshenv file, and completion worked (with no customizations).

{Note: by completion I mean just the basics: command completion and
'cd' completion.}

Then I started sourcing files one at a time and testing completion
(deleting ~/.zcompdump between each test by telling ~/.zlogout to
delete the file and exiting the shell after each one).

Completion broke ONLY when I added the explicit call to 'autoload'
completion stuff, which seemed completely nonsensical.

So I started all over again with a completely empty .zshenv EXCEPT for
the calls the source the various files.  I thought it might have to do
with the *order* something was being loaded (bindkeys before setopts,
etc) so I organized my .zshenv to source the files in a way that I
thought made sense (i.e. environmental variables first, setting my
prompt last, etc).

I left everything commented out, and started un-commenting things one
at a time, from the bottom-up (I don't know why I chose the bottom-up,
it seemed like a good idea at the time).

So it went like this:

* uncomment last file-to-source from within .zshenv
* start a new shell
* test to see if command and 'cd' completion worked
* exit shell (deleting ~/.zcompdump)

* uncomment last-1 file-to-source from within .zshenv
* start a new shell
* test to see if command and 'cd' completion worked
* exit shell (deleting ~/.zcompdump)

and so on.

When I got to the top of the .zshenv file, I realized completion STILL
worked. So I looked at that .zshenv compared to the one I had used
earlier (Thank you, Dropbox versioning!)

The only difference was that the old .zshenv was that I had put in a
few settings that I thought were important at the top of the .zshenv,
including the IFS line.

When I had a giant .zshenv/.zshrc file, the IFS line was after the
completion stuff, but when I had broken it up into pieces, I thought
that I should put the IFS near the top because it seemed like
otherwise I might get inconsistent results. (Not sure if that would
have been a problem, but that was my thinking.)

I started adding those "odds and ends" back into the top of my
.zshenv, and then did the same logout/login.

Everything worked fine UNTIL I put the IFS line in at the top of the
.zshenv. When I did that, completion broke. When I took it out,
completion worked.

Truthfully, I forgot why I had the IFS line in there, so I had to go
back to Google (which wasn't easy) and realized that I had done that
for when I was running a command and capturing the output into an
array, and the output of the command might have newlines in it.

I went *back* to Google and looked for an alternative, assuming that
there was a better way.

That's when I learned about defining IFS at the front of the array, like this:

IFS=$'\n' array_of_lines=($(my_command))

or

array_of_lines=("${(@f)$(my_command)}")

(both of those examples gleaned from
http://unix.stackexchange.com/questions/29724/how-to-properly-collect-an-array-of-lines-in-zsh
FYI)

Then I went to the bash-to-zsh book and learned more about it, or tried to.

I think that's about all I can remember.

No, I don't think it makes sense either.

TjL

Re: compinit causes completion to fail?

[Bart Schaefer]

On Apr 30,  1:04am, TJ Luoma wrote:
}
} So it went like this:
} 
} * uncomment last file-to-source from within .zshenv
} * start a new shell
} * test to see if command and 'cd' completion worked
} * exit shell (deleting ~/.zcompdump)
} 
} * uncomment last-1 file-to-source from within .zshenv
} * start a new shell
} * test to see if command and 'cd' completion worked
} * exit shell (deleting ~/.zcompdump)
} 
} and so on.

Future hint:  Uncomment half the lines, then try.  If it works, leave
those uncommented and uncomment half the remaining lines.  If it fails,
re-comment half way back to the last place it succeeded.  Repeat.

(Binary search usually takes fewer comparisons than linear search.)

} Everything worked fine UNTIL I put the IFS line in at the top of the
} .zshenv. When I did that, completion broke. When I took it out,
} completion worked.

Did some quick comparison of the output of "setopt xtrace" with IFS=$'\n'
vs. the default.

I wrote:
> IFS shouldn't affect parsing of scripts etc., it should only affect
> "read" and parameter substitution.

Turns out compinit uses "read" to parse #compdef lines while walking
$fpath.  When IFS=$'\n' the entire line is in the first element of
the array, so compdef is run with the wrong arguments.  Then compdef
itself uses "read" to parse bindkey output, etc.

I'm not sure if it's worthwhile to fix this as having IFS=$'\n' during
a whole interactive shell session is likely to cause all sorts of havoc,
but here's a patch anyway.


Index: Completion/compinit
===================================================================
retrieving revision 1.19
diff -u -r1.19 compinit
--- Completion/compinit	20 Dec 2011 17:13:37 -0000	1.19
+++ Completion/compinit	30 Apr 2012 14:28:21 -0000
@@ -326,7 +326,7 @@
         [[ $2 = .menu-select ]] && zmodload -i zsh/complist
 	zle -C "$1" "$2" "$func"
 	if [[ -n $new ]]; then
-	  bindkey "$3" | read -A opt
+	  bindkey "$3" | IFS=$' \t' read -A opt
 	  [[ $opt[-1] = undefined-key ]] && bindkey "$3" "$1"
 	else
 	  bindkey "$3" "$1"
@@ -353,7 +353,7 @@
       # And bind the keys...
       for i; do
         if [[ -n $new ]]; then
-	   bindkey "$i" | read -A opt
+	   bindkey "$i" | IFS=$' \t' read -A opt
 	   [[ $opt[-1] = undefined-key ]] || continue
 	fi
         bindkey "$i" "$func"
@@ -469,7 +469,7 @@
 
 if [[ -f "$_comp_dumpfile" ]]; then
   if [[ -n "$_i_check" ]]; then
-    read -rA _i_line < "$_comp_dumpfile"
+    IFS=$' \t' read -rA _i_line < "$_comp_dumpfile"
     if [[ _i_autodump -eq 1 && $_i_line[2] -eq $#_i_files &&
         $ZSH_VERSION = $_i_line[4] ]]
     then
@@ -491,7 +491,7 @@
       _i_name="${_i_file:t}"
       (( $+_i_test[$_i_name] + $_i_wfiles[(I)$_i_file] )) && continue
       _i_test[$_i_name]=yes
-      read -rA _i_line < $_i_file
+      IFS=$' \t' read -rA _i_line < $_i_file
       _i_tag=$_i_line[1]
       shift _i_line
       case $_i_tag in
@@ -527,7 +527,7 @@
 
 # If the default completer set includes _expand, and tab is bound
 # to expand-or-complete, rebind it to complete-word instead.
-bindkey '^i' | read -A _i_line
+bindkey '^i' | IFS=$' \t' read -A _i_line
 if [[ ${_i_line[2]} = expand-or-complete ]] &&
   zstyle -a ':completion:' completer _i_line &&
   (( ${_i_line[(i)_expand]} <= ${#_i_line} )); then

-- 
Barton E. Schaefer

Re: compinit causes completion to fail?

[TJ Luoma]

On Monday, April 30, 2012 at 10:37 AM, Bart Schaefer wrote:

> Future hint: Uncomment half the lines, then try. If it works, leave
> those uncommented and uncomment half the remaining lines. If it fails,
> re-comment half way back to the last place it succeeded. Repeat.
>  
> (Binary search usually takes fewer comparisons than linear search.)
Oh, I know… I remember that from CS101 all those years ago… I actually did group a few of them, but I had a few false positives and negatives (I think due to ~/.zcompdump not being deleted) so eventually I just went to the awfukkit stage of debugging. :-)
  
> Turns out compinit uses "read" to parse #compdef lines while walking
> $fpath. When IFS=$'\n' the entire line is in the first element of
> the array, so compdef is run with the wrong arguments. Then compdef
> itself uses "read" to parse bindkey output, etc.
>  
> I'm not sure if it's worthwhile to fix this as having IFS=$'\n' during
> a whole interactive shell session is likely to cause all sorts of havoc,
> but here's a patch anyway.

FWIW I have used

IFS='
'

in my .zshrc for ages, and it has actually proven very helpful as far as parsing arrays, etc. I realize it's not the ideal solution but this is the first time that I know of that it has caused a problem.

Then again, my usage of zsh is so basic that I may not be attempting some of the things which would have otherwise caused problems.

Thanks again for your time. I can't tell you how many times I've googled something about ZSH only to find the answer from some online cache of this mailing list.

TjL