git completion horribly slow in kernel tree

git completion horribly slow in kernel tree

[Jörg Sommer]

Hi,

using completion for git in the kernel tree is horribly slow. It takes
more than ten seconds to get the answer that no completion for “git log
o” is available. The reason might be that git ls-files gives 23,000 file
names and zsh puts them in an array. It there any chance to speed this
up?

Bye, Jörg.
-- 
[dpkg] We are the apt. Resistance is futile. You will be packaged.

Re: git completion horribly slow in kernel tree

[Mikael Magnusson]

On 14/01/2008, Jörg Sommer <joerg@alea.gnuu.de> wrote:
> Hi,
>
> using completion for git in the kernel tree is horribly slow. It takes
> more than ten seconds to get the answer that no completion for "git log
> o" is available. The reason might be that git ls-files gives 23,000 file
> names and zsh puts them in an array. It there any chance to speed this
> up?

I tried to fix that once but I gave up because I don't use any git
trees that large usually. basically the problem is _git calls
git-ls-tree with -r, which means recursive. I have this  patch lying
around but it isn't finished. I don't remember what isn't finished
either but maybe it helps someone anyway?

diff --git a/Completion/Unix/Command/_git b/Completion/Unix/Command/_git
index 5f71848..0f3db0e 100644
--- a/Completion/Unix/Command/_git
+++ b/Completion/Unix/Command/_git
@@ -826,12 +826,12 @@ _git-diff-tree () {
         # new tree:
         #   ...
         _alternative \
-          "original tree:original tree:__git_tree_files $line[1]" \
-          "new tree:new tree:__git_tree_files $line[2]" && ret=0
+          "original tree:original tree:__git_tree_files . $line[1]" \
+          "new tree:new tree:__git_tree_files . $line[2]" && ret=0
       else
         _alternative \
           ': :__git_tree_ishs' \
-          ": :__git_tree_files $line[1]" && ret=0
+          ": :__git_tree_files . $line[1]" && ret=0
       fi
       ;;
   esac
@@ -924,7 +924,7 @@ _git-ls-tree () {

   case $state in
     files)
-      __git_tree_files $line[1] && ret=0
+      __git_tree_files . $line[1] && ret=0
       ;;
   esac
 }
@@ -1366,7 +1366,7 @@ _git-archive () {

   case $state in
     (files)
-      __git_tree_files $line[1] && ret=0
+      __git_tree_files . $line[1] && ret=0
       ;;
   esac
 }
@@ -1504,7 +1504,7 @@ _git-checkout () {
   case $state in
     (files)
       if [[ -n $line[1] ]]; then
-        __git_tree_files $line[1] && ret=0
+        __git_tree_files . $line[1] && ret=0
       else
         __git_cached_files && ret=0
       fi
@@ -1690,13 +1690,13 @@ _git-grep () {
         else
           _alternative \
             'tree:tree:__git_trees' \
-            "tree file:tree-files:__git_tree_files
$line[first_tree,last_tree]" && ret=0
+            "tree file:tree-files:__git_tree_files .
$line[first_tree,last_tree]" && ret=0
         fi
       else
         if (( first_tree == 0 )); then
           __git_cached_files
         else
-          __git_tree_files $line[first_tree,last_tree]
+          __git_tree_files . $line[first_tree,last_tree]
         fi
       fi
       ;;
@@ -1836,7 +1836,7 @@ _git-reset () {

   if [[ $words[2] == --mixed ]]; then
     commit_arg=':commit:__git_revisions'
-    path_arg="*:file:__git_tree_files $words[3]"
+    path_arg="*:file:__git_tree_files . $words[3]"
   else
     commit_arg='::commit:__git_revisions'
   fi
@@ -2559,7 +2559,7 @@ __git_command_successful () {
 __git_objects () {
   compset -P '*:'
   if [[ -n $IPREFIX ]]; then
-    __git_tree_files ${IPREFIX%:}
+    __git_tree_files "$PREFIX" "${IPREFIX%:}"
   else
     _alternative \
       'revisions:revision:__git_revisions' \
@@ -2652,13 +2652,16 @@ __git_tree_files () {

   zparseopts -D -E -a multi_parts_opts M: J: V: 1 2 n F: X:

-  local tree
+  local tree Path
   integer at_least_one_tree_added
   local -a tree_files

+  Path=${1%/*}/
+  [[ $Path = / ]] && Path=.
+  shift
   (( at_least_one_tree_added = 0 ))
   for tree in $*; do
-    tree_files+=(${(ps:\0:)"$(_call_program tree-files git ls-tree
--name-only -z -r $tree 2>/dev/null)"})
+    tree_files+=(${(ps:\0:)"$(_call_program tree-files git ls-tree
--name-only -z $tree $Path 2>/dev/null)"})
     __git_command_successful && (( at_least_one_tree_added = 1 ))
   done



-- 
Mikael Magnusson

Re: git completion horribly slow in kernel tree

[Clint Adams]

On Mon, Jan 14, 2008 at 10:39:58PM +0100, Mikael Magnusson wrote:
> I tried to fix that once but I gave up because I don't use any git
> trees that large usually. basically the problem is _git calls
> git-ls-tree with -r, which means recursive. I have this  patch lying
> around but it isn't finished. I don't remember what isn't finished
> either but maybe it helps someone anyway?

Committing this, and hopefully someone will figure it out quickly if
there's anything missing.

Re: git completion horribly slow in kernel tree

[Mikael Magnusson]

On 22/01/2008, Clint Adams <clint@zsh.org> wrote:
> On Mon, Jan 14, 2008 at 10:39:58PM +0100, Mikael Magnusson wrote:
> > I tried to fix that once but I gave up because I don't use any git
> > trees that large usually. basically the problem is _git calls
> > git-ls-tree with -r, which means recursive. I have this  patch lying
> > around but it isn't finished. I don't remember what isn't finished
> > either but maybe it helps someone anyway?
>
> Committing this, and hopefully someone will figure it out quickly if
> there's anything missing.

Hm maybe my disclaimer wasn't clear enough, the code doesn't actually work.
Specifically it doesn't complete anything after typing a partial
directory name after a sha1: expression, so git show HEAD:<tab> will
list all dirs and files, but HEAD:D<tab> won't complete to Doc.

-- 
Mikael Magnusson

Re: git completion horribly slow in kernel tree

[Clint Adams]

On Tue, Jan 22, 2008 at 02:40:03AM +0100, Mikael Magnusson wrote:
> Hm maybe my disclaimer wasn't clear enough, the code doesn't actually work.
> Specifically it doesn't complete anything after typing a partial
> directory name after a sha1: expression, so git show HEAD:<tab> will
> list all dirs and files, but HEAD:D<tab> won't complete to Doc.

This helps with that case.  Two things that are still broken are
1) suffix of completed directories is a space, not /
2) completion of dir/ won't work; you need to input the next letter
first.

Index: Completion/Unix/Command/_git
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Unix/Command/_git,v
retrieving revision 1.55
diff -u -r1.55 _git
--- Completion/Unix/Command/_git	22 Jan 2008 01:33:40 -0000	1.55
+++ Completion/Unix/Command/_git	22 Jan 2008 16:16:24 -0000
@@ -2655,8 +2655,7 @@
   integer at_least_one_tree_added
   local -a tree_files
 
-  Path=${1%/*}/
-  [[ $Path = / ]] && Path=.
+  Path=${1:h}/
   shift
   (( at_least_one_tree_added = 0 ))
   for tree in $*; do

Re: git completion horribly slow in kernel tree

[Clint Adams]

On Tue, Jan 22, 2008 at 11:22:22AM -0500, Clint Adams wrote:
> 2) completion of dir/ won't work; you need to input the next letter
> first.

Index: Completion/Unix/Command/_git
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Unix/Command/_git,v
retrieving revision 1.56
diff -u -r1.56 _git
--- Completion/Unix/Command/_git	22 Jan 2008 16:44:10 -0000	1.56
+++ Completion/Unix/Command/_git	22 Jan 2008 17:03:26 -0000
@@ -2655,7 +2655,7 @@
   integer at_least_one_tree_added
   local -a tree_files
 
-  Path=${1:h}/
+  [[ "$1" == */ ]] && Path="$1" || Path="${1:h}/"
   shift
   (( at_least_one_tree_added = 0 ))
   for tree in $*; do

Re: git completion horribly slow in kernel tree

[Nikolai Weibull]

On Jan 22, 2008 6:04 PM, Clint Adams <clint@zsh.org> wrote:

> -  Path=${1:h}/

Please, can we keep to the general style in the file?  If you're
worried about overriding a global variable, use a different name.
Capitalized variable names is for VB.

Re: git completion horribly slow in kernel tree

[Mikael Magnusson]

On 22/01/2008, Nikolai Weibull <now@bitwi.se> wrote:
> On Jan 22, 2008 6:04 PM, Clint Adams <clint@zsh.org> wrote:
>
> > -  Path=${1:h}/
>
> Please, can we keep to the general style in the file?  If you're
> worried about overriding a global variable, use a different name.
> Capitalized variable names is for VB.

Don't blame me, I didn't send it for inclusion. :)

-- 
Mikael Magnusson

Re: git completion horribly slow in kernel tree

[Clint Adams]

On Tue, Jan 22, 2008 at 11:22:22AM -0500, Clint Adams wrote:
> 1) suffix of completed directories is a space, not /

Any other issues left?

Index: Completion/Unix/Command/_git
===================================================================
RCS file: /cvsroot/zsh/zsh/Completion/Unix/Command/_git,v
retrieving revision 1.57
diff -u -r1.57 _git
--- Completion/Unix/Command/_git	22 Jan 2008 17:07:17 -0000	1.57
+++ Completion/Unix/Command/_git	22 Jan 2008 17:11:17 -0000
@@ -2668,9 +2668,7 @@
   fi
 
   local expl
-
-  # FIXME: Why doesn’t -f to compadd work here?
-  _wanted files expl 'tree file' _multi_parts $multi_parts_opts - / tree_files
+  _wanted files expl 'tree file' compadd $multi_parts_opts -f -a tree_files
 }
 
 # TODO: deal with things that __git_heads and __git_tags has in common (i.e.,

Re: git completion horribly slow in kernel tree

[Mikael Magnusson]

On 22/01/2008, Clint Adams <clint@zsh.org> wrote:
> On Tue, Jan 22, 2008 at 11:22:22AM -0500, Clint Adams wrote:
> > 1) suffix of completed directories is a space, not /
>
> Any other issues left?

I don't know if this problem is due to this change, but git add
completion doesn't work from subdirs when the file to add is above the
current dir, even if you write the ../ yourself (which i think you
should have to).

-- 
Mikael Magnusson

Re: git completion horribly slow in kernel tree

[Clint Adams]

On Wed, Jan 23, 2008 at 12:20:26PM +0100, Mikael Magnusson wrote:
> I don't know if this problem is due to this change, but git add
> completion doesn't work from subdirs when the file to add is above the
> current dir, even if you write the ../ yourself (which i think you
> should have to).

Probably directly related to git ls-tree HEAD .. not doing what
the function expects.

How can we achieve the equivalent with git?

Re: git completion horribly slow in kernel tree

[Mikael Magnusson]

On 23/01/2008, Clint Adams <clint@zsh.org> wrote:
> On Wed, Jan 23, 2008 at 12:20:26PM +0100, Mikael Magnusson wrote:
> > I don't know if this problem is due to this change, but git add
> > completion doesn't work from subdirs when the file to add is above the
> > current dir, even if you write the ../ yourself (which i think you
> > should have to).
>
> Probably directly related to git ls-tree HEAD .. not doing what
> the function expects.
>
> How can we achieve the equivalent with git?

Hm, I don't really know... git ls-tree seems to not want to output ..,
if you specify --full-name, it does show the dir you specify, but
relative to the root of the project. So if you're in proj/dir1/dir2
and git ls-tree --full-name --name-only HEAD .., you will get files
shown as dir1/file1. I guess it's possible to use that output, but
maybe it's easier to just manually cd to the proj/ dir first :). I
just looked at the bash completer (which is very confusing code), and
it seems to use something like
git --git-dir=$(git rev-parse --git-dir) ls-tree $ref:prefix
which lists the file as file1 instead of dir1/file1. (no idea why it
doesn't use --name-only).

-- 
Mikael Magnusson

Re: git completion horribly slow in kernel tree

[Clint Adams]

On Wed, Jan 23, 2008 at 05:30:05PM +0100, Mikael Magnusson wrote:
> Hm, I don't really know... git ls-tree seems to not want to output ..,
> if you specify --full-name, it does show the dir you specify, but
> relative to the root of the project. So if you're in proj/dir1/dir2
> and git ls-tree --full-name --name-only HEAD .., you will get files
> shown as dir1/file1. I guess it's possible to use that output, but
> maybe it's easier to just manually cd to the proj/ dir first :). I
> just looked at the bash completer (which is very confusing code), and
> it seems to use something like
> git --git-dir=$(git rev-parse --git-dir) ls-tree $ref:prefix
> which lists the file as file1 instead of dir1/file1. (no idea why it
> doesn't use --name-only).

I just looked and git-add completion is using ls-files, which can't
handle relative paths either.

Re: git completion horribly slow in kernel tree

[Mikael Magnusson]

On 23/01/2008, Clint Adams <clint@zsh.org> wrote:
> On Wed, Jan 23, 2008 at 05:30:05PM +0100, Mikael Magnusson wrote:
> > Hm, I don't really know... git ls-tree seems to not want to output ..,
> > if you specify --full-name, it does show the dir you specify, but
> > relative to the root of the project. So if you're in proj/dir1/dir2
> > and git ls-tree --full-name --name-only HEAD .., you will get files
> > shown as dir1/file1. I guess it's possible to use that output, but
> > maybe it's easier to just manually cd to the proj/ dir first :). I
> > just looked at the bash completer (which is very confusing code), and
> > it seems to use something like
> > git --git-dir=$(git rev-parse --git-dir) ls-tree $ref:prefix
> > which lists the file as file1 instead of dir1/file1. (no idea why it
> > doesn't use --name-only).
>
> I just looked and git-add completion is using ls-files, which can't
> handle relative paths either.

Oh hm yeah, I tried th e bash completer and it completes all files for
git add, not just new/changed ones.

-- 
Mikael Magnusson

Re: git completion horribly slow in kernel tree

[Mikael Magnusson]

On 23/01/2008, Clint Adams <clint@zsh.org> wrote:
> On Wed, Jan 23, 2008 at 05:30:05PM +0100, Mikael Magnusson wrote:
> > Hm, I don't really know... git ls-tree seems to not want to output ..,
> > if you specify --full-name, it does show the dir you specify, but
> > relative to the root of the project. So if you're in proj/dir1/dir2
> > and git ls-tree --full-name --name-only HEAD .., you will get files
> > shown as dir1/file1. I guess it's possible to use that output, but
> > maybe it's easier to just manually cd to the proj/ dir first :). I
> > just looked at the bash completer (which is very confusing code), and
> > it seems to use something like
> > git --git-dir=$(git rev-parse --git-dir) ls-tree $ref:prefix
> > which lists the file as file1 instead of dir1/file1. (no idea why it
> > doesn't use --name-only).
>
> I just looked and git-add completion is using ls-files, which can't
> handle relative paths either.

Hm, this "works", but it definitely isn't nice, and doesn't work when
you type the shorter path manually first.
# Untracked files:
#	Src/Modules/curses_keys.h
if i'm sitting in Src/Builtins, the following code will produce
git add <tab>
git add ../<tab>
git add ../../<tab>
git add ../../Src/Modules<tab>
git add ../../Src/Modules/curses_keys.h
which is not so great.
Annoyingly, when i'm sitting in Src/Builtins, git status produces the following
# Untracked files:
#	../Modules/curses_keys.h

I don't know how often one really wants to git add files above the
current dir, but it just annoys me that it's so hard to fix. :)

(( $+functions[__git_files] )) ||
__git_files () {
  local expl files ls_opts opts gitdir cdup

  zparseopts -D -E -a opts -- -cached -deleted -modified -others
-ignored -unmerged -killed

  gitdir=$(_call_program gitdir git rev-parse --git-dir 2>/dev/null)
  __git_command_successful || return

  ls_opts=("--exclude-per-directory=.gitignore")
  [[ -f "$gitdir/info/exclude" ]] &&
ls_opts+="--exclude-from=$gitdir/info/exclude"

  cdup=$(_call_program revparse git rev-parse --show-cdup)
  files=(${(ps:\0:)"$(_call_program files git ls-files --full-name -z
$ls_opts $opts $cdup 2>/dev/null)"})
  files=$cdup${^files}
  __git_command_successful || return

  _wanted files expl 'index file' _multi_parts $@ - / files
}


-- 
Mikael Magnusson

Re: git completion horribly slow in kernel tree

[Mikael Magnusson]

On 24/01/2008, Mikael Magnusson <mikachu@gmail.com> wrote:
> Hm, this "works", but it definitely isn't nice, and doesn't work when
> you type the shorter path manually first.
> # Untracked files:
> #       Src/Modules/curses_keys.h
> if i'm sitting in Src/Builtins, the following code will produce
> git add <tab>
> git add ../<tab>
> git add ../../<tab>
> git add ../../Src/Modules<tab>
> git add ../../Src/Modules/curses_keys.h
> which is not so great.

I should note that what's extra bad about it is that it does the same
thing in the Src/Modules dir, so don't commit it or anything. I just
posted it in the hopes that it might help somehow.

-- 
Mikael Magnusson