Git completion slowness

Git completion slowness

[Brett Simmers]

[-- Attachment #1: Type: text/plain, Size: 991 bytes --]

I asked about this in IRC and got a request to send some details to the
mailing list, so here they are -

Whenever I try to use tab completion with a git command, zsh hangs for a few
minutes using full cpu before coming back with an answer.  It usually isn't
causing much disk activity, just cpu.  The command I try to use most often
is "git checkout foo<tab>", where foo is the first few letters of a branch
name, but the slowness seems to also happen anywhere else I need to complete
a filename or branch/tag name.

Some details of my repository:
- 96,000 files
- .git is 17GB
- Deepest directory nesting in the repository is 15 levels but most files
are no deeper than 7 levels

I'm running zsh 4.3.10 with _git from today's CVS.  Host OS is Ubuntu
10.04.1, 6GB of RAM with an Intel Core2Quad 2.4ghz.  My git repo is on an
SSD (OSZ Vertex I think), so none of the hardware should be a bottleneck.
 Bash-completion on the same setup in the same repo works fine without any
delays.

-Brett

Re: Git completion slowness

[Benjamin R. Haskell]

On Thu, 9 Dec 2010, Brett Simmers wrote:

> I asked about this in IRC and got a request to send some details to 
> the mailing list, so here they are -
>
> Whenever I try to use tab completion with a git command, zsh hangs for 
> a few minutes using full cpu before coming back with an answer.  It 
> usually isn't causing much disk activity, just cpu.  The command I try 
> to use most often is "git checkout foo<tab>", where foo is the first 
> few letters of a branch name, but the slowness seems to also happen 
> anywhere else I need to complete a filename or branch/tag name.
>
> Some details of my repository:
> - 96,000 files
> - .git is 17GB
> - Deepest directory nesting in the repository is 15 levels but most 
> files are no deeper than 7 levels
>
> I'm running zsh 4.3.10 with _git from today's CVS.  Host OS is Ubuntu 
> 10.04.1, 6GB of RAM with an Intel Core2Quad 2.4ghz.  My git repo is on 
> an SSD (OSZ Vertex I think), so none of the hardware should be a 
> bottleneck.  Bash-completion on the same setup in the same repo works 
> fine without any delays.

I'm still using the workaround of pre-defining this function in a 
startup script:

function __git_files () {}

That prevents all git file completion, which is the cause of the 
slowness (specifically, _multi_parts completion for filenames).  Recall 
that paths are valid arguments to a `git checkout`.

It does mean that git completion won't ever offer up filenames, which I 
can live with, because:

1) I have <C-x><C-f> bound to filename completion (so I can ^X^F 
complete, if I really don't want to type the name, and it exists on 
disk)

and

2) It prevents pegging a CPU at 100% for minutes.

Prior messages:

See: http://www.zsh.org/mla/users/2010/msg00434.html
and my response at: http://www.zsh.org/mla/users/2010/msg00435.html
original diagnosis of _multi_parts being the cause: http://www.zsh.org/mla/workers/2008/msg01535.html

-- 
Best,
Ben

Re: Git completion slowness

[Brett Simmers]

[-- Attachment #1: Type: text/plain, Size: 525 bytes --]

On Thu, Dec 9, 2010 at 2:18 PM, Benjamin R. Haskell <zsh@benizi.com> wrote:

>
> I'm still using the workaround of pre-defining this function in a startup
> script:
>
> function __git_files () {}
>
> That prevents all git file completion, which is the cause of the slowness
> (specifically, _multi_parts completion for filenames).  Recall that paths
> are valid arguments to a `git checkout`.
>
>
Thanks, that works great for now.  I do occasionally use filename completion
so I'll probably bind it to a key as well.

-Brett

Re: Git completion slowness

[Nikolai Weibull]

On Thu, Dec 9, 2010 at 23:06, Brett Simmers <swtaarrs@gmail.com> wrote:
> Whenever I try to use tab completion with a git command, zsh hangs for a few
> minutes using full cpu before coming back with an answer.

> Some details of my repository:
> - 96,000 files
> - .git is 17GB
> - Deepest directory nesting in the repository is 15 levels but most files
> are no deeper than 7 levels

Now there’s your problem.

Re: Git completion slowness

[Nikolai Weibull]

On Fri, Dec 10, 2010 at 12:07, Frank Terbeck <ft@bewatermyfriend.org> wrote:

> We've heard people complaining about _git performance on IRC for a while
> now. I have started to rewrite the _git file related completions some
> time ago, but have since been swamped at work. So that has slowed down
> to a halt.
>
> I'll attach the code to this mail though; maybe it'll be useful to
> someone (search for "Git file listing helpers" to find the helper
> functions). It doesn't have a helper for pathspecs yet, which the
> git-checkout completion (and others) may need.
>
> Also, it's something I experimented with for only a short while. The
> completions did work for me in my limited set of use-cases - they are
> probably not perfect, yet. But they scaled well even on huge
> repositories with many *many* files (like the linux kernel's git
> repository).

> Cc:ing Nikolai, too.

I’ve been rewriting _git for a while now.  I’ve updated all the Main
Porcelain Commands.  I’m currently stuck on something as unimportant
as git-config, but I want to get everything right and I’m doing it all
in man page order.  I’ll hopefully have a delayed Christmas gift for
you all at the beginning of 2011.  I’ll take a look at your changes
and integrate them if they are still applicable.

My main problem right now is that I’m working off of the Debian Zsh
Git repository where I don’t have write permissions.  I’d like to set
up a Zsh Git repository at GitHub so that I can more easily distribute
my changes, but I haven’t had time to look into setting this up.
Perhaps someone can explain how you set up a CVS-to-Git mirror on
GitHub?

I’ll also try to set up zstyles to the effect of allowing you to
disable potentially costly completions.

Re: Git completion slowness

[Frank Terbeck]

Nikolai Weibull wrote:
> On Fri, Dec 10, 2010 at 12:07, Frank Terbeck <ft@bewatermyfriend.org> wrote:
>> We've heard people complaining about _git performance on IRC for a while
>> now. I have started to rewrite the _git file related completions some
>> time ago, but have since been swamped at work. So that has slowed down
>> to a halt.
>>
>> I'll attach the code to this mail though; maybe it'll be useful to
>> someone (search for "Git file listing helpers" to find the helper
>> functions). It doesn't have a helper for pathspecs yet, which the
>> git-checkout completion (and others) may need.
>>
>> Also, it's something I experimented with for only a short while. The
>> completions did work for me in my limited set of use-cases - they are
>> probably not perfect, yet. But they scaled well even on huge
>> repositories with many *many* files (like the linux kernel's git
>> repository).

(The ML software ate my mail, due to my attaching the code...)

> I’ve been rewriting _git for a while now.  I’ve updated all the Main
> Porcelain Commands.  I’m currently stuck on something as unimportant
> as git-config, but I want to get everything right and I’m doing it all
> in man page order.  I’ll hopefully have a delayed Christmas gift for
> you all at the beginning of 2011.  I’ll take a look at your changes
> and integrate them if they are still applicable.

Yay! :)

> My main problem right now is that I’m working off of the Debian Zsh
> Git repository where I don’t have write permissions.  I’d like to set
> up a Zsh Git repository at GitHub so that I can more easily distribute
> my changes, but I haven’t had time to look into setting this up.
> Perhaps someone can explain how you set up a CVS-to-Git mirror on
> GitHub?

There's already a regularly updated git mirror of the zsh CVS sources at
sourceforge:

    git://zsh.git.sf.net/gitroot/zsh/zsh

When I was working on a larger update of `vcs_info', I cloned that one
and published the result at github. That worked rather smoothly.

> I’ll also try to set up zstyles to the effect of allowing you to
> disable potentially costly completions.

That was something I wanted to do, too. We'll need those properly
documented to be able to point people to how to workaround problems with
costly actions.

Regards, Frank

-- 
In protocol design, perfection has been reached not when there is
nothing left to add, but when there is nothing left to take away.
                                                  -- RFC 1925

Re: Git completion slowness

[Thomas Gstädtner]

On Fri, Dec 10, 2010 at 12:37, Nikolai Weibull <now@bitwi.se> wrote:
> On Fri, Dec 10, 2010 at 12:07, Frank Terbeck <ft@bewatermyfriend.org> wrote:
>
>> We've heard people complaining about _git performance on IRC for a while
>> now. I have started to rewrite the _git file related completions some
>> time ago, but have since been swamped at work. So that has slowed down
>> to a halt.
>>
>> I'll attach the code to this mail though; maybe it'll be useful to
>> someone (search for "Git file listing helpers" to find the helper
>> functions). It doesn't have a helper for pathspecs yet, which the
>> git-checkout completion (and others) may need.
>>
>> Also, it's something I experimented with for only a short while. The
>> completions did work for me in my limited set of use-cases - they are
>> probably not perfect, yet. But they scaled well even on huge
>> repositories with many *many* files (like the linux kernel's git
>> repository).
>
>> Cc:ing Nikolai, too.
>
> I’ve been rewriting _git for a while now.  I’ve updated all the Main
> Porcelain Commands.  I’m currently stuck on something as unimportant
> as git-config, but I want to get everything right and I’m doing it all
> in man page order.  I’ll hopefully have a delayed Christmas gift for
> you all at the beginning of 2011.  I’ll take a look at your changes
> and integrate them if they are still applicable.
>
> My main problem right now is that I’m working off of the Debian Zsh
> Git repository where I don’t have write permissions.  I’d like to set
> up a Zsh Git repository at GitHub so that I can more easily distribute
> my changes, but I haven’t had time to look into setting this up.
> Perhaps someone can explain how you set up a CVS-to-Git mirror on
> GitHub?
>

That's easy.
1) create a project at github or any other git hoster and a plain git
repository in this project.
2) create a local git repo with 'git init', use git-cvsimport to
import the cvs repo into this git repo.
3) add the git hoster (i.e. github) as a remote in your git repo via
'git remote add' (or just copy'n'paste this from the hosters
description).
4) push the imported cvs repo, which is now your git repo, to the remote.

You can use git-cvsexportcommit to get it back into CVS.
You can also use git-cvsimport to get your git-repo up-to-date,
because I don't think there is a way that github itself can keep track
of an external CVS repo.

Hope this helps -- looking forward to the shiny new git-completion!

Re: Git completion slowness

[Nikolai Weibull]

On Fri, Dec 10, 2010 at 12:49, Frank Terbeck <ft@bewatermyfriend.org> wrote:

> There's already a regularly updated git mirror of the zsh CVS sources at
> sourceforge:
>
>    git://zsh.git.sf.net/gitroot/zsh/zsh
>
> When I was working on a larger update of `vcs_info', I cloned that one
> and published the result at github. That worked rather smoothly.

Yeah, why haven’t I simply set up a remote?  Jeesh.  I am an idiot.
I’ll make my branch available for public consumption sometime next
week so that people can begin testing.

Re: Git completion slowness

[Nikolai Weibull]

On Fri, Dec 10, 2010 at 12:54, Thomas Gstädtner <thomas@gstaedtner.net> wrote:

> On Fri, Dec 10, 2010 at 12:37, Nikolai Weibull <now@bitwi.se> wrote:

>> Perhaps someone can explain how you set up a CVS-to-Git mirror on
>> GitHub?

> That's easy.
> 1) create a project at github or any other git hoster and a plain git
> repository in this project.
> 2) create a local git repo with 'git init', use git-cvsimport to
> import the cvs repo into this git repo.
> 3) add the git hoster (i.e. github) as a remote in your git repo via
> 'git remote add' (or just copy'n'paste this from the hosters
> description).
> 4) push the imported cvs repo, which is now your git repo, to the remote.

Thanks.

> Hope this helps -- looking forward to the shiny new git-completion!

Well, I wouldn’t say that it’s shiny nor new, but it’s more correct
and more flexibly and hopefully faster where it can be.

Re: Git completion slowness

[Oliver Kiddle]

"Benjamin R. Haskell" wrote:
> That prevents all git file completion, which is the cause of the 
> slowness (specifically, _multi_parts completion for filenames).  Recall 

It's never a great idea when some function tries to do it's own filename
completion. While perhaps not blazingly fast, I very much think that the
approach I took in _subversion is the best. That approach is to use
_files for the actual completion and use the e glob qualifier to filter
the files. That way, the many configuration styles and everything for
filename completion works as normal. I notice that _git is completing
dot files for me without an initial dot for instance.

So for _git-add, we'd need something like:
__git_filestatus() {
  [[ $(git status --porcelain $REPLY) = ?M* ]]
}

__git_modified_files () {
  _files -g "*(.e:__git_filestatus:)"
}

That may need to run git status for every file but that's way better
than recursing through every subdirectory. Note that it is possible to
add non-existant files - look at _subversion for an example.

This also loses you the division of files into ignored-modified-files,
ignored-other-files etc but you can actually put that back in with a
suitable file-patterns style provided __git_filestatus doesn't hard code
the regex/pattern (and options needed for e.g. ignored files).

I would work on this but if Nikolai has a rewritten _git, I'll hold off.

By the way, what's with all the double underscores in _git. Is there
some meaning to those or has someone just been doing too much Python.

Oliver

Re: Git completion slowness

[Nikolai Weibull]

On Fri, Dec 10, 2010 at 13:48, Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
> "Benjamin R. Haskell" wrote:

> So for _git-add, we'd need something like:
> __git_filestatus() {
>  [[ $(git status --porcelain $REPLY) = ?M* ]]
> }
>
> __git_modified_files () {
>  _files -g "*(.e:__git_filestatus:)"
> }
>
> That may need to run git status for every file but that's way better
> than recursing through every subdirectory. Note that it is possible to
> add non-existant files - look at _subversion for an example.

Hm, I thought we had some functions like that already.  Perhaps that
was in an earlier version.  I will look into using this.  Let’s hope
libgit2 finally arrives and someone (OK, I) implement zgit so that we
don’t have to do so much forking (think Cygwin).

> This also loses you the division of files into ignored-modified-files,
> ignored-other-files etc but you can actually put that back in with a
> suitable file-patterns style provided __git_filestatus doesn't hard code
> the regex/pattern (and options needed for e.g. ignored files).

OK, good.  We can have a look at that subdivision later (because I
think it’s desirable, especially for _git-add).

> I would work on this but if Nikolai has a rewritten _git, I'll hold off.

Thanks.  I’m sorry that it’s taking so long, but I want to clean it
all up so that future revisions will be easier to make.

> By the way, what's with all the double underscores in _git. Is there
> some meaning to those or has someone just been doing too much Python.

Yes, although it’s slightly redundant.  _git-* is completion for git
*.  __git_* is an internal helper function.

> Oliver

__self__