Improvement to python module list generation used for command completion

Improvement to python module list generation used for command completion

[Antony Lee]

[-- Attachment #1: Type: text/plain, Size: 762 bytes --]

Currently, both the completion for python -m and for pydoc require a list
of available python modules, generated in _python_modules.  This uses
pydoc's ModuleScanner, which is unbearably slow on systems with just a few
big packages installed because it also searches for submodules, and thus
has to import all packages(!) available on the path.

Instead, pkgutil.iter_modules (available at least since python 2.6) only
returns toplevel modules, so replacing

    local script='import sys, pydoc
def f(p,m,d):
    if m.find(".") < 0: sys.stdout.write(m+"\n")
pydoc.ModuleScanner().run(f)'

by

    local script='import pkgutil
for importer, name, ispkg in pkgutil.iter_modules(): print(name)'

in _python_modules yields a big improvement in speed.

Best,

Antony

Re: Improvement to python module list generation used for command completion

[Jun T.]

2015/07/25 12:37, Antony Lee <antony.lee@berkeley.edu> wrote:
> so replacing
> 
>   local script='import sys, pydoc
> def f(p,m,d):
>   if m.find(".") < 0: sys.stdout.write(m+"\n")
> pydoc.ModuleScanner().run(f)'
> 
> by
> 
>   local script='import pkgutil
> for importer, name, ispkg in pkgutil.iter_modules(): print(name)'
> 
> in _python_modules yields a big improvement in speed.

Thanks. It is indeed *much* faster.

Do you think using 'print(name)' is sufficiently portable?
It seems it works with Python 2.x, because it is interpreted
as a print statement with a single-element tuple '(name)'.

Re: Improvement to python module list generation used for command completion

[Antony Lee]

[-- Attachment #1: Type: text/plain, Size: 881 bytes --]

Actually on Python2 "(name)" is not interpreted as a single element tuple
but as a parenthesized string, which is the same as the string itself, so
it works the same.
Antony

2015-07-25 11:18 GMT-07:00 Jun T. <takimoto-j@kba.biglobe.ne.jp>:

>
> 2015/07/25 12:37, Antony Lee <antony.lee@berkeley.edu> wrote:
> > so replacing
> >
> >   local script='import sys, pydoc
> > def f(p,m,d):
> >   if m.find(".") < 0: sys.stdout.write(m+"\n")
> > pydoc.ModuleScanner().run(f)'
> >
> > by
> >
> >   local script='import pkgutil
> > for importer, name, ispkg in pkgutil.iter_modules(): print(name)'
> >
> > in _python_modules yields a big improvement in speed.
>
> Thanks. It is indeed *much* faster.
>
> Do you think using 'print(name)' is sufficiently portable?
> It seems it works with Python 2.x, because it is interpreted
> as a print statement with a single-element tuple '(name)'.
>

Re: Improvement to python module list generation used for command completion

[Jun T.]

2015/07/26 06:33, Antony Lee <antony.lee@berkeley.edu> wrote:
> Actually on Python2 "(name)" is ... a parenthesized string

Oops, yes, thanks again. I've pushed your patch.

Jun