excessive memory usage?

excessive memory usage?

[Clint Adams]

I received this bug report last week.  Due to lack of time,
I haven't had a chance to look into it thoroughly, but I
just tried it out, and it did suck up (rather slowly) about
42 megs (temporarily) before it finally completed.

I'm guessing that this happens during compadd.  So is this
evidence of some inefficiency or is all that memory necessary?

As for the second issue, I grabbed a vera.index that is probably
more recent (it has 9109 lines), and zsh happily completes from all
of the 6380 unique values.  I assume that his 7617->5996 effect
is also from duplicate values.

----- Forwarded message -----

Hi,

I have ~/.zshfunc directory in $fpath, and I made _dict file and put it
there. That file contained:

#compdef dict
_arguments '*:dictword:_dictwords'

The file _dictwords in the same directory contained:

#autoload

local expl dictwords

dictwords=($(awk '{print $1}' /usr/share/dictd/*.index))

_wanted dictwords expl dictword \
  compadd -M 'm:{a-zA-Z}={A-Za-z} r:|=*' "$@" - "$dictwords[@]"

The intent was to get 'dict <TAB>' to complete words from dictd's
dictionaries.

I also tried this to define $dictwords:

(( $+_cache_dictwords )) || \
  : ${(A)_cache_dictwords:=${${(f)"$(</usr/share/dictd/wn.index)"}%%[   ]*}}

dictwords=("$_cache_dictwords[@]")

(wn.index is the largest file from there)

The result was the same for both cases, pressing 'dict <TAB>' produced an
unusable shell (^C or ^\ didn't exit), which sucked up loads of memory, 42MB
at one time (and then I killed it).

% awk '{print $1}' /usr/share/dictd/*.index | wc -lc
 143539 1203636

1.2MB of data shouldn't take that much memory to get parsed. :)
Note that it worked when I used just one smaller index file:

% awk '{print $1}' /usr/share/dictd/vera.index | wc -lc
   7617   35197

But, after typing 'dict <TAB>', zsh said:

zsh: do you wish to see all 5996 possibilities?

The array was limited to ~6000 entries?

Please fix this, TIA.

----- End forwarded message -----

Re: excessive memory usage?

[Peter Stephenson]

> I received this bug report last week.  Due to lack of time,
> I haven't had a chance to look into it thoroughly, but I
> just tried it out, and it did suck up (rather slowly) about
> 42 megs (temporarily) before it finally completed.
...
> ----- Forwarded message -----
...
> _wanted dictwords expl dictword \
>   compadd -M 'm:{a-zA-Z}={A-Za-z} r:|=*' "$@" - "$dictwords[@]"

Change the second line to:

  compadd -M 'm:{a-zA-Z}={A-Za-z} r:|=*' -a "$@" - dictwords     

which is a good deal faster --- haven't checked the memory, but that ought
to be more manageable to.  This is why the -a option was added.

-- 
Peter Stephenson <pws@cambridgesiliconradio.com>
Cambridge Silicon Radio, Unit 300, Science Park, Milton Road,
Cambridge, CB4 0XL, UK                          Tel: +44 (0)1223 392070

Re: excessive memory usage?

[Clint Adams]

> which is a good deal faster --- haven't checked the memory, but that ought
> to be more manageable to.  This is why the -a option was added.

Indeed, this seems to save about 15 megs from being allocated.

RE: excessive memory usage?

[Andrej Borsenkow]

> >
> > #compdef dict
> > _arguments '*:dictword:_dictwords'
>
> If there isn't anything else in this file, then we don't need to use
> _arguments. Because calling _arguments with only one `*:...' spec and
> no options, no other arguments specs is the same as doing the action
> from the `*:...' spec. Only slower.
>

But _arguments will setup proper context names, thus enabling use of
per-argument styles. This alone warrants usage of _arguments IMHO.

-andrej

Re: excessive memory usage?

[Clint Adams]

> If there isn't anything else in this file, then we don't need to use
> _arguments. Because calling _arguments with only one `*:...' spec and
> no options, no other arguments specs is the same as doing the action
> from the `*:...' spec. Only slower.

dict takes a number of options, so some of them will probably find their
way into the _arguments call.

> And if this is ever intended to be included in the distribution, the
> -M should be removed, there are enough styles around to give users the 
> possibility to define them if they want them. In a simple personal
> completion function, of course...

dict doesn't require any data to exist on the client machine; I content
myself to use dict.org, rather than running a server.  For the
distribution, I'd rather see dict used itself to get the available
words, say, by parsing `dict -m -s re '.*'` or perhaps something
more efficient.  Of course, I just tried this, and was greeted with
dict (xmalloc): Out of memory while allocating 4072 bytes

Figures.

Re: excessive memory usage?

[Sven Wischnowsky]

I had seen this message on the debian list, but didn't have the time
to reply yet.


Clint Adams wrote:

> I received this bug report last week.  Due to lack of time,
> I haven't had a chance to look into it thoroughly, but I
> just tried it out, and it did suck up (rather slowly) about
> 42 megs (temporarily) before it finally completed.
> 
> I'm guessing that this happens during compadd.  So is this
> evidence of some inefficiency or is all that memory necessary?

As Peter already pointed out the problem is really that the function
passes down huge arrays and has to copy them several times and the
solution is to use the -a option to compadd. Some more comments below.

> As for the second issue, I grabbed a vera.index that is probably
> more recent (it has 9109 lines), and zsh happily completes from all
> of the 6380 unique values.  I assume that his 7617->5996 effect
> is also from duplicate values.

This is what I was thinking, too.

> ...
> 
> I have ~/.zshfunc directory in $fpath, and I made _dict file and put it
> there. That file contained:
> 
> #compdef dict
> _arguments '*:dictword:_dictwords'

If there isn't anything else in this file, then we don't need to use
_arguments. Because calling _arguments with only one `*:...' spec and
no options, no other arguments specs is the same as doing the action
from the `*:...' spec. Only slower.

> ...
> 
> I also tried this to define $dictwords:
> 
> (( $+_cache_dictwords )) || \
>   : ${(A)_cache_dictwords:=${${(f)"$(</usr/share/dictd/wn.index)"}%%[   ]*}}
> 
> dictwords=("$_cache_dictwords[@]")

For things that change as seldom as dictionaries, caching is
definetely a good idea. But no need to copy the cache-array into yet
another local array -- that would cause some more unnecessary
allocation and copying. Just use `compadd ... -a _cache_dictwords'.


And if this is ever intended to be included in the distribution, the
-M should be removed, there are enough styles around to give users the 
possibility to define them if they want them. In a simple personal
completion function, of course...


Bye
 Sven


--
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de