Re: maximum value for HISTSIZE ?

Re: maximum value for HISTSIZE ?

[Barton Schaefer]

On Mon, 10 Apr 2006 7:14am, Andy Spiegl wrote:
>
> Do you mind forwarding it to the developers ML?

All zsh-users mail is automatically forwarded to the developers ML.  We 
believe our developers should know what our users are complaining about.

Re: maximum value for HISTSIZE ?

[Wayne Davison]

On Sat, Apr 01, 2006 at 10:54:09PM -0800, Bart Schaefer wrote:
> Perhaps Wayne remembers, or Zefram if he's still reading this list ...

Sorry, I don't recall doing anything with the array size limit.  Having
it be 0x40000 seems unusually small, though, so your guess that it might
have to do with the total memory size for the array (and perhaps some
old limitations of malloc()?) seems like a reasonable supposition.

..wayne..

Re: maximum value for HISTSIZE ?

[Andy Spiegl]

Hi Bart,

> Perhaps Wayne remembers, or Zefram if he's still reading this list ...
Apparently not?

Do you mind forwarding it to the developers ML?

Thanks,
 Andy.

-- 
 Show me a piano falling down a mine shaft and I'll show you A-flat minor.

Re: maximum value for HISTSIZE ?

[Bart Schaefer]

On Apr 2,  2:33am, Andy Spiegl wrote:
} Subject: Re: maximum value for HISTSIZE ?
}
} > There isn't a fixed maximum.  Zsh will try to use all the memory your
} > system will allocate to it to maintain as large a history as you ask.
} > When the memory runs out depends on your system.
} 
} Hm, okay. But an array with 318937 elements should easily be possible
} with 1GB RAM.

You're right.  Although the history can be arbitrarily big, an array can
only have 262144 elements.  I have no idea why that particular number --
it appeared sometime between zsh-2.4 and zsh-3.0, and prior to that an
array was IIRC effectively unbounded.  I don't have records of changes
between 1992 and 1995 ... not only was I busy starting my own business
then, but I was having a disagreement with some of the other developers
and had stopped using new zsh builds.

It's 1/4 of a megabyte, so an array of 262144 32-bit pointers would use
exactly a meg of memory.  Maybe that was/is related to the size of a heap
allocation block.

Perhaps Wayne remembers, or Zefram if he's still reading this list ...

In any case, it might nowadays be possible to increase the constant
MAX_ARRAY in zsh.h.

Re: maximum value for HISTSIZE ?

[Andy Spiegl]

> There isn't a fixed maximum.  Zsh will try to use all the memory your
> system will allocate to it to maintain as large a history as you ask.
> When the memory runs out depends on your system.

Hm, okay.  But an array with 318937 elements should easily be possible with
1GB RAM.  Unless the elements are really large themselves.  But they
shouldn't, as they only contain the commands in the history, right?

Good night,
 Andy.

-- 
 Do Lipton employees take coffee breaks?

Re: maximum value for HISTSIZE ?

[Bart Schaefer]

On Apr 1, 11:32am, Andy Spiegl wrote:
} Subject: Re: maximum value for HISTSIZE ?
}
} > So is there a maximum value or/and on what does it depend?
} 
} Really no one knows?

There isn't a fixed maximum.  Zsh will try to use all the memory your
system will allocate to it to maintain as large a history as you ask.
When the memory runs out depends on your system.

I don't know exactly what's happening in _all_labels, but it's most
likely trying to create an array indexed on history number.  When zsh
creates an array and is asked to insert an element at index N, all of
the N-1 elements at lesser index positions are also allocated, even if
there's nothing to put in them.

Re: maximum value for HISTSIZE ?

[Andy Spiegl]

Hi Carlos,

good to see that I'm not alone. :-)

> As a workaround I do the following:
> ...
Yep, same here.

Yesterday I wrote a simple perl script which will run weekly
and cleans up my history file.

Basically it does the following to delete all simple lines:
 (watch out, it contains lots of personal aliases)

# `:<beginning time>:<elapsed seconds>;<command>'.
# Warning: <command> can contain multiple lines (with \ at line end)
my ($line, $cmd, $continued, $lastLinePrinted);
$lastLinePrinted=0;
while ($line = <STDIN>)
{
  if ($continued and $lastLinePrinted)
  {
    print $line;
    $continued=($line =~ /\\$/);   # still continued?
    next;
  }

  # new history event
  if (not $line =~ /^:[^:]+:[^;]+;(.*)$/)
  {
    print STDERR "skipping invalid zsh-history line in line $..\n";
    next;
  }
  $cmd = $1;
  $continued=1 if $cmd =~ /\\$/;

  # short commands/aliases without arguments
  next if $cmd =~ /^(bb|ls|l|ltr|lS|cd)*$/;

  # don't skip more complex commands
  if (not $cmd =~ /&|;|\|/)
  {
    # short often used commands without options
    next if $cmd =~ /^(l|ls|ll|lh|ltr|lltr|fff|xv|rm|rd|(d)?loc(ate)?|stat|g(un)?zip|b(un)?zip2|unzip|run-help|where|which|gimp|ping|(tk)?diff|cat|adr|kill|sr (google|ebay)|qiv-max|mutt|disown) +[^-]/;

    # short often used commands with some options
    next if $cmd =~ /^(rm -(r|rf)|l(s|l|h)? -(d|R|u)|rd -p|gzip -9|bzip2 -9|diff -r|du -(sh|h|s)) +[^-]/;

    # specials
    next if $cmd =~ /^cd (diario|~\/diario)/;
    next if $cmd =~ /^(heute|diario|stunden)/;

    # keep mplayer with options OR some arg with a colon
    next if $cmd =~ /^\\?mplayer[^ ]* +[^-]/ and not $cmd =~ /^\\?mplayer[^ ]* +.*:/;
  }

  # okay, line ist interesting enough -> keep it
  print $line;
  $lastLinePrinted=1;
}

Suggestions are very welcome!

Bye,
 Andy.

-- 
 You should aim to be independent of any one vote, of any one fashion,
 of any one century.  (Baltasar Gracián, Spanish writer)

Re: maximum value for HISTSIZE ?

[Carlos Carvalho]

Andy Spiegl (zsh.Andy@spiegl.de) wrote on 30 March 2006 14:26:
 >I am lazy and forget a lot, so I set HISTSIZE to 150000.
 >But now I frequently get to see:
 > _all_labels:39: subscript too big: 318937

I don't know the answer but I have a similar usage problem: keep only
the interesting lines in history. As a workaround I do the following:
- set the the following options
  histignorealldups
  histignorespace
  histnostore

- define several aliases of the form ls=' ls', free=' free', etc to
  avoid having them go to the hist file and running zsh list.

I also type most of the easy commands beginning with a space for the
same reason.

 >So is there a maximum value or/and on what does it depend?
 >
 >BTW, I'd love to see an option like HIST_EXPIRE_ShortLines_FIRST
 >so that really short history lines like "ls", "du -sh" etc. expire
 >before long and complicated ones.  Maybe even with a configurable
 >line length.

Me too! I recognize however that a developer might find this hmm...
not of particularly high priority...

Re: maximum value for HISTSIZE ?

[Andy Spiegl]

> So is there a maximum value or/and on what does it depend?

Really no one knows?

Bye,
 Andy.

-- 
 When I was a boy of 14, my father was so ignorant I could hardly stand to have
 the old man around. But when I got to be 21, I was astonished at how much the
 old man had learned in seven years. -- Mark Twain

Re: maximum value for HISTSIZE ?

[Andy Spiegl]

> That's got to be a record!
:-)
Besides the problem I described I didn't notice any slowdown during startup
or exit of zsh so there is no drawback.

> When I was working as a sysadmin, I had a cron job that appended all unique
> history entries into some other file; and would grep that file as needed.
Good idea.

> If I was to do it, I would use the number of arguments, instead of line
> length.
Not a bad idea either. :-)

Chau,
 Andy.

-- 
 Fotos: francisco.spiegl.de            o      _     _         _              
 Infos: peru.spiegl.de       __o      /\_   _ \\o  (_)\__/o  (_)         -o) 
 Andy, Heidi, Francisco    _`\<,_    _>(_) (_)/<_    \_| \   _|/' \/      /\\
 heidi.und.andy@spiegl.de (_)/ (_)  (_)        (_)   (_)    (_)'  _\o_   _\_v
 ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~ ~
 No one can pull your strings unless you hand them to him.

Re: maximum value for HISTSIZE ?

[Francisco Borges]

» On Thu, Mar 30, 2006 at 02:26PM +0200, Andy Spiegl wrote:

> I am lazy and forget a lot, so I set HISTSIZE to 150000.

150000!!!

8-)

That's got to be a record!

When I was working as a sysadmin, I had a cron job that appended all unique
history entries into some other file; and would grep that file as needed.

That would give me this level of command memory, without the history loading
& saving time problem. Though back then I used tcsh, which was noticeably
slower to load or exit than zsh (at least when I migrated).

> BTW, I'd love to see an option like HIST_EXPIRE_ShortLines_FIRST
> so that really short history lines like "ls", "du -sh" etc. expire
> before long and complicated ones.  Maybe even with a configurable
> line length.  Or can that already be done somehow?  Of course I

If I was to do it, I would use the number of arguments, instead of line
length.

-- 
Francisco Borges

maximum value for HISTSIZE ?

[Andy Spiegl]

I am lazy and forget a lot, so I set HISTSIZE to 150000.
But now I frequently get to see:
 _all_labels:39: subscript too big: 318937

So is there a maximum value or/and on what does it depend?

BTW, I'd love to see an option like HIST_EXPIRE_ShortLines_FIRST
so that really short history lines like "ls", "du -sh" etc. expire
before long and complicated ones.  Maybe even with a configurable
line length.  Or can that already be done somehow?  Of course I
could write a perl script that "cleans" my the history file nightly
but I'd prefer the zsh-way.

Thx,
 Andy.

-- 
 The world is moving so fast these days that the person who says
 it can't be done is generally interrupted by someone doing it.  (Fosdick)