Purified zsh-3.0.1

Purified zsh-3.0.1

[Hrvoje Niksic]

I have run a purified version of zsh-3.0.1 on solaris2.5, and here is
what I got:

1)
ABR: Array bounds read

This is in refreshline [zle_refresh.c:583]:
               /* 3: main display loop - write out the buffer using whatever tricks we can */
               
                   for (;;) {
            =>         if (*nl && nl[1] == ol[1])      /* skip only if second chars match */
                       /* skip past all matching characters */
                           for (; *nl && (*nl == *ol); nl++, ol++, ccs++) ;

The call chain was:
            refresh        [zle_refresh.c:398]
            zleread        [zle_main.c:487]
            inputline      [input.c:264]
            ingetc         [input.c:220]
            hgetc          [hist.c:109]

It is obviously an off-by-one error (as Purify would have it, "Reading
1 byte from 0x157019 in the text section.".
I got that when I tried various expansions, e.g. ls **/*<TAB>, or
pressing C-u, and similar.

Other than that, I encountered no ABR-s (or ABW-s), except the one
with max_fds & co., but that has been fixed.

However, there were many memory leaks.  All of these I got before ever
reaching the command-line -- only while executing startup scripts.

2) MLK
         MLK: 344 bytes leaked in 2 blocks
         This memory was allocated from:
               malloc         [rtlib.o]
               zalloc         [mem.c:315]
               metafy         [utils.c:2918]
               createparamtable [params.c:115]
               setupvals      [init.c:600]
               main           [init.c:66]
         Block of 256 bytes at 0x192b18
         Block of 88 bytes at 0x191df8

It seems that memory allocated in line 115 of params.c:
                 if ((!(pm = (Param) paramtab->getnode(paramtab, iname)) ||
                      !(pm->flags & PM_DONTIMPORT)) &&

never gets freed.

3) MLK
Another leak of 344 bytes.  Backtrace follows:
         MLK: 344 bytes leaked in 2 blocks
         This memory was allocated from:
               malloc         [rtlib.o]
               zalloc         [mem.c:315]
               zjoin          [utils.c:1442]
               arrfixenv      [params.c:1778]
               colonarrsetfn  [params.c:1330]
               setstrvalue    [params.c:783]
         Block of 256 bytes at 0x1932e8
         Block of 88 bytes at 0x192100

Here it seems that the result of zjoin() was never freed, or the
retval of arrfixenv.

4) MLK
A similar leak, again.
         MLK: 344 bytes leaked in 2 blocks
         This memory was allocated from:
               malloc         [rtlib.o]
               zalloc         [mem.c:315]
               zjoin          [utils.c:1442]
               colonarrgetfn  [params.c:1318]
               getstrvalue    [params.c:699]
               getsparam      [params.c:956]
         Block of 256 bytes at 0x193590
         Block of 88 bytes at 0x192258

5) MLK
This one is in execcmd, which is not well:
         MLK: 33 bytes leaked in 2 blocks
         This memory was allocated from:
               malloc         [rtlib.o]
               zalloc         [mem.c:315]
               ztrdup         [mem.c:387]
               addvars        [exec.c:1099]
               execcmd        [exec.c:1253]
               execpline2     [exec.c:801]
         Block of 30 bytes at 0x18d720
         Block of 3 bytes at 0x1a2338

In addvars() pm = setsparam(v->name, ztrdup(val)); never gets freed, I
suppose (if errflag is set?).


I hope this will help debugging...  Those ABR-s in line editor could
be the cause of coredumps seen on Ultrix, that are quite hard to
repeat.

-- 
Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
I'm a Lisp variable -- bind me!

Re: Purified zsh-3.0.1

[Geoff Wing]

Hrvoje Niksic wrote:
:I have run a purified version of zsh-3.0.1 on solaris2.5, and here is
:what I got:
:ABR: Array bounds read
:This is in refreshline [zle_refresh.c:583]:
:               /* 3: main display loop - write out the buffer using whatever tricks we can */
:                   for (;;) {
:            =>         if (*nl && nl[1] == ol[1])      /* skip only if second chars match */
:I hope this will help debugging...  Those ABR-s in line editor could
:be the cause of coredumps seen on Ultrix, that are quite hard to
:repeat.

This is a patch (on top of 2351 and its predecessor though it'll patch without
them)
I believe it will fix said problem.  It doesn't seem likely to me that it
would be the cause of any coredumps.

*** zle_refresh.c.~2~	Sat Nov  9 21:08:12 1996
--- zle_refresh.c	Sun Nov 10 16:01:03 1996
***************
*** 615,621 ****
  /* 3: main display loop - write out the buffer using whatever tricks we can */
  
      for (;;) {
! 	if (*nl && nl[1] == ol[1])	/* skip only if second chars match */
  	/* skip past all matching characters */
  	    for (; *nl && (*nl == *ol); nl++, ol++, ccs++) ;
  
--- 615,621 ----
  /* 3: main display loop - write out the buffer using whatever tricks we can */
  
      for (;;) {
! 	if (*nl && *ol && nl[1] == ol[1]) /* skip only if second chars match */
  	/* skip past all matching characters */
  	    for (; *nl && (*nl == *ol); nl++, ol++, ccs++) ;
  

-- 
Geoff Wing [gwing@primenet.com.au]   Technical Manager
  Phone    : +61-3-9818 2977	     PrimeNet - Internet Consultancy
  Facsimile: +61-3-9819 3788	     Web : <URL:http://www.primenet.com.au/>
  Mobile   : 0412 162 441

Re: Purified zsh-3.0.1

[Hrvoje Niksic]

Geoff Wing (gwing@primenet.com.au) wrote:
> :I hope this will help debugging...  Those ABR-s in line editor could
> :be the cause of coredumps seen on Ultrix, that are quite hard to
> :repeat.
> This is a patch (on top of 2351 and its predecessor though it'll
> patch without them)

I applied the patch manually.

> I believe it will fix said problem.

Yup, that fixes it (at least I cannot repeat it anymore).  Anyway, zsh
is very stable, in the sense of purify.  Now there are no more
array-bound reads or writes.

> It doesn't seem likely to me that it would be the cause of any
> coredumps.

But it is possible, as soon as you read memory outside of the
allocated block, isn't it?  The fact that the read was off-by-one
could account for those dumps being hard to repeat.

-- 
Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
main(){printf(&unix["\021%six\012\0"],(unix)["have"]+"fun"-0x60);}

Re: Purified zsh-3.0.1

[Geoff Wing]

Hrvoje Niksic wrote:
:Geoff Wing (gwing@primenet.com.au) wrote:
:> :I hope this will help debugging...  Those ABR-s in line editor could
:> :be the cause of coredumps seen on Ultrix, that are quite hard to
:> :repeat.
:> This is a patch (on top of 2351 and its predecessor though it'll
:Yup, that fixes it (at least I cannot repeat it anymore).  Anyway, zsh
:is very stable, in the sense of purify.  Now there are no more
:array-bound reads or writes.
:> It doesn't seem likely to me that it would be the cause of any
:> coredumps.
:But it is possible, as soon as you read memory outside of the
:allocated block, isn't it?  The fact that the read was off-by-one
:could account for those dumps being hard to repeat.

Yes, though very unlikely. I don't know how your malloc stuff aligns things,
but the buffers involved are normally 81 bytes (for an 80 column screen)
and if accessing the 82nd byte places it outside a page boundary, then
that would be strange.

I think the most likely way for you to get it to reoccur, is by having 
delete/insert happen on a line, so scroll between lines which have
lots of similarities.
I play around a bit with a segment of my news file  `active' - 20k to 80k
worth.  Then do something like
% echo $(< active)<TAB>
and CTRL-A / CTRL-E combinations and other cursor movements.

Don't use the full `active' file: 800-900k, expand it on line, then press
return.  I had it going for over twenty minutes, memory use up to 27 meg,
before I got sick of it and killed it EXTERNALLY (no internal interrupt).
Not that many people are going to be doing this sort of thing, but...
The spelling correction stuff could use a bit of improvement -
I assume it rereads the directory for every single word on line.
-- 
Geoff Wing [gwing@primenet.com.au]   Technical Manager
  Phone    : +61-3-9818 2977	     PrimeNet - Internet Consultancy
  Facsimile: +61-3-9819 3788	     Web : <URL:http://www.primenet.com.au/>
  Mobile   : 0412 162 441

Re: Purified zsh-3.0.1

[Zoltan Hidvegi]

> 2) MLK
>          MLK: 344 bytes leaked in 2 blocks
>          This memory was allocated from:
>                malloc         [rtlib.o]
>                zalloc         [mem.c:315]
>                metafy         [utils.c:2918]
>                createparamtable [params.c:115]
>                setupvals      [init.c:600]
>                main           [init.c:66]
>          Block of 256 bytes at 0x192b18
>          Block of 88 bytes at 0x191df8

That seems to be the same as the last one.  I hope that the patch below
fixes that but I'm not at all sure.  If not please try to figure out which
variable causes that.  That happens when zsh imports the environment.
Start with an empty environment using

(typeset +x -m '*' ; zsh)

and add environment variables.  Hopefully you'll be able to identify the
problematic variable.

> 3) MLK
> Another leak of 344 bytes.  Backtrace follows:
>          MLK: 344 bytes leaked in 2 blocks
>          This memory was allocated from:
>                malloc         [rtlib.o]
>                zalloc         [mem.c:315]
>                zjoin          [utils.c:1442]
>                arrfixenv      [params.c:1778]
>                colonarrsetfn  [params.c:1330]
>                setstrvalue    [params.c:783]
>          Block of 256 bytes at 0x1932e8
>          Block of 88 bytes at 0x192100

I've fount that one that's also fixed by the patch below.

> 4) MLK
> A similar leak, again.
>          MLK: 344 bytes leaked in 2 blocks
>          This memory was allocated from:
>                malloc         [rtlib.o]
>                zalloc         [mem.c:315]
>                zjoin          [utils.c:1442]
>                colonarrgetfn  [params.c:1318]
>                getstrvalue    [params.c:699]
>                getsparam      [params.c:956]
>          Block of 256 bytes at 0x193590
>          Block of 88 bytes at 0x192258

That's bad.  It seems that there is a leak each time getsparam() is called
to a colon array like PATH.  It may be difficult to fix so I'll not even
try it now (2 am).

Zoltan

*** Src/params.c	1996/10/30 23:41:58	3.1.0.0
--- Src/params.c	1996/11/11 00:40:11
***************
*** 1000,1008 ****
  	    v = NULL;
  	}
      }
!     if (!v)
! 	if (!(v = getvalue(&t, 1)))
  	    return NULL;
      setstrvalue(v, val);
      return v->pm;
  }
--- 1000,1011 ----
  	    v = NULL;
  	}
      }
!     if (!v) {
! 	if (!(v = getvalue(&t, 1))) {
! 	    zsfree(val);
  	    return NULL;
+ 	}
+     }
      setstrvalue(v, val);
      return v->pm;
  }
***************
*** 1787,1792 ****
--- 1790,1797 ----
  	pm->flags |= PM_EXPORTED;
      if (pm->flags & PM_EXPORTED)
  	pm->env = addenv(s, u);
+     else
+ 	zsfree(u);
  }
  
  /* Given *name = "foo", it searchs the environment for string *

Re: Purified zsh-3.0.1

[Hrvoje Niksic]

Zoltan Hidvegi (hzoli@cs.elte.hu) wrote:
> ***************
> *** 1787,1792 ****
> --- 1790,1797 ----
>   	pm->flags |= PM_EXPORTED;
>       if (pm->flags & PM_EXPORTED)
>   	pm->env = addenv(s, u);
> +     else
> + 	zsfree(u);
>   }

This one is wrong, I'm afraid.  Here I get FUM (Freeing Uninitialized
Memory), which is *bad*.

If I use only the first part of the patch, there are no FUMs, but the
leaks seem to persist. :(

BTW, after quitting zsh, it seems that a descriptor is in use, which
shouldn't be:

         FIU: file descriptor 10: "/home/srce/hniksic/.zcompctl", O_RDONLY
         Duplication of file descriptor 5.
         File info: crwx------  1 hniksic  tty              0 Nov 11 02:25
         File position: 1787837
         This file descriptor was allocated from:
               fcntl          [libc.so.1]
               movefd         [utils.c:877]
               source         [init.c:781]
               bin_dot        [builtin.c:4726]
               execbuiltin    [builtin.c:185]
               execcmd        [exec.c:1653]

Why does .zcompctl stay open (.zcompctl is the name of a file in my
home directory that I source from .zshrc)?

-- 
Hrvoje Niksic <hniksic@srce.hr> | Student at FER Zagreb, Croatia
--------------------------------+--------------------------------
I'm a Lisp variable -- bind me!