From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 28359 invoked from network); 1 Mar 2000 10:06:40 -0000 Received: from sunsite.auc.dk (130.225.51.30) by ns1.primenet.com.au with SMTP; 1 Mar 2000 10:06:40 -0000 Received: (qmail 27757 invoked by alias); 1 Mar 2000 10:06:28 -0000 Mailing-List: contact zsh-workers-help@sunsite.auc.dk; run by ezmlm Precedence: bulk X-No-Archive: yes X-Seq: 9947 Received: (qmail 27744 invoked from network); 1 Mar 2000 10:06:26 -0000 Date: Wed, 1 Mar 2000 11:06:24 +0100 (MET) Message-Id: <200003011006.LAA24751@beta.informatik.hu-berlin.de> From: Sven Wischnowsky To: zsh-workers@sunsite.auc.dk Subject: PATCH: wordcode files [ Even if you don't want to read all this, you should look below at the *Important note*. ] Ok, here is the patch. The zcompile builtin can be used to create wordcode files. Options are -U (to avoid alias expansion), -r (to create a wordcode file that is read), -m (to create a wordcode file that is mapped) and -t (to list the functions in the file (`table') and to test if certain functions are in the file. If neither -r nor -m are given zcompile decides if the file is to be read or mapped: if the resulting file (one version of the content, see below) is less than 4096 bytes long it is read. As I said, I didn't want to try anything more complicated here until we get some feedback. The file created contains two versions: one for either endian-ness. The shell reading the file will decide which one it needs and if the file is mapped, only one half will be mapped (so you shouldn't worry too much about the dump file for all _* functions being 755544 bytes long -- at least for the function in the state I have them now). Wordcode files can be used in two ways. One way is to put them into the directories in $fpath with the .zwc extension. When the function- file-search process looks up the definition for function foo and it finds a file foo.zwc and that is a valid wordcode file and it is newer than the file foo in the same directory (or that doesn't exist), then foo.zwc will be used. The other way is to put the name of the wordcode file into $fpath. In that case the .zwc extension isn't required and the file should (but need not) contain more than one function. Such a digest file is then searched for the definition of the function and if it contains that, that definition is loaded. I.e. if the wordcode file says that it is to be read (not mapped), we only take the pre-compiled function definition from it. If the file says that it is to be mapped, we map the whole file. In other words: currently we don't try to map only parts of a digest file. I *really* prefer it this way. If users don't like it, they can easily create multiple dump files, sorted by importance. E.g. create one file for the functions they use very often and one for the function they use never or almost never and put the paths of both files int $fpath. Note also, that such digest files are really only searched when they appear as elements of $fpath. The shell does not look into every *.zwc file in every diretory in $fpath to see if it contains the function (that's why I prefer to do it this way -- the lookup would be much more expensive if digest files are stored in diretories in $fpath). Both types of wordcode files (the automatically found .zwc files and the digest files) have exactly the same format. It's just a matter of user-preference which form of file they use. The contents of the files is not completely verified, but the header they contain (which is a bit of version information, some flags and the directory listing the functions) is, partly implicit due to the way the headers are parsed. I.e., odds are that a file with a valid header is indeed a valid wordcode file. Btw, I'm not too sure if we should really make them dependent on the shell-version. Maybe we should change that to some kind of wordcode-file-format-version before release. Internal changes: mostly the stuff at the end of parse.c, namely the bin_zcompile() and friends. I also had to change the wordcode format for function definitions a bit, storing the strings in the string table of the whole Eprog, because otherwise generating the other- endian-version would be much more complicated. Other changes: the patterns used in comp{init,dump} now exclude *.zwc files. I have not written any of the functions I spoke about in 9930... yet. I also find it tempting to turn most of my .z* files (including the completion dump file) into functions and putting them into a wordcode file, but I haven't implemented any support for that either. Maybe we could extend the mechanism with the *.zwc files to also apply to sourced files -- creating dump files for them is simple, and can be done already with zcompile, it's just that the function for sourcing a file does not check if there is a *.zwc file. Should we? *Important note*: since mapped wordcode files are mapped read-only and most of the execution code is careful to duplicate strings only when needed, I had to modify some functions (like untokenize(), remnulargs() and zzlex()). There were writing into the strings without really modifying them. There may be other functions that do that which I may have missed. So, if you use a mapped wordcode file and get mysterious segmentation faults, you can try if uncommenting the code in the comment that starts with /*** in parse.c helps. This will make the strings used by the execution code be copied. However, even when that fixes it for you, I'd like to hear about the function where you got the SEGV so that we can fix it. To have fun, I'd suggest trying % zcompile foo ../Completion/*/_*~*~ % fpath=(foo) And then do completion (without having it done before in the same shell)... the first completions (where normally the functions are loaded and parsed) should be a lot faster, especially when using the larger functions like _cvs, _rpm or _pbm. Bye Sven diff -ru ../z.old/Completion/Core/compdump Completion/Core/compdump --- ../z.old/Completion/Core/compdump Tue Feb 29 13:11:33 2000 +++ Completion/Core/compdump Wed Mar 1 10:26:36 2000 @@ -14,13 +14,14 @@ # to see if auto-dump should re-dump the dump-file. emulate -L zsh +setopt extendedglob typeset _d_file _d_f _d_bks _d_line _d_als _d_file=${_comp_dumpfile-${0:h}/compinit.dump}.$HOST.$$ typeset -U _d_files -_d_files=( ${^~fpath:/.}/_(|*[^~])(N:t) ) +_d_files=( ${^~fpath:/.}/^([^_]*|*~|*.zwc)(N:t) ) print "#files: $#_d_files" > $_d_file diff -ru ../z.old/Completion/Core/compinit Completion/Core/compinit --- ../z.old/Completion/Core/compinit Tue Feb 29 13:11:33 2000 +++ Completion/Core/compinit Wed Mar 1 10:26:15 2000 @@ -56,6 +56,7 @@ # default dumpfile) is now the default; to turn off dumping use -D. emulate -L zsh +setopt extendedglob typeset _i_dumpfile _i_files _i_line _i_done _i_dir _i_autodump=1 typeset _i_tag _i_file _i_addfiles @@ -419,7 +420,7 @@ # Now we automatically make the definition files autoloaded. typeset -U _i_files -_i_files=( ${^~fpath:/.}/_(|*[^~])(N:t) ) +_i_files=( ${^~fpath:/.}/^([^_]*|*~|*.zwc)(N:t) ) if [[ $#_i_files -lt 20 || $_compdir = */Core || -d $_compdir/Core ]]; then # Too few files: we need some more directories, # or we need to check that all directories (not just Core) are present. @@ -438,7 +439,7 @@ _i_addfiles[$_i_line]= done fpath=($fpath $_i_addfiles) - _i_files=( ${^~fpath:/.}/_(|*[^~])(N:t) ) + _i_files=( ${^~fpath:/.}/^([^_]*|*~|*.zwc)(N:t) ) fi fi @@ -468,7 +469,7 @@ if [[ -z "$_i_done" ]]; then for _i_dir in $fpath; do [[ $_i_dir = . ]] && continue - for _i_file in $_i_dir/_(|*[^~])(N); do + for _i_file in $_i_dir/^([^_]*|*~|*.zwc)(N); do read -rA _i_line < $_i_file _i_tag=$_i_line[1] shift _i_line diff -ru ../z.old/Doc/Zsh/builtins.yo Doc/Zsh/builtins.yo --- ../z.old/Doc/Zsh/builtins.yo Tue Feb 29 13:11:12 2000 +++ Doc/Zsh/builtins.yo Wed Mar 1 10:11:04 2000 @@ -1283,6 +1283,55 @@ item(tt(which) [ tt(-wpams) ] var(name) ...)( Equivalent to tt(whence -c). ) +findex(zcompile) +cindex(wordcode, creation) +cindex(compilation) +xitem(tt(zcompile) [ tt(-U) ] [ tt(-r) | tt(-m) ] var(file) [ var(function) ... ]) +item(tt(zcompile -t) var(file) [ var(name) ... ])( +This builtin command can be used to create and display files +containing the wordcode for functions. In the first form, a wordcode +file is created. If called with only the var(file) argument, the +wordcode file has the name `var(file)tt(.zwc)' and will be placed in +the same directory as the var(file). This will make the wordcode file +be loaded instead of the normal function file when the function is +autoloaded (see +ifzman(\ +the section `Autoloading Functions' in zmanref(zshfunc) +)\ +ifnzman(\ +noderef(Functions) +) +for a description of how autoloaded functions are searched). + +If there is at least one var(function) argument, the wordcode for all +these functions will be put in the created wordcode var(file). Such +files containing the code for multiple functions are intended to be +used as elements of the tt(FPATH)/tt(fpath) special array. + +If the tt(-U) option is given, aliases in the var(function)s will not +be expanded. If the tt(-r) option is given, the function(s) in the +file will be read and copied into the shell's memory when they are +autoloaded. If the tt(-m) option is given instead, the wordcode file +will be mapped into the shell's memory. This is done in such a way +that multiple instances of the shell running on the same host will +share this mapped function. If neither tt(-r) nor tt(-m) are given, +the tt(zcompile) builtin decides which style is used based on the size +of the resulting wordcode file. + +In every case, the created file contains two versions of the wordcode, +one for big-endian machines and one for small-endian machines. The +upshot of this is that the wordcode file is machine independent and if +it is read or mapped, only one half of the file will really be used +(and mapped). + +In the second form, with the tt(-t) option, an existing wordcode file is +tested. Without further arguments, the names of the function files +used for it are listed. The first line tells the version of the shell +the file was created with and how the file will be used (mapping or +reading the file). With arguments, only the return value is set +to zero if all var(name)s name functiones defined in the file and +non-zero if at least one var(name) is not contained in the wordcode file. +) findex(zmodload) cindex(modules, loading) cindex(loading modules) diff -ru ../z.old/Doc/Zsh/func.yo Doc/Zsh/func.yo --- ../z.old/Doc/Zsh/func.yo Tue Feb 29 13:11:13 2000 +++ Doc/Zsh/func.yo Wed Mar 1 10:23:05 2000 @@ -33,16 +33,29 @@ cindex(functions, autoloading) A function can be marked as em(undefined) using the tt(autoload) builtin (or `tt(functions -u)' or `tt(typeset -fu)'). Such a function has no -body. When the function is first executed, the tt(fpath) -variable will be searched for a file with the same name as the -function. The usual alias expansion during reading will be suppressed if -the tt(autoload) builtin or its equivalent is given the option tt(-U); +body. When the function is first executed, each element of the tt(fpath) +variable will first be searched for a file with the same name as the +function plus the extension tt(.zwc) and then with the name of the +function. The first file will only be used if it was created with the +tt(zcompile) builtin command, if it contains the wordcode for the +function and it is either older than the file with the name of the +function in the same directory or if such a file does not exist. The +usual alias expansion during reading will be suppressed +if the tt(autoload) builtin or its equivalent is given the option +tt(-U), for wordcode files this has to be decided when creating the +file with the tt(-U) option of the tt(zcompile) builtin command; this is recommended for the use of functions supplied with the zsh distribution. Thus to define functions for autoloading, a typical sequence is: example(fpath=(~/myfuncs $fpath) autoload myfunc1 myfunc2 ...) + +The elements of the tt(fpath) array may also name wordcode files +directly. This is mostly useful for wordcode files containing multiple +functions, in which case the file is treated like a directory +containing files for functions and will be searched for the definition +of the function. pindex(KSH_AUTOLOAD, use of) If the tt(KSH_AUTOLOAD) option is set, or the file contains only a simple diff -ru ../z.old/Src/builtin.c Src/builtin.c --- ../z.old/Src/builtin.c Tue Feb 29 13:11:01 2000 +++ Src/builtin.c Tue Feb 29 13:32:09 2000 @@ -124,6 +124,7 @@ BUILTIN("where", 0, bin_whence, 0, -1, 0, "pmsw", "ca"), BUILTIN("which", 0, bin_whence, 0, -1, 0, "ampsw", "c"), BUILTIN("zmodload", 0, bin_zmodload, 0, -1, 0, "ILabcfdipue", NULL), + BUILTIN("zcompile", 0, bin_zcompile, 0, -1, 0, "tUmr", NULL), }; /****************************************/ @@ -2151,7 +2152,8 @@ p->shf = shf; p->npats = 0; p->pats = NULL; - p->heap = 0; + p->alloc = EA_REAL; + p->dump = NULL; p->prog[0] = WCB_LIST((Z_SYNC | Z_END), 0); p->prog[1] = WCB_SUBLIST(WC_SUBLIST_END, 0, 3); diff -ru ../z.old/Src/cond.c Src/cond.c --- ../z.old/Src/cond.c Tue Feb 29 13:11:02 2000 +++ Src/cond.c Tue Feb 29 13:31:56 2000 @@ -206,7 +206,7 @@ &htok)); if (htok) singsub(&right); - save = (!state->prog->heap && + save = (state->prog->alloc != EA_HEAP && !strcmp(opat, right) && pprog != dummy_patprog2); if (!(pprog = patcompile(right, (save ? PAT_ZDUP : PAT_STATIC), diff -ru ../z.old/Src/exec.c Src/exec.c --- ../z.old/Src/exec.c Tue Feb 29 13:11:02 2000 +++ Src/exec.c Wed Mar 1 10:12:49 2000 @@ -1265,16 +1265,25 @@ untokenize(char *s) { if (*s) { - char *p = s; int c; while ((c = *s++)) if (itok(c)) { + char *p = s - 1; + if (c != Nularg) *p++ = ztokens[c - Pound]; - } else - *p++ = c; - *p = '\0'; + + while ((c = *s++)) { + if (itok(c)) { + if (c != Nularg) + *p++ = ztokens[c - Pound]; + } else + *p++ = c; + } + *p = '\0'; + break; + } } } @@ -3010,7 +3019,7 @@ { Shfunc shf; char *s; - int signum, nprg, npats, len, plen, i, htok = 0; + int signum, nprg, sbeg, nstrs, npats, len, plen, i, htok = 0; Wordcode beg = state->pc, end; Eprog prog; Patprog *pp; @@ -3018,26 +3027,40 @@ end = beg + WC_FUNCDEF_SKIP(state->pc[-1]); names = ecgetlist(state, *state->pc++, EC_DUPTOK, &htok); - nprg = *state->pc++ - 4; + nprg = end - beg; + sbeg = *state->pc++; + nstrs = *state->pc++; npats = *state->pc++; - plen = (end - state->pc) * sizeof(wordcode); - len = plen + (npats * sizeof(Patprog)); + nprg = (end - state->pc); + plen = nprg * sizeof(wordcode); + len = plen + (npats * sizeof(Patprog)) + nstrs; if (htok) execsubst(names); while ((s = (char *) ugetnode(names))) { prog = (Eprog) zalloc(sizeof(*prog)); - prog->heap = 0; - prog->len = len; prog->npats = npats; - prog->pats = pp = (Patprog *) zalloc(len); - prog->prog = (Wordcode) (prog->pats + npats); + prog->len = len; + if (state->prog->dump) { + prog->alloc = EA_MAP; + incrdumpcount(state->prog->dump); + prog->pats = pp = (Patprog *) zalloc(npats * sizeof(Patprog)); + prog->prog = state->pc; + prog->strs = state->strs + sbeg; + prog->dump = state->prog->dump; + } else { + prog->alloc = EA_REAL; + prog->pats = pp = (Patprog *) zalloc(len); + prog->prog = (Wordcode) (prog->pats + npats); + prog->strs = (char *) (prog->prog + nprg); + prog->dump = NULL; + memcpy(prog->prog, state->pc, plen); + memcpy(prog->strs, state->strs + sbeg, nstrs); + } for (i = npats; i--; pp++) *pp = dummy_patprog1; - memcpy(prog->prog, state->pc, plen); - prog->strs = (char *) (prog->prog + nprg); prog->shf = NULL; shf = (Shfunc) zalloc(sizeof(*shf)); @@ -3150,7 +3173,10 @@ } } else { freeeprog(shf->funcdef); - shf->funcdef = zdupeprog(stripkshdef(prog, shf->nam)); + if (prog->alloc == EA_MAP) + shf->funcdef = stripkshdef(prog, shf->nam); + else + shf->funcdef = zdupeprog(stripkshdef(prog, shf->nam)); shf->flags &= ~PM_UNDEFINED; } popheap(); @@ -3180,7 +3206,10 @@ } if (!prog) prog = &dummy_eprog; - shf->funcdef = zdupeprog(stripkshdef(prog, shf->nam)); + if (prog->alloc == EA_MAP) + shf->funcdef = stripkshdef(prog, shf->nam); + else + shf->funcdef = zdupeprog(stripkshdef(prog, shf->nam)); shf->flags &= ~PM_UNDEFINED; popheap(); @@ -3339,6 +3368,8 @@ sprintf(buf, "%s/%s", *pp, s); else strcpy(buf, s); + if ((r = try_dump_file(*pp, s, buf))) + return r; unmetafy(buf, NULL); if (!access(buf, R_OK) && (fd = open(buf, O_RDONLY | O_NOCTTY)) != -1) { if ((len = lseek(fd, 0, 2)) != -1) { @@ -3372,7 +3403,7 @@ * contents of that definition. Otherwise, use the entire file. */ /**/ -static Eprog +Eprog stripkshdef(Eprog prog, char *name) { Wordcode pc = prog->prog; @@ -3399,25 +3430,34 @@ { Eprog ret; Wordcode end = pc + WC_FUNCDEF_SKIP(code); - int nprg = pc[2] - 4; - int npats = pc[3]; - int plen, len, i; + int sbeg = pc[2], nstrs = pc[3], nprg, npats = pc[4], plen, len, i; Patprog *pp; - pc += 4; - - plen = (end - pc) * sizeof(wordcode); - len = plen + (npats * sizeof(Patprog)); + pc += 5; - ret = (Eprog) zhalloc(sizeof(*ret)); - ret->heap = 1; + nprg = end - pc; + plen = nprg * sizeof(wordcode); + len = plen + (npats * sizeof(Patprog)) + nstrs; + + if (prog->alloc == EA_MAP) { + ret = prog; + free(prog->pats); + ret->pats = pp = (Patprog *) zalloc(npats * sizeof(Patprog)); + ret->prog = pc; + ret->strs = prog->strs + sbeg; + } else { + ret = (Eprog) zhalloc(sizeof(*ret)); + ret->alloc = EA_HEAP; + ret->pats = pp = (Patprog *) zhalloc(len); + ret->prog = (Wordcode) (ret->pats + npats); + memcpy(ret->prog, pc, plen); + memcpy(ret->strs, prog->strs + sbeg, nstrs); + ret->dump = NULL; + } ret->len = len; ret->npats = npats; - ret->pats = pp = (Patprog *) zhalloc(len); - ret->prog = (Wordcode) (ret->pats + npats); for (i = npats; i--; pp++) *pp = dummy_patprog1; - memcpy(ret->prog, pc, plen); ret->strs = (char *) (ret->prog + nprg); ret->shf = NULL; diff -ru ../z.old/Src/glob.c Src/glob.c --- ../z.old/Src/glob.c Tue Feb 29 13:11:02 2000 +++ Src/glob.c Tue Feb 29 13:31:57 2000 @@ -2327,16 +2327,22 @@ remnulargs(char *s) { if (*s) { - char *t = s, *p = s, c; + char *o = s, c; while ((c = *s++)) - if (!INULL(c)) - *p++ = c; - *p = '\0'; - if (!*t) { - t[0] = Nularg; - t[1] = '\0'; - } + if (INULL(c)) { + char *t = s - 1; + + while ((c = *s++)) + if (!INULL(c)) + *t++ = c; + *t = '\0'; + if (!*o) { + o[0] = Nularg; + o[1] = '\0'; + } + break; + } } } diff -ru ../z.old/Src/lex.c Src/lex.c --- ../z.old/Src/lex.c Tue Feb 29 13:11:03 2000 +++ Src/lex.c Wed Mar 1 09:05:22 2000 @@ -194,7 +194,7 @@ int eclen, ecused, ecfree, ecnpats; Wordcode ecbuf; Eccstr ecstrs; - int ecsoffs; + int ecsoffs, ecssub, ecnfunc; unsigned char *cstack; int csp; @@ -255,6 +255,8 @@ ls->ecbuf = ecbuf; ls->ecstrs = ecstrs; ls->ecsoffs = ecsoffs; + ls->ecssub = ecssub; + ls->ecnfunc = ecnfunc; cmdsp = 0; inredir = 0; hdocs = NULL; @@ -314,6 +316,8 @@ ecbuf = lstack->ecbuf; ecstrs = lstack->ecstrs; ecsoffs = lstack->ecsoffs; + ecssub = lstack->ecssub; + ecnfunc = lstack->ecnfunc; hlinesz = lstack->hlinesz; errflag = 0; diff -ru ../z.old/Src/loop.c Src/loop.c --- ../z.old/Src/loop.c Tue Feb 29 13:11:03 2000 +++ Src/loop.c Tue Feb 29 13:31:57 2000 @@ -524,7 +524,7 @@ opat = pat = ecgetstr(state, EC_DUP, NULL); singsub(&pat); - save = (!state->prog->heap && + save = (state->prog->alloc != EA_HEAP && !strcmp(pat, opat) && *spprog != dummy_patprog2); pat2 = dupstring(pat); @@ -548,7 +548,7 @@ state->pc - 2, &htok)); if (htok) singsub(&pat); - save = (!state->prog->heap && + save = (state->prog->alloc != EA_HEAP && !strcmp(pat, opat) && *spprog != dummy_patprog2); } if (!(pprog = patcompile(pat, (save ? PAT_ZDUP : PAT_STATIC), diff -ru ../z.old/Src/math.c Src/math.c --- ../z.old/Src/math.c Tue Feb 29 13:11:03 2000 +++ Src/math.c Wed Mar 1 09:05:22 2000 @@ -396,7 +396,7 @@ } if (iident(*ptr)) { int func = 0; - char *p, q; + char *p; p = ptr; while (iident(*++ptr)); @@ -413,10 +413,7 @@ ptr++; } } - q = *ptr; - *ptr = '\0'; - yylval = dupstring(p); - *ptr = q; + yylval = dupstrpfx(p, ptr - p); return (func ? FUNC : (cct ? CID : ID)); } else if (cct) { diff -ru ../z.old/Src/mem.c Src/mem.c --- ../z.old/Src/mem.c Tue Feb 29 13:11:03 2000 +++ Src/mem.c Tue Feb 29 13:31:58 2000 @@ -1272,7 +1272,7 @@ printf("blocks is shown. For otherwise used blocks the first few\n"); printf("bytes are shown as an ASCII dump.\n"); } - printf("\nblock list:\nnum\ttnum\taddr\tlen\tstate\tcum\n"); + printf("\nblock list:\nnum\ttnum\taddr\t\tlen\tstate\tcum\n"); for (m = m_l, mf = m_free, ii = fi = ui = 1; ((char *)m) < m_high; m = (struct m_hdr *)(((char *)m) + M_ISIZE + m->len), ii++) { for (j = 0, ms = NULL; j < M_NSMALL && !ms; j++) diff -ru ../z.old/Src/parse.c Src/parse.c --- ../z.old/Src/parse.c Tue Feb 29 13:11:04 2000 +++ Src/parse.c Wed Mar 1 10:16:27 2000 @@ -131,10 +131,11 @@ * - if (type == PIPE), followed by pipe * * WC_FUNCDEF - * - data contains offset to after body-strings + * - data contains offset to after body * - followed by number of names * - followed by names - * - followed by number of codes for body + * - followed by offset to first string + * - followed by length of string table * - followed by number of patterns for body * - follwoed by codes for body * - followed by strings for body @@ -230,28 +231,7 @@ /**/ Eccstr ecstrs; /**/ -int ecsoffs; - -/* Make at least n bytes free (aligned to sizeof(wordcode)). */ - -static int -ecspace(int n) -{ - n = (n + sizeof(wordcode) - 1) / sizeof(wordcode); - - if (ecfree < n) { - int a = (n > 256 ? n : 256); - - ecbuf = (Wordcode) hrealloc((char *) ecbuf, eclen * sizeof(wordcode), - (eclen + a) * sizeof(wordcode)); - eclen += a; - ecfree += a; - } - ecused += n; - ecfree -= n; - - return ecused - 1; -} +int ecsoffs, ecssub, ecnfunc; /* Insert n free code-slots at position p. */ @@ -323,7 +303,7 @@ Eccstr p, q = NULL; for (p = ecstrs; p; q = p, p = p->next) - if (!strcmp(s, p->str)) + if (p->nfunc == ecnfunc && !strcmp(s, p->str)) return p->offs; p = (Eccstr) zhalloc(sizeof(*p)); @@ -332,8 +312,9 @@ q->next = p; else ecstrs = p; - p->offs = (ecsoffs << 2) | (t ? 1 : 0); + p->offs = ((ecsoffs - ecssub) << 2) | (t ? 1 : 0); p->str = s; + p->nfunc = ecnfunc; ecsoffs += l; return p->offs; @@ -370,6 +351,8 @@ ecused = 0; ecstrs = NULL; ecsoffs = ecnpats = 0; + ecssub = 0; + ecnfunc = 0; } /* Build eprog. */ @@ -393,7 +376,8 @@ ret->prog = (Wordcode) (ret->pats + ecnpats); ret->strs = (char *) (ret->prog + ecused); ret->shf = NULL; - ret->heap = 1; + ret->alloc = EA_HEAP; + ret->dump = NULL; for (l = 0; l < ecnpats; l++) ret->pats[l] = dummy_patprog1; memcpy(ret->prog, ecbuf, ecused * sizeof(wordcode)); @@ -1288,8 +1272,8 @@ static void par_funcdef(void) { - int oecused = ecused, oldlineno = lineno, num = 0, sbeg, onp, p, c = 0; - Eccstr ostrs; + int oecused = ecused, oldlineno = lineno, num = 0, onp, p, c = 0; + int so, oecssub = ecssub; lineno = 0; nocorrect = 1; @@ -1311,6 +1295,7 @@ } ecadd(0); ecadd(0); + ecadd(0); nocorrect = 0; if (tok == INOUTPAR) @@ -1318,10 +1303,8 @@ while (tok == SEPER) yylex(); - sbeg = ecsoffs; - ecsoffs = 0; - ostrs = ecstrs; - ecstrs = NULL; + ecnfunc++; + ecssub = so = ecsoffs; onp = ecnpats; ecnpats = 0; @@ -1330,43 +1313,28 @@ par_list(&c); if (tok != OUTBRACE) { lineno += oldlineno; - ecsoffs = sbeg; - ecstrs = ostrs; ecnpats = onp; + ecssub = oecssub; YYERRORV(oecused); } yylex(); } else if (unset(SHORTLOOPS)) { lineno += oldlineno; - ecsoffs = sbeg; - ecstrs = ostrs; ecnpats = onp; + ecssub = oecssub; YYERRORV(oecused); } else par_list1(&c); ecadd(WCB_END()); - ecbuf[p + num + 2] = ecused - num - p; - ecbuf[p + num + 3] = ecnpats; + ecbuf[p + num + 2] = so - oecssub; + ecbuf[p + num + 3] = ecsoffs - so; + ecbuf[p + num + 4] = ecnpats; ecbuf[p + 1] = num; - if (ecsoffs) { - int beg = ecused, l; - Eccstr sp; - char *sq; - - ecspace(ecsoffs); - - for (sp = ecstrs, sq = (char *) (ecbuf + beg); sp; - sp = sp->next, sq += l) { - l = strlen(sp->str) + 1; - memcpy(sq, sp->str, l); - } - } lineno += oldlineno; - ecsoffs = sbeg; - ecstrs = ostrs; ecnpats = onp; + ecssub = oecssub; ecbuf[p] = WCB_FUNCDEF(ecused - 1 - p); } @@ -1481,8 +1449,7 @@ p += 3; /* 3 codes per redirection */ sr++; } else if (tok == INOUTPAR) { - int oldlineno = lineno, sbeg, onp; - Eccstr ostrs; + int oldlineno = lineno, onp, so, oecssub = ecssub; *complex = c; lineno = 0; @@ -1496,11 +1463,10 @@ ecbuf[p + 1] = argc; ecadd(0); ecadd(0); + ecadd(0); - sbeg = ecsoffs; - ecsoffs = 0; - ostrs = ecstrs; - ecstrs = NULL; + ecnfunc++; + ecssub = so = ecsoffs; onp = ecnpats; ecnpats = 0; @@ -1512,9 +1478,8 @@ if (tok != OUTBRACE) { cmdpop(); lineno += oldlineno; - ecsoffs = sbeg; - ecstrs = ostrs; ecnpats = onp; + ecssub = oecssub; YYERROR(oecused); } yylex(); @@ -1532,26 +1497,13 @@ cmdpop(); ecadd(WCB_END()); - ecbuf[p + argc + 2] = ecused - argc - p; - ecbuf[p + argc + 3] = ecnpats; + ecbuf[p + argc + 2] = so - oecssub; + ecbuf[p + argc + 3] = ecsoffs - so; + ecbuf[p + argc + 4] = ecnpats; - if (ecsoffs) { - int beg = ecused, l; - Eccstr sp; - char *sq; - - ecspace(ecsoffs); - - for (sp = ecstrs, sq = (char *) (ecbuf + beg); sp; - sp = sp->next, sq += l) { - l = strlen(sp->str) + 1; - memcpy(sq, sp->str, l); - } - } lineno += oldlineno; - ecsoffs = sbeg; - ecstrs = ostrs; ecnpats = onp; + ecssub = oecssub; ecbuf[p] = WCB_FUNCDEF(ecused - 1 - p); @@ -2020,7 +1972,8 @@ return p; r = (Eprog) zalloc(sizeof(*r)); - r->heap = 0; + r->alloc = EA_REAL; + r->dump = NULL; r->len = p->len; r->npats = p->npats; pp = r->pats = (Patprog *) zcalloc(r->len); @@ -2056,7 +2009,11 @@ while ((p = (Eprog) getlinknode(eprog_free))) { for (i = p->npats, pp = p->pats; i--; pp++) freepatprog(*pp); - zfree(p->pats, p->len); + if (p->dump) { + decrdumpcount(p->dump); + zfree(p->pats, p->npats * sizeof(Patprog)); + } else + zfree(p->pats, p->len); zfree(p, sizeof(*p)); } } @@ -2083,6 +2040,17 @@ } if (tok) *tok = (c & 1); + + /*** Since function dump files are mapped read-only, avoiding to + * to duplicate strings when they don't contain tokens may fail + * when one of the many utility functions happens to write to + * one of the strings (without really modifying it). + * If that happens to you and you don't feel like debugging it, + * just change the line below to: + * + * return (dup ? dupstring(r) : r); + */ + return ((dup == EC_DUP || (dup && (c & 1))) ? dupstring(r) : r); } @@ -2193,3 +2161,530 @@ eprog_free = znewlinklist(); } + +/* Code for function dump files. + * + * Dump files consist of a header and the function bodies (the wordcode + * plus the string table) and that twice: once for the byte-order of the + * host the file was created on and once for the other byte-order. The + * header describes where the beginning of the `other' version is and it + * is up to the shell reading the file to decide which version it needs. + * This is done by checking if the first word is FD_MAGIC (then the + * shell reading the file has the same byte order as the one that created + * the file) or if it is FD_OMAGIC, then the `other' version has to be + * read. + * The header is the magic number, a word containing the flags (if the + * file should be mapped or read and if this header is the `other' one), + * the version string in a field of 40 characters and the descriptions + * for the functions in the dump file. + * Each description consists of a struct fdhead followed by the name, + * aligned to sizeof(wordcode) (i.e. 4 bytes). + */ + +#include "version.h" + +#define FD_EXT ".zwc" +#define FD_MINMAP 4096 + +#define FD_PRELEN 12 +#define FD_MAGIC 0x01020304 +#define FD_OMAGIC 0x04030201 + +#define FDF_MAP 1 +#define FDF_OTHER 2 + +typedef struct fdhead *FDHead; + +struct fdhead { + wordcode start; /* offset to function definition */ + wordcode len; /* length of wordcode/strings */ + wordcode npats; /* number of patterns needed */ + wordcode strs; /* offset to strings */ + wordcode hlen; /* header length (incl. name) */ + wordcode tail; /* offset to name tail */ +}; + +#define fdheaderlen(f) (((Wordcode) (f))[FD_PRELEN]) + +#define fdmagic(f) (((Wordcode) (f))[0]) +#define fdbyte(f, i) ((wordcode) (((unsigned char *) (((Wordcode) (f)) + 1))[i])) +#define fdflags(f) fdbyte(f, 0) +#define fdother(f) (fdbyte(f, 1) + (fdbyte(f, 2) << 8) + (fdbyte(f, 3) << 16)) +#define fdsetother(f, o) \ + do { \ + fdbyte(f, 1) = (o & 0xff); \ + fdbyte(f, 2) = (o >> 8) & 0xff; \ + fdbyte(f, 3) = (o >> 16) & 0xff; \ + } while (0) +#define fdversion(f) ((char *) ((f) + 2)) + +#define firstfdhead(f) ((FDHead) (((Wordcode) (f)) + FD_PRELEN)) +#define nextfdhead(f) ((FDHead) (((Wordcode) (f)) + (f)->hlen)) + +#define fdname(f) ((char *) (((FDHead) (f)) + 1)) + +/* Try to find the description for the given function name. */ + +static FDHead +dump_find_func(Wordcode h, char *name) +{ + FDHead n, e = (FDHead) (h + fdheaderlen(h)); + + for (n = firstfdhead(h); n < e; n = nextfdhead(n)) + if (!strcmp(name, fdname(n) + n->tail)) + return n; + + return NULL; +} + +/**/ +int +bin_zcompile(char *nam, char **args, char *ops, int func) +{ + int map; + + if (ops['t']) { + Wordcode f; + + if (!*args) { + zerrnam(nam, "too few arguments", NULL, 0); + return 1; + } + if (!(f = load_dump_header(*args))) { + zerrnam(nam, "invalid dump file: %s", *args, 0); + return 1; + } + if (args[1]) { + for (args++; *args; args++) + if (!dump_find_func(f, *args)) + return 1; + return 0; + } else { + FDHead h, e = (FDHead) (f + fdheaderlen(f)); + + printf("function dump file (%s) for zsh-%s\n", + ((fdflags(f) & FDF_MAP) ? "mapped" : "read"), fdversion(f)); + for (h = firstfdhead(f); h < e; h = nextfdhead(h)) + printf("%s\n", fdname(h)); + return 0; + } + } + if (!*args) { + zerrnam(nam, "too few arguments", NULL, 0); + return 1; + } + map = (ops['m'] ? 2 : (ops['r'] ? 0 : 1)); + + if (!args[1]) + return build_dump(nam, dyncat(*args, FD_EXT), args, ops['U'], map); + + return build_dump(nam, *args, args + 1, ops['U'], map); +} + +/* Load the header of a dump file. Returns NULL if the file isn't a + * valid dump file. */ + +/**/ +static Wordcode +load_dump_header(char *name) +{ + int fd; + wordcode buf[FD_PRELEN + 1]; + + if ((fd = open(name, O_RDONLY)) < 0) + return NULL; + + if (read(fd, buf, (FD_PRELEN + 1) * sizeof(wordcode)) != + ((FD_PRELEN + 1) * sizeof(wordcode)) || + strcmp(ZSH_VERSION, fdversion(buf))) { + close(fd); + return NULL; + } else { + int len; + Wordcode head; + + if (fdmagic(buf) == FD_MAGIC) { + len = fdheaderlen(buf) * sizeof(wordcode); + head = (Wordcode) zhalloc(len); + } + else { + int o = fdother(buf); + + if (lseek(fd, o, 0) == -1 || + read(fd, buf, (FD_PRELEN + 1) * sizeof(wordcode)) != + ((FD_PRELEN + 1) * sizeof(wordcode))) { + close(fd); + return NULL; + } + len = fdheaderlen(buf) * sizeof(wordcode); + head = (Wordcode) zhalloc(len); + } + memcpy(head, buf, (FD_PRELEN + 1) * sizeof(wordcode)); + + if (read(fd, head + (FD_PRELEN + 1), + len - ((FD_PRELEN + 1) * sizeof(wordcode))) != + len - ((FD_PRELEN + 1) * sizeof(wordcode))) { + close(fd); + return NULL; + } + close(fd); + return head; + } +} + +/* Swap the bytes in a wordcode. */ + +static void +fdswap(Wordcode p, int n) +{ + wordcode c; + + for (; n--; p++) { + c = *p; + *p = (((c & 0xff) << 24) | + ((c & 0xff00) << 8) | + ((c & 0xff0000) >> 8) | + ((c & 0xff000000) >> 24)); + } +} + +/* Write a dump file. */ + +/**/ +static int +build_dump(char *nam, char *dump, char **files, int ali, int map) +{ + int dfd, fd, hlen, tlen, flen, tmp, ona = noaliases, other = 0, ohlen; + LinkList progs; + LinkNode node; + struct fdhead head; + wordcode pre[FD_PRELEN]; + char *file, **ofiles = files, **oofiles = files, *name, *tail; + Eprog prog; + + if ((dfd = open(dump, O_WRONLY|O_CREAT, 0600)) < 0) { + zerrnam(nam, "can't write dump file: %s", dump, 0); + return 1; + } + progs = newlinklist(); + noaliases = ali; + + for (hlen = FD_PRELEN, tlen = 0; *files; files++) { + if ((fd = open(*files, O_RDONLY)) < 0 || + (flen = lseek(fd, 0, 2)) == -1) { + if (fd >= 0) + close(fd); + close(dfd); + zerrnam(nam, "can't open file: %s", *files, 0); + noaliases = ona; + return 1; + } + file = (char *) zalloc(flen + 1); + file[flen] = '\0'; + lseek(fd, 0, 0); + if (read(fd, file, flen) != flen) { + close(fd); + close(dfd); + zfree(file, flen); + zerrnam(nam, "can't read file: %s", *files, 0); + noaliases = ona; + return 1; + } + close(fd); + file = metafy(file, flen, META_REALLOC); + + if (!(prog = parse_string(file, 1)) || errflag) { + close(dfd); + zfree(file, flen); + zerrnam(nam, "can't read file: %s", *files, 0); + noaliases = ona; + return 1; + } + zfree(file, flen); + + addlinknode(progs, prog); + + flen = (strlen(*files) + sizeof(wordcode)) / sizeof(wordcode); + hlen += (sizeof(head) / sizeof(wordcode)) + flen; + + tlen += (prog->len - (prog->npats * sizeof(Patprog)) + + sizeof(wordcode) - 1) / sizeof(wordcode); + } + noaliases = ona; + + tlen = (tlen + hlen) * sizeof(wordcode); + if (map == 1) + map = (tlen >= FD_MINMAP); + + for (ohlen = hlen; ; hlen = ohlen) { + fdmagic(pre) = (other ? FD_OMAGIC : FD_MAGIC); + fdflags(pre) = (map ? FDF_MAP : 0) | other; + fdsetother(pre, tlen); + strcpy(fdversion(pre), ZSH_VERSION); + write(dfd, pre, FD_PRELEN * sizeof(wordcode)); + + for (node = firstnode(progs), ofiles = oofiles; node; + ofiles++, incnode(node)) { + prog = (Eprog) getdata(node); + head.start = hlen; + hlen += (prog->len - (prog->npats * sizeof(Patprog)) + + sizeof(wordcode) - 1) / sizeof(wordcode); + head.len = prog->len - (prog->npats * sizeof(Patprog)); + head.npats = prog->npats; + head.strs = prog->strs - ((char *) prog->prog); + head.hlen = (sizeof(struct fdhead) / sizeof(wordcode)) + + (strlen(*ofiles) + sizeof(wordcode)) / sizeof(wordcode); + for (name = tail = *ofiles; *name; name++) + if (*name == '/') + tail = name + 1; + head.tail = tail - *ofiles; + if (other) + fdswap((Wordcode) &head, sizeof(head) / sizeof(wordcode)); + write(dfd, &head, sizeof(head)); + tmp = strlen(*ofiles) + 1; + write(dfd, *ofiles, tmp); + if ((tmp &= (sizeof(wordcode) - 1))) + write(dfd, &head, sizeof(wordcode) - tmp); + } + for (node = firstnode(progs); node; incnode(node)) { + prog = (Eprog) getdata(node); + tmp = (prog->len - (prog->npats * sizeof(Patprog)) + + sizeof(wordcode) - 1) / sizeof(wordcode); + if (other) + fdswap(prog->prog, (((Wordcode) prog->strs) - prog->prog)); + write(dfd, prog->prog, tmp * sizeof(wordcode)); + } + if (other) + break; + other = FDF_OTHER; + } + close(dfd); + return 0; +} + +#if defined(HAVE_SYS_MMAN_H) && defined(HAVE_MMAP) && defined(HAVE_MUNMAP) + +#include + +#if defined(MAP_SHARED) && defined(PROT_READ) + +#define USE_MMAP 1 + +#endif +#endif + +#ifdef USE_MMAP + +/* List of dump files mapped. */ + +static FuncDump dumps; + +/* Load a dump file (i.e. map it). */ + +static void +load_dump_file(char *dump, int other, int len) +{ + FuncDump d; + Wordcode addr; + int fd, off; + + if (other) { + static size_t pgsz = 0; + + if (!pgsz) { + +#ifdef _SC_PAGESIZE + pgsz = sysconf(_SC_PAGESIZE); /* SVR4 */ +#else +# ifdef _SC_PAGE_SIZE + pgsz = sysconf(_SC_PAGE_SIZE); /* HPUX */ +# else + pgsz = getpagesize(); +# endif +#endif + + pgsz--; + } + off = len & ~pgsz; + } else + off = 0; + + if ((fd = open(dump, O_RDONLY)) < 0) + return; + + fd = movefd(fd); + + if ((addr = (Wordcode) mmap(NULL, len, PROT_READ, MAP_SHARED, fd, off)) == + ((Wordcode) -1)) { + close(fd); + return; + } + d = (FuncDump) zalloc(sizeof(*d)); + d->next = dumps; + dumps = d; + d->name = ztrdup(dump); + d->fd = fd; + d->map = addr + (other ? (len - off) / sizeof(wordcode) : 0); + d->addr = addr; + d->len = len; + d->count = 0; +} + +/* See if `dump' is the name of a dump file and it has the definition + * for the function `name'. If so, return an eprog for it. */ + +/**/ +Eprog +try_dump_file(char *dump, char *name, char *func) +{ + int isrec = 0; + Wordcode d; + FDHead h; + FuncDump f; + + rec: + + d = NULL; + for (f = dumps; f; f = f->next) + if (!strcmp(dump, f->name)) { + d = f->map; + break; + } + if (!f && (isrec || !(d = load_dump_header(dump)))) { + if (!isrec) { + struct stat stc, stn; + char *p = (char *) zhalloc(strlen(dump) + strlen(name) + + strlen(FD_EXT) + 2); + + sprintf(p, "%s/%s%s", dump, name, FD_EXT); + + /* Ignore the dump file if it is older than the normal one. */ + if (stat(p, &stc) || stat(func, &stn) || stn.st_mtime > stc.st_mtime) + return NULL; + + if (!(d = load_dump_header(dump = p))) + return NULL; + + } else + return NULL; + } + if ((h = dump_find_func(d, name))) { + /* Found the name. If the file is already mapped, return the eprog, + * otherwise map it and just go up. */ + if (f) { + Eprog prog = (Eprog) zalloc(sizeof(*prog)); + Patprog *pp; + int np; + + prog->alloc = EA_MAP; + prog->len = h->len; + prog->npats = np = h->npats; + prog->pats = pp = (Patprog *) zalloc(np * sizeof(Patprog)); + prog->prog = f->map + h->start; + prog->strs = ((char *) prog->prog) + h->strs; + prog->shf = NULL; + prog->dump = f; + + incrdumpcount(f); + + while (np--) + *pp++ = dummy_patprog1; + + return prog; + } else if (fdflags(d) & FDF_MAP) { + load_dump_file(dump, (fdflags(d) & FDF_OTHER), fdother(d)); + isrec = 1; + goto rec; + } else { + Eprog prog; + Patprog *pp; + int np, fd, po = h->npats * sizeof(Patprog); + + if ((fd = open(dump, O_RDONLY)) < 0 || + lseek(fd, ((h->start * sizeof(wordcode)) + + ((fdflags(d) & FDF_OTHER) ? fdother(d) : 0)), 0) < 0) { + if (fd >= 0) + close(fd); + return NULL; + } + d = (Wordcode) zalloc(h->len + po); + + if (read(fd, ((char *) d) + po, h->len) != h->len) { + close(fd); + zfree(d, h->len); + + return NULL; + } + close(fd); + + prog = (Eprog) zalloc(sizeof(*prog)); + + prog->alloc = EA_MAP; + prog->len = h->len + po; + prog->npats = np = h->npats; + prog->pats = pp = (Patprog *) d; + prog->prog = (Wordcode) (((char *) d) + po); + prog->strs = ((char *) prog->prog) + h->strs; + prog->shf = NULL; + prog->dump = f; + + while (np--) + *pp++ = dummy_patprog1; + + return prog; + } + } + return NULL; +} + +/* Increment the reference counter for a dump file. */ + +/**/ +void +incrdumpcount(FuncDump f) +{ + f->count++; +} + +/* Decrement the reference counter for a dump file. If zero, unmap the file. */ + +/**/ +void +decrdumpcount(FuncDump f) +{ + f->count--; + if (!f->count) { + FuncDump p, q; + + for (q = NULL, p = dumps; p && p != f; q = p, p = p->next); + if (p) { + if (q) + q->next = p->next; + else + dumps = p->next; + munmap((void *) f->addr, f->len); + zclose(f->fd); + zfree(f, sizeof(*f)); + } + } +} + +#else + +Eprog +try_dump_file(char *dump, char *name, char *func) +{ + return NULL; +} + +void +incrdumpcount(FuncDump f) +{ +} + +void +decrdumpcount(FuncDump f) +{ +} + +#endif diff -ru ../z.old/Src/text.c Src/text.c --- ../z.old/Src/text.c Tue Feb 29 13:11:05 2000 +++ Src/text.c Wed Mar 1 09:05:23 2000 @@ -378,8 +378,8 @@ n = tpush(code, 1); n->u._funcdef.strs = state->strs; n->u._funcdef.end = end; - state->strs = (char *) (p + (*state->pc)); - state->pc += 2; + state->strs += *state->pc; + state->pc += 3; } } else { state->strs = s->u._funcdef.strs; diff -ru ../z.old/Src/utils.c Src/utils.c --- ../z.old/Src/utils.c Tue Feb 29 13:11:05 2000 +++ Src/utils.c Tue Feb 29 13:31:58 2000 @@ -1704,7 +1704,7 @@ { char *t, **ret, **ptr; int l = sizeof(*ret) * (wordcount(s, NULL, -!allownull) + 1); - char *(*dup)(char *) = (heap ? dupstring : ztrdup); + char *(*dup)(const char *) = (heap ? dupstring : ztrdup); ptr = ret = (heap ? (char **) hcalloc(l) : (char **) zcalloc(l)); diff -ru ../z.old/Src/zsh.h Src/zsh.h --- ../z.old/Src/zsh.h Tue Feb 29 13:11:05 2000 +++ Src/zsh.h Wed Mar 1 09:05:23 2000 @@ -476,24 +476,40 @@ #define MAX_ARRLEN 262144 /********************************************/ -/* Defintions for byte code */ +/* Defintions for word code */ /********************************************/ typedef unsigned int wordcode; typedef wordcode *Wordcode; +typedef struct funcdump *FuncDump; typedef struct eprog *Eprog; +struct funcdump { + FuncDump next; /* next in list */ + char *name; /* path name */ + int fd; /* file descriptor */ + Wordcode map; /* pointer to header */ + Wordcode addr; /* mapped region */ + int len; /* length */ + int count; /* reference count */ +}; + struct eprog { - int heap; /* != 0 if in heap memory */ + int alloc; /* EA_* below */ int len; /* total block length */ int npats; /* Patprog cache size */ Patprog *pats; /* the memory block, the patterns */ Wordcode prog; /* memory block ctd, the code */ char *strs; /* memory block ctd, the strings */ Shfunc shf; /* shell function for autoload */ + FuncDump dump; /* dump file this is in */ }; +#define EA_REAL 0 +#define EA_HEAP 1 +#define EA_MAP 2 + typedef struct estate *Estate; struct estate { @@ -508,6 +524,7 @@ Eccstr next; char *str; wordcode offs; + int nfunc; }; #define EC_NODUP 0 -- Sven Wischnowsky wischnow@informatik.hu-berlin.de