Re: architecture independent wordcode files (.zwc)

Re: architecture independent wordcode files (.zwc)

[Paul Lew]

>>>>> "Bart" == Bart Schaefer <schaefer@candle.brasslantern.com> writes:

    Bart> On May 12,  1:09pm, Paul Lew wrote:
    Bart> } Subject: architecture independent wordcode files (.zwc)
    Bart> }
    Bart> }     Right now we have .zshenv compiled into .zshenv.zwc
    Bart> }     and .zshrc compiled into .zshrc.zwc on the solaris
    Bart> }     machine. However, this does not work for the Linux. I
    Bart> }     got zsh cored when reading the .zwc file for the wrong
    Bart> }     architecture.
    Bart> }
    Bart> }     The emacs .elc files works on different endian
    Bart> }     machines. Any plan to do the same with .zwc files?

    Bart> It's supposed to work -- the zcompile doc says:

    Bart> The created file always contains two versions of the
    Bart> compiled format, one for big-endian machines and one for
    Bart> small-endian machines.  The upshot of this is that the
    Bart> compiled file is machine indepen- dent and if it is read or
    Bart> mapped, only one half of the file is actually used (and
    Bart> mapped).

    Bart> If possible, please send a stack trace from your core dump
    Bart> to zsh-workers.  Thanks.

muggle paullew Src>> uname -sr
Linux 2.2.14-5.0mvfs

muggle paullew Src>> file ~/zsh-i686-gdb.core
/users/paullew/zsh-i686-gdb.core: ELF 32-bit LSB core file of 'zsh'
(signal 11), Intel 80386, version 1

muggle paullew Src>> gdb =zsh ~/zsh-i686-gdb.core
GNU gdb 19991004
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and
you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "i386-redhat-linux"...
Core was generated by `zsh'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib/libdl.so.2...done.
Reading symbols from /lib/libnsl.so.1...done.
Reading symbols from /lib/libtermcap.so.2...done.
Reading symbols from /lib/libm.so.6...done.
Reading symbols from /lib/libc.so.6...done.
Reading symbols from /lib/ld-linux.so.2...done.
Reading symbols from /lib/libnss_files.so.2...done.
Reading symbols from /lib/libnss_nis.so.2...done.
#0  0x400b5c27 in strlen (str=0x4001aff8 <Address 0x4001aff8 out of #bounds>)
    at ../sysdeps/i386/strlen.c:27
27      ../sysdeps/i386/strlen.c: No such file or directory.
(gdb) where
#0  0x400b5c27 in strlen (str=0x4001aff8 <Address 0x4001aff8 out of #bounds>)
    at ../sysdeps/i386/strlen.c:27
#1  0x80a8a59 in dupstring (s=0x4001aff8 <Address 0x4001aff8 out of #bounds>)
    at string.c:39
#2  0x809d869 in ecgetstr (s=0xbffff90c, dup=2, tok=0xbffff880) at parse.c:2110
#3  0x8063ded in addvars (state=0xbffff90c, pc=0x4001a374, export=0)
    at exec.c:1505
#4  0x8061da0 in execsimple (state=0xbffff90c) at exec.c:753
#5  0x8061f06 in execlist (state=0xbffff90c, dont_change_job=1, exiting=0)
    at exec.c:806
#6  0x8061d07 in execode (p=0x80f0288, dont_change_job=1, exiting=0)
    at exec.c:729
#7  0x8079f84 in source (s=0xbffff974 "/users/paullew/.zshenv") at init.c:996
#8  0x807a230 in sourcehome (s=0x80bc7e0 ".zshenv") at init.c:1044
#9  0x8079c49 in run_init_scripts () at init.c:893
#10 0x807aa26 in zsh_main (argc=1, argv=0xbffffb34) at init.c:1204
#11 0x80517b0 in main (argc=1, argv=0xbffffb34) at ./main.c:37
(gdb) quit

Please let me know if more info is needed.  Thanks.

Re: architecture independent wordcode files (.zwc)

[Bart Schaefer]

A thought that occurred to me, though I don't know if this is actually the
problem Paul is having:

/********************************************/
/* Defintions for word code                 */
/********************************************/

typedef unsigned int wordcode;
typedef wordcode *Wordcode;


There are architectures where `unsigned int' is 16 bits or 64 bits.  We
should be determining a 32-bit type via configure and using it here.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: architecture independent wordcode files (.zwc)

[Andrej Borsenkow]

On Sun, 13 May 2001, Bart Schaefer wrote:

> A thought that occurred to me, though I don't know if this is actually the
> problem Paul is having:
>
> /********************************************/
> /* Defintions for word code                 */
> /********************************************/
>
> typedef unsigned int wordcode;
> typedef wordcode *Wordcode;
>
>
> There are architectures where `unsigned int' is 16 bits or 64 bits.  We
> should be determining a 32-bit type via configure and using it here.
>

Yes, that occured to me as well. We cover big/little endian case, but
what about 32 vs 64 bit systems? There may be subtle differences there,
e.g. in the way integer parameter are interpreted.

-andrej

Re: architecture independent wordcode files (.zwc)

[Sven Wischnowsky]

Bart Schaefer wrote:

> ...
> 
> There are architectures where `unsigned int' is 16 bits or 64 bits.  We
> should be determining a 32-bit type via configure and using it here.

When we added zcompile, I asked if there were systems on which
sizeof(int) != 4 but got no reply.  And I suggested using a configure
test but I still don't know enough about configure to write this test
myself...


Paul, I had tested it with the three architectures I have access to
here: Redhat Linux 5.someting, Solaris 2.6 and Tru64 Unix.  That still
works for me for the test cases I tried (between the latter to systems
this includes the completion system).  So I need more information, could
you send us the uncompiled files (or a simplified but still not-working
version if there's something in there you don't want to show)?

Bye
  Sven


-- 
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de

Re: architecture independent wordcode files (.zwc)

[Paul Lew]

[-- Attachment #1: message body text --]
[-- Type: text/plain, Size: 1200 bytes --]

>>>>> "Sven" == Sven Wischnowsky <wischnow@informatik.hu-berlin.de> writes:

    Sven> Bart Schaefer wrote:
    >> ...
    >> 
    >> There are architectures where `unsigned int' is 16 bits or 64
    >> bits.  We should be determining a 32-bit type via configure and
    >> using it here.

    Sven> When we added zcompile, I asked if there were systems on
    Sven> which sizeof(int) != 4 but got no reply.  And I suggested
    Sven> using a configure test but I still don't know enough about
    Sven> configure to write this test myself...

    Sven> Paul, I had tested it with the three architectures I have
    Sven> access to here: Redhat Linux 5.someting, Solaris 2.6 and
    Sven> Tru64 Unix.  That still works for me for the test cases I
    Sven> tried (between the latter to systems this includes the
    Sven> completion system).  So I need more information, could you
    Sven> send us the uncompiled files (or a simplified but still
    Sven> not-working version if there's something in there you don't
    Sven> want to show)?

I have lots of startup files for zsh, fortunately, the .zshenv is good
enough to trigger the problem.  Attached below.

					-- Paul  05/14/01  09:10 AM --


[-- Attachment #2: .zshenv --]
[-- Type: text/plain, Size: 7090 bytes --]

#! /auto/insbu-cnstools/bin/zsh -f
#
#	Z shell environment setup
#
#	Created at:	06/05/91  08:44 PM
#	Last update:	05/12/01  07:34 PM  (Edition: 177)
#
[[ -O /tmp/zshenv-debug.$LOGNAME ]] && . /tmp/zshenv-debug.$LOGNAME
#---------------------------------------------------------------#
#		      Command Search Path			#
#---------------------------------------------------------------#
artool=/auto/insbu-cnstools/ar
[[ -d $HOME/.zdata ]] || mkdir $HOME/.zdata
zpath=$HOME/.zdata/zpath.$(/bin/uname -n)
[[ $zpath -nt .zshenv ]] || /bin/rm -f $zpath
case $(/bin/uname -sr) in
	Linux\ 2.2.14*)					# Linux redhat 7.0
		bindir=binx86
		libdir=libx86
		zshenv=/bin/zsh				;;
	SunOS\ 5.5.*)
		bindir=bin-5.5
		libdir=lib-5.5
		zshenv=/usr/local/bin/zsh		;;
	*)	bindir=bin
		libdir=lib
		zshenv=/auto/insbu-cnstools/bin/zsh	;;
	esac
if [[ -r $zpath ]]; then
	path=($(< $zpath ))
else
	#odeuname=/usr/sde/tools/common/ode_uname
	dirs=(.)
	dirs=($dirs $artool/$bindir $artool/sh $artool/perl)
	dirs=($dirs /auto/insbu-cnstools/$bindir)		#cisco CNS
	dirs=($dirs /auto/insbu-cnstools/s$bindir)		#cisco CNS

	#dirs=($dirs /auto/insbu-cnstools/Office51/bin)		#Star Office
	dirs=($dirs /auto/cnsadpub/cns/$bindir)			#cisco CNS
	dirs=($dirs /auto/insbu-cnstools/apache/$bindir)	#Cisco apache
	dirs=($dirs /auto/insbu-cnstools/mysql/$bindir)		#Cisco mysql
	dirs=($dirs /opt/AICar1/usrbin)				#cisco AR
	dirs=($dirs /nfs/csc/mib-release/bin)			#cisco MIB
	dirs=($dirs /auto/insbu-cnstools/java/$bindir)		#cisco java
	dirs=($dirs /usr/local/java/bin)
	dirs=($dirs /usr/local/bin /usr/local/sbin)
	dirs=($dirs /sw/licensed/sun/SUNWspro/5.0/sparc-sun-solaris2.6/bin)
								#SunPro C++
	dirs=($dirs /usr/dt/bin /router/bin)			#cisco
	dirs=($dirs /sw/current/solaris2bin)			#cisco
	dirs=($dirs /usr/atria/bin)				#Clearcase
	dirs=($dirs /usr/local/ddts/bin)			#cisco
	dirs=($dirs $([[ -r $odeuname ]] && $odeuname -p))	#css stuff
	dirs=($dirs /www/bin)					#mhttpd stuff
	dirs=($dirs /usr/X11R6/bin /usr/openwin/bin)		#SunOS
	dirs=($dirs /usr/etc/netls)				#hpux
	dirs=($dirs /usr/bin/X11)
	dirs=($dirs /sbin /usr/sbin /usr/lbin)			#osf + ultrix
	dirs=($dirs /usr/ccs/bin)				#solaris
	dirs=($dirs /usr/X11/bin /usr/openwin/bin)		#Linux
	dirs=($dirs /usr/andrew/bin)				#Linux
	dirs=($dirs /usr/ucb /usr/bin /bin)
	dirs=($dirs /etc /usr/etc)
	dirs=($dirs /usr/local/games /usr/games /pub/games)
	unset odeuname
	# if [[ $(uname -s) = "OSF1" && $(uname -v) -ge 878 ]]; then
	#	export _RLD_ARGS=-ignore_all_versions           #osf 4.0d
	#							#need for less
	#	fi
	path=()
	foreach dir ($dirs)
		if [[ -d $dir ]]; then path=(${path} $dir); fi
		end
	echo $path > $zpath
	chmod go-w $zpath
	unset dirs dir
	fi

case $(uname) in
	SunOS)
		LD_LIBRARY_PATH=/auto/insbu-cnstools/$libdir
		LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
		LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/sw/licensed/sun/SUNWspro/5.0/sparc-sun-solaris2.6/lib
		export LD_LIBRARY_PATH					;;
	Linux)
		LD_LIBRARY_PATH=/auto/insbu-cnstools/$libdir
		LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
		export LD_LIBRARY_PATH					;;
	OSF1)	export XMCD_LIBDIR=/usr/lib/X11/xmcd			;;
	*)								;;
	esac
#---------------------------------------------------------------#
#		Application Environment Variables		#
#---------------------------------------------------------------#
# Allow group write, to protect valueble data, set up .in and .out
# file in the directory
umask 002
HPATH=~/info/tcsh
PAGER=less
MORE='-cs'

LESS='-Xj5-PM?pb%pb\%. ?f%f:stdin.  ?ltLine\: %lt-%lb:Byte\: %bt-%bb.'
LESS="$LESS"'  ?ssize\: %s. ?m(%i of %m). ?e(EOF) ?x- Next\: %x.$'
LESS="${LESS}MicesaO/usr/tmp/less_$UID.xxxxx"

EDITOR='vi'
#PHONES=~paullew/dat/phones.tip
#REMOTE=~paullew/dat/remote.tip
#HELPDIR=~/lib/gnuplot-help
HELPDIR=$artool/zsh/zsh-help
TORSO='-n -c3'
FCODES=~/log/$USER.bg
FTPR_PASSWORD="-$LOGNAME@cisco.com"

# make sure sub-shells are zsh
export SHELL=$zshenv

# for GNU emacs shell mode
ESHELL=/bin/csh

TGIF_LOGIC=~/lib/tgif/logic
TGIF_FLOWCHART=~/lib/tgif/flowchart
TGIF_PETRI=~/lib/tgif/petri
TGIF_LATEX=~/lib/tgif/latex

GUI_BUILDER=/usr/local/lib/GuiBuilder

# Basic types handled by zsh postfix characters, we just use color to
# highlight the archive, program source code, etc.
LS_COLORS='no=00:fi=00:di=00:ln=00:pi=00:so=00:bd=00:cd=00:ex=00'
LS_COLORS="${LS_COLORS}:*.cmd=00;35:*.exe=00;35:*.com=00;35"
LS_COLORS="${LS_COLORS}:*.btm=00;35:*.bat=00;35"
LS_COLORS="${LS_COLORS}:*.tar=00;31:*.tgz=00;31:*.arj=00;31:*.taz=00;31"
LS_COLORS="${LS_COLORS}:*.lzh=00;31:*.zip=00;31"
LS_COLORS="${LS_COLORS}:*.z=00;31:*.Z=00;31:*.gz=00;31:*.zoo=00;31"
LS_COLORS="${LS_COLORS}:*.jpg=01;35:*.gif=01;35:*.bmp=01;35"
LS_COLORS="${LS_COLORS}:*.xbm=01;35:*.xpm=01;35:*.tif=01;35"
LS_COLORS="${LS_COLORS}:*.c=00;34:*.h=00;34"

ICAL_LIBRARY=/usr/local/lib/ical/v2.0p2

# netscape 4.73 (06/15/00  09:08 AM)
export MOZILLA_HOME=/auto/insbu-cnstools/netscape

# netshow 2.9 (10/04/99  08:08 AM)
#export NETSHOW_HOME_DIR=/usr/cns/netshow

export PAGER MORE LESS EDITOR PHONES REMOTE HELPDIR TORSO FCODES
export TGIF_LOGIC TGIF_FLOWCHART TGIF_PETRI TGIF_LATEX GUI_BUILDER
export LS_COLORS FTPR_PASSWORD ICAL_LIBRARY

export TZ=PST8PDT

# Java stuff - for j2sdk-1.3.0.02 (04/07/01  08:59 PM)
CLASSPATH=/auto/insbu-cnstools/java:/auto/insbu-cnstools/java/lib
CLASSPATH=$CLASSPATH:/auto/insbu-cnstools/java/jre/lib
export CLASSPATH

# turn off zle for emacs (04/24/01  02:48 PM)
if [[ $EMACS == "t" ]]; then
	unsetopt zle
	fi

# ssh setup (09/14/99  07:38 AM)
# To allow no password ssh and scp
if [[ -r ~/.ssh/authorized_keys && $(hash -m ssh-agent) != "" ]]; then
	# run ssh-agent once for each engineer''s first login
	/bin/ps -U $(id | sed -n 's/uid=//;s/(.*$//p') | \
		grep ssh-agen > /dev/null
	if [[ $? -ne 0 ]]; then
		racefile=~/.ssh/ssh-agent.$HOST.tmp
		echo $pid >> $racefile
		if [[ $(head -1 $racefile) -eq $pid ]]; then
			# I am the first one
			ssh-agent -s > ~/.ssh/ssh-agent.$HOST
			(sleep 5; rm -f $racefile) &
			fi
		unset racefile
		fi
	if [[ -r ~/.ssh/ssh-agent.$HOST ]]; then
		. ~/.ssh/ssh-agent.$HOST
		fi
	fi
#---------------------------------------------------------------#
#		  Project Environment Variables			#
#---------------------------------------------------------------#
# PAW stuff
#export CVSROOT=/nfs/paw/src
#export PAWDEV=~lew/paw/u-world.CUR
#
# CSS ODE stuff
#export ODE_VER=ode4.0
#export PRINCIPAL=Paul_Lew
#export ODE_DIFF=ode-wdiff
#export ODE_VDIFF=ode-ediff
#
# Cisco Clearcase
export CC_PREPEND_PATH=$artool/sh:/auto/insbu-cnstools/bin
export CC_DISABLE_COPYRIGHT_CHECK=1
if [[ "$CLEARCASE_ROOT" != "" ]]; then
	export DOT_TAGS=/vob/cnsar/thin/destiny/.tags
	fi

# Printer Setup
# 11/06/00  10:44 AM
export LPDEST=sj20-b9
# printer spool directory for StarOffice
export STAR_SPOOL_DIR=/tmp

# export DISPLAY=${DISPLAY:=$LOGNAME-ultra:0.0}
#---------------------------------------------------------------#
#		source private zshenv setup			#
#---------------------------------------------------------------#
[[ -r $HOME/.zshenv-private ]] && . $HOME/.zshenv-private

## Local Variables:	#
## mode: ksh		#
## End:			#

PATCH: Re: architecture independent wordcode files (.zwc)

[Sven Wischnowsky]

Paul Lew wrote:

> ...
> 
> I have lots of startup files for zsh, fortunately, the .zshenv is good
> enough to trigger the problem.  Attached below.

That helped.

Obviously I had only tried it with functions that were too small.

The zwc files contain the compiled versions for both endianesses, first
in the one for the machine the file was compiled on and after that the
one for the other endianess. When the shell has to use that other
version (with mapping), it has to calculate the offset into the zwc file
for the header (containing the directory of files/functions in the zwc
file including their positions relative to the header etc.). That worked
fine. But it also has to map the part of the part of the zwc it needs
and for that it has to adjust the offset to a page boundary. That worked
fine, too. But then it forgot to increment the length of the mapped
reagion by the difference between the real offset and the aligned one.
For large functions that happened to cross a page boundary that could
result in the string table of the function being not or only partially
mapped.

> ...
> 	dirs=(.)
> 	dirs=($dirs $artool/$bindir $artool/sh $artool/perl)
> 	dirs=($dirs /auto/insbu-cnstools/$bindir)		#cisco CNS
> 	dirs=($dirs /auto/insbu-cnstools/s$bindir)		#cisco CNS
>       ...

Ugh, that's pretty inefficient -- zsh has no problems with arrays
definitions spanning multiple lines.

Bye
  Sven

Index: Src/parse.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/parse.c,v
retrieving revision 1.19
diff -u -r1.19 parse.c
--- Src/parse.c	2001/03/07 12:58:41	1.19
+++ Src/parse.c	2001/05/15 08:25:50
@@ -2782,7 +2782,7 @@
 {
     FuncDump d;
     Wordcode addr;
-    int fd, off;
+    int fd, off, mlen;
 
     if (other) {
 	static size_t pgsz = 0;
@@ -2802,15 +2802,17 @@
 	    pgsz--;
 	}
 	off = len & ~pgsz;
-    } else
+        mlen = len + (len - off);
+    } else {
 	off = 0;
-
+        mlen = len;
+    }
     if ((fd = open(dump, O_RDONLY)) < 0)
 	return;
 
     fd = movefd(fd);
 
-    if ((addr = (Wordcode) mmap(NULL, len, PROT_READ, MAP_SHARED, fd, off)) ==
+    if ((addr = (Wordcode) mmap(NULL, mlen, PROT_READ, MAP_SHARED, fd, off)) ==
 	((Wordcode) -1)) {
 	close(fd);
 	return;

-- 
Sven Wischnowsky                         wischnow@informatik.hu-berlin.de