modules!??

modules!??

[Wez Furlong]

Right, here we go...

I have zsh3.1.2-beta on two systems (IRIX 5.3 and SunOS 4.1.3).

On IRIX zsh 'goes away' when it loads the comp1 module on startup. If it
can't find the comp1 module, it carries on without zle or compctl.

As a result I am using the non-dynamic version for IRIX.

Under SunOS, the modules work fine, except:

When you explicitly zmodload comp1:

	ld.so: Undefined symbol: _fallback_compctlread

The shell then exits.

also,

	% zmodload deltochar
	ld.so: Undefined symbol: _zmod

AFAICT, comp1 is not zmodloaded by zsh under SunOs on startup: you can
unload and load the zle and compctl modules as much as you like.

What I don't understand is what is going on! The man page for the
modules says that 

      THE COMP1 MODULE
          The comp1 module does nothing that is visible to the user.
          Its purpose is to provide the internal basis of the
          programmable completion mechanism.  It must be loaded before
          any module that provides a means of controlling completion
          (such as the compctl module), or that uses completions (such
          as the zle module).  This is done automatically for modules
          distributed with zsh, and for other modules can be effected
          by the use of zmodload -d.

It doesn't appear to follow this behaviour for SunOS, and seg-faults, as
is correct(?!) under IRIX....

Can anyone shed some light on this? I didn't expect the modules to work
first time on IRIX 5.3 (in fact, they all do except for the comp1
dependents), and the strange behaviour of the SunOs version is...
strange.

TIA




-- 
Wez - Electronics Undergraduate at the University of York
URL : http://www.twinklestar.demon.co.uk/

Insult Of The Day: Thou mangled plume-plucked malt-worm!

Re: modules!??

[Zoltan Hidvegi]

> On IRIX zsh 'goes away' when it loads the comp1 module on startup. If it
> can't find the comp1 module, it carries on without zle or compctl.
> 
> As a result I am using the non-dynamic version for IRIX.
> 
> Under SunOS, the modules work fine, except:
> 
> When you explicitly zmodload comp1:
> 
> 	ld.so: Undefined symbol: _fallback_compctlread
> 
> The shell then exits.
> 
> also,
> 
> 	% zmodload deltochar
> 	ld.so: Undefined symbol: _zmod

Perhaps I should have mentioned this in the readme.  On some systems
modules cannot export symbols to other modules, so the comp1 module has to
be compiled into the main binary.  configure autodetects these systems, and
if you do not create modules-bltin manually, make will create it with comp1
which forces comp1 into the main binary.

But really you should never load comp1 manually.  Zsh knows that zle needs
comp1 and if you try to load zle it will load comp1 first if it doesn't
already have it.  Perhaps the output of zmodload should include the
statically linked modules as well (with some indication that this module is
static).  This would give the user a more clear picture.  The IRIX crash is
probably caused by the duplication of the comp1 module.

Deltochar have a similar problem, it wants to use the zmod symbol exported
by the zle module.  The symbols defined by a module can be discovered using
dlsym even on these systems, so we may find some way to overcome this
limitation.  I think it is possible to do that with some script-generated C
code.  For that we also need a list of exported and imported function for
each module and for the main binary, which is also needed for dynamic
loading on AIX.  The imported function list is hard to portably generate
automatically, so these lists will probably have to be maintained by hand
(with the help of a script, which can generate this list automatically
using nm, ld or some other tool on an OS I have access to).

Zoltan

Re: modules!??

[Zefram]

Zoltan Hidvegi wrote:
>                  Perhaps the output of zmodload should include the
>statically linked modules as well (with some indication that this module is
>static).

Probably a good idea.  It should refuse to load the redundant dynamic
module, and refuse to unload the static module.

>Deltochar have a similar problem, it wants to use the zmod symbol exported
>by the zle module.  The symbols defined by a module can be discovered using
>dlsym even on these systems, so we may find some way to overcome this
>limitation.  I think it is possible to do that with some script-generated C

How?  Would this be a case of #defining all imported symbols to
(*symbol_table[SYMBOL_NUM]) and having the module base manage home-made
symbol tables?  If so, what do we do about systems where there are
several sizes of pointer?

>                 The imported function list is hard to portably generate
>automatically,

What's wrong with `nm obj.o | grep " U "` and comparing against the
lists of exported symbols?  This could even be used to generate module
dependencies automatically.

-zefram

Re: modules!??

[Zoltan T. Hidvegi]

Zefram wrote:
> >Deltochar have a similar problem, it wants to use the zmod symbol exported
> >by the zle module.  The symbols defined by a module can be discovered using
> >dlsym even on these systems, so we may find some way to overcome this
> >limitation.  I think it is possible to do that with some script-generated C
>
> How?  Would this be a case of #defining all imported symbols to
> (*symbol_table[SYMBOL_NUM]) and having the module base manage home-made
> symbol tables?  If so, what do we do about systems where there are
> several sizes of pointer?

Yes, I was thingking something like that.  I do not understand what you
mean about several sizes of pointer.  Could you give some more detail?

>
> >                 The imported function list is hard to portably generate
> >automatically,
>
> What's wrong with `nm obj.o | grep " U "` and comparing against the
> lists of exported symbols?  This could even be used to generate module
> dependencies automatically.

The nm output format varies from system to system, so it's probably hard to
make it portable.  And one should not have to have nm to compile zsh.  But
if we can automatize everything on Linux I'd be happy.

Zoltan

Re: modules!??

[Zefram]

Zoltan T. Hidvegi wrote:
>Yes, I was thingking something like that.  I do not understand what you
>mean about several sizes of pointer.  Could you give some more detail?

The C standard permits most pointer types to have different
representations and sizes.  In particular, function pointers and data
pointers are not even required to be losslessly convertible.  So if we
were to make a symbol table store void *s, it might not be possible to
extract a working function pointer from it, and vice versa.  However, on
second thoughts, the dlsym() interface assumes that all pointers can
be converted to void * anyway, so I suppose it wouldn't be a problem to
assume that in our symbol tables too.

>The nm output format varies from system to system, so it's probably hard to
>make it portable.  And one should not have to have nm to compile zsh.  But
>if we can automatize everything on Linux I'd be happy.

There is one format for nm that is overwhelmingly common (BSD in origin,
perhaps?).  A script could detect whether the local nm conforms to this
convention, and if so will have the capability to generate these symbol
lists.  (As the GNU nm uses the popular format, I expect this will include
all of our usual development systems.)  In case there isn't an appropriate
nm around, we'd have to include the symbol lists in the distribution.

More thoughts...

We don't need to know the exact list of symbols imported by each module.
If we know which *modules* it references, all symbols in that module can
be turned into #defined symbol table references, and the module boot
code only needs to get its hands on those particular symbol tables (a
matter of looking throught the module table).  There is no overhead for
#defining a symbol that is not actually referenced.  Since the module
reference tree is small and doesn't often change, it can continue to be
maintained by hand.  We therefore only need to automatically generate
the *ex*ported symbol lists.

We already have a first approximation to the exported symbol lists in
the *.pro files.  If we extend these to include data objects (i.e.,
put data objects in their logical places and automatically generate
extern declarations -- globals.h disappears), the *.pro files would be
almost perfect symbol lists, without ever having to touch nm, or even
compile anything.  The icing on the cake would be for makepro.sh to pass
#ifdef lines through to the .pro file, to cope with symbols that only
exist conditionally.

(Oh yes, I really like the things you are making makepro.sh do in 3.1.2.
It's a much neater solution to the varying name problem than I had.)

I should have some time to play around with these ideas this week, having
just finished my exams.

-zefram

Re: modules!??

[Zoltan T. Hidvegi]

Zefram wrote:
> >The nm output format varies from system to system, so it's probably hard to
> >make it portable.  And one should not have to have nm to compile zsh.  But
> >if we can automatize everything on Linux I'd be happy.
>
> There is one format for nm that is overwhelmingly common (BSD in origin,
> perhaps?).  A script could detect whether the local nm conforms to this
> convention, and if so will have the capability to generate these symbol
> lists.  (As the GNU nm uses the popular format, I expect this will include
> all of our usual development systems.)  In case there isn't an appropriate
> nm around, we'd have to include the symbol lists in the distribution.

Of course AIX is always an exception )-:.

Zoltan

Re: modules!??

[Zoltan T. Hidvegi]

Zefram wrote:
> We don't need to know the exact list of symbols imported by each module.
> If we know which *modules* it references, all symbols in that module can

Of course AIX is an exception here too, as it always needs the exported
symbols list.  AIX can import symbols from the main binary as well as from
other modules without ifdefs, but each module and the main binary must have
an export list which has to be given to the linker whan a module imports
some symbols from that list.

Zoltan

Re: modules!??

[Stefan Monnier]

Zefram <zefram@dcs.warwick.ac.uk> writes:
> extract a working function pointer from it, and vice versa.  However, on
> second thoughts, the dlsym() interface assumes that all pointers can
> be converted to void * anyway, so I suppose it wouldn't be a problem to

IIRC, the ANSI C standard allows different sizes for different kinds of
pointers but requires that void* is big enough to hold any pointer (so that
casting to void* and back is a safe operation). So it's not just the dlsym
interface.


        Stefan

Re: modules!??

[Zefram]

Stefan Monnier wrote:
>IIRC, the ANSI C standard allows different sizes for different kinds of
>pointers but requires that void* is big enough to hold any pointer (so that
>casting to void* and back is a safe operation). So it's not just the dlsym
>interface.

That's only for data pointers.  Function pointers are not guaranteed to be
losslessly convertible to void *, nor vice versa.  Function pointer types
are all guaranteed to be losslessly convertible to each other, however.

-zefram

Re: modules!??

[Peter Stephenson]

Wez Furlong wrote:
> : The IRIX crash is probably caused by the duplication of the comp1 module.
> 
> The IRIX version loads the comp1 module on startup itself; this leads me
> to believe that comp1 is not builtin to the IRIX binary. When it
> successfully loads the comp1.so, it seg-faults; without it it carries on
> happily.

The problem I was seeing on IRIX, if it helps, was that comp1 loaded OK,
then there was a silent crash when attempting to load zle on top, I
think in the boot_ function.  I got around it by adding both comp1 and zle
to modules.bltin and recompiling and everything has been OK with dynamic
loading since then.  As Zoltan pointed out, adding zle should only really
be necessary if you have ZLE add-on modules, of which deltochar is currently
the only one (and will soon be implementable as a function if it isn't
already).

-- 
Peter Stephenson <pws@ifh.de>       Tel: +49 33762 77366
WWW:  http://www.ifh.de/~pws/       Fax: +49 33762 77413
Deutsches Elektronen-Synchrotron --- Institut fuer Hochenergiephysik Zeuthen
DESY-IfH, 15735 Zeuthen, Germany.

Re: modules!??

[Wez Furlong]

On Jun 5,  1:45am, Zoltan Hidvegi wrote:
: [I wrote]
: > On IRIX zsh 'goes away' when it loads the comp1 module on startup. If it
: > can't find the comp1 module, it carries on without zle or compctl.

: > Under SunOS, the modules work fine, except:
: > When you explicitly zmodload comp1:
: > 	ld.so: Undefined symbol: _fallback_compctlread
: > The shell then exits.
: > also,
: > 	% zmodload deltochar
: > 	ld.so: Undefined symbol: _zmod
: 
: Perhaps I should have mentioned this in the readme.  On some systems
: modules cannot export symbols to other modules, so the comp1 module has to
: be compiled into the main binary.  configure autodetects these systems, and
: if you do not create modules-bltin manually, make will create it with comp1
: which forces comp1 into the main binary.

Is it safe to remove the redundant comp1.so that is created in this
case? Also, is it possible to have the Makefile not install it if its
not needed?

: But really you should never load comp1 manually.  Zsh knows that zle needs
: comp1 and if you try to load zle it will load comp1 first if it doesn't
: already have it.

I was only explicitly loading it in an attempt to narrow down the cause
of the problem.

: Perhaps the output of zmodload should include the
: statically linked modules as well (with some indication that this module is
: static).  This would give the user a more clear picture.

That would be a good thing :)

: The IRIX crash is probably caused by the duplication of the comp1 module.

The IRIX version loads the comp1 module on startup itself; this leads me
to believe that comp1 is not builtin to the IRIX binary. When it
successfully loads the comp1.so, it seg-faults; without it it carries on
happily.

: Deltochar have a similar problem, it wants to use the zmod symbol exported
: by the zle module.  The symbols defined by a module can be discovered using
: dlsym even on these systems, so we may find some way to overcome this
: limitation.  I think it is possible to do that with some script-generated C
: code.  For that we also need a list of exported and imported function for
: each module and for the main binary, which is also needed for dynamic
: loading on AIX.  The imported function list is hard to portably generate
: automatically, so these lists will probably have to be maintained by hand
: (with the help of a script, which can generate this list automatically
: using nm, ld or some other tool on an OS I have access to).

I hope it all comes together soon!

Cheers,



-- 
Wez - Electronics Undergraduate at the University of York
URL : http://www.twinklestar.demon.co.uk/

Insult Of The Day: Thou vain spur-galled foot-licker!