Module hierarchies, aliasing, and dependencies

Module hierarchies, aliasing, and dependencies

[Bart Schaefer]

At present, the name used to load a module is the only indication zsh has
of whether a particular module is has been loaded.  This is used both by
the dependency system and by the zmodload command.  So for example if one
does

	module_path=($module_path $^module_path/zsh)
	zmodload zleparameter

one has now loaded zsh/zleparameter, but zsh doesn't know it; and loading
it again as zsh/zleparameter fails (because the widgets variable is read-
only).  Even modules that can be successfully loaded more than once give
lots of error messages like:

zsh/zle: name clash when adding hook `list_matches'
zsh/zle: name clash when adding hook `complete'
zsh/zle: name clash when adding hook `before_complete'
zsh/zle: name clash when adding hook `after_complete'
zsh/zle: name clash when adding hook `accept_completion'
zsh/zle: name clash when adding hook `reverse_menu'
zsh/zle: name clash when adding hook `invalidate_list'

Of course this can be written off as misuse of zmodload and $module_path.
However, suppose I have my own locally-hacked version of the "files"
module (adds chmod and mkfifo, or some such).  I'd like to make both
this module and the distributed zsh/files module available for loading.
If I put it in the default module path, under a different hierarchy --
say, /usr/local/lib/zsh/3.1.9/local/files -- then it can be loaded with
"zmodload local/files", but it won't satisy dependencies on "zsh/files".
I could put my local one in a completely different directory in the
module_path -- /usr/local/lib/zsh/local, or something -- still under the
name zsh/files, so that it satisfies the dependencies; but then there's
no way to load some local modules and other standard modules without
changing the value of $module_path before every zmodload.

Wouldn't it be better to rely on a symbol that's defined *inside* the
module to determine whether a module has been loaded?  Similar to the way
(provides 'name) works in emacs libraries?

One way to approach this would be to add another function, like the boot
and cleanup functions that already exist, which is called when searching
the list of modules.  We could define this function to return a string
that is compared to the module being searched for; or we could pass the
string being searched for into the function and let it return a truth
value.  If the function isn't defined by a given module, then compare the
name strings as is done now.

Does anyone foresee any serious problems with such an approach?  Is there
a better way?  Is there disagreement over whether this is worth doing?

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: Module hierarchies, aliasing, and dependencies

[Peter Stephenson]

"Bart Schaefer" wrote:
> Wouldn't it be better to rely on a symbol that's defined *inside* the
> module to determine whether a module has been loaded?  Similar to the way
> (provides 'name) works in emacs libraries?

Yes, this sounds better.  That would have fixed [some of] Will's
difficulties.  Also, I've been wondering for a while how to provide a test
version of a module under a different name.  This would be a good solution.

> One way to approach this would be to add another function, like the boot
> and cleanup functions that already exist, which is called when searching
> the list of modules.  We could define this function to return a string
> that is compared to the module being searched for; or we could pass the
> string being searched for into the function and let it return a truth
> value.  If the function isn't defined by a given module, then compare the
> name strings as is done now.

Given the way modules work, a function-based system is presumably the right
one.

-- 
Peter Stephenson <pws@pwstephenson.fsnet.co.uk>
Work: pws@csr.com
Web: http://www.pwstephenson.fsnet.co.uk

Re: Module hierarchies, aliasing, and dependencies

[Zefram]

Peter Stephenson wrote:
>"Bart Schaefer" wrote:
>> Wouldn't it be better to rely on a symbol that's defined *inside* the
>> module to determine whether a module has been loaded?  Similar to the way
>> (provides 'name) works in emacs libraries?
>
>Yes, this sounds better.  That would have fixed [some of] Will's
>difficulties.  Also, I've been wondering for a while how to provide a test
>version of a module under a different name.  This would be a good solution.

There's an issue here, that a module that calls functions from another
module can't be loaded at the dlopen() level (depending on system).
The module dependency system is there to handle this, but it will
have problems with the kind of scheme you describe.  For example,
zsh/deltochar depends on zsh/zle: with dependencies set up right,
zsh/zle will be loaded automatically before zsh/deltochar.  But if you
load zsh/deltochar under the name "local/deltochar" (or just "deltochar",
or anything else) zsh/zle won't be loaded first, and so the load will fail
(unless zsh/zle was already loaded).

There's another issue that on some systems name clashes between symbols
exported by different modules isn't permitted, so every module's boot
function has to be named differently, so we need to know the module's
internal name in order to call any code in it.  Loading under a different
name just won't work on that kind of system (as I mentioned upthread).

We're stuck with this external naming in many respects, at least until we
start putting dependency-type information in the module files themselves,
in a form that can be read without doing a dlopen().  This is something
I've considered before, but not implemented because of concerns about
what various dynamic linkers might accept.  But if we could do it, we
could include in this metadata the name of the module's boot function,
making it possible to load a module under any name on any system.

If we can relax the connection between modules' internal names and disk
names in this way, it would then be feasible to introduce a distinction
between the module file and the logical modules in contains.  We already
have a special-case relaxation of the one-to-one correspondence --
modules linked in to the main executable -- and I can see advantages in
fully generalising it.

-zefram

Re: Module hierarchies, aliasing, and dependencies

[Bart Schaefer]

On Sep 10, 10:35pm, Zefram wrote:
} Subject: Re: Module hierarchies, aliasing, and dependencies
}
} [E.g.] zsh/deltochar depends on zsh/zle: with dependencies set up right,
} zsh/zle will be loaded automatically before zsh/deltochar.  But if you
} load zsh/deltochar under the name "local/deltochar" (or just "deltochar",
} or anything else) zsh/zle won't be loaded first, and so the load will fail
} (unless zsh/zle was already loaded).

I don't think this is really a problem -- the issue is not automatically
discovering that local/deltochar depends on zsh/zle, it's discovering that
local/zle has already substituted for zsh/zle when loading zsh/deltochar.

Put another way, I'm quite happy to have to explicitly state dependency
of local/deltochar on zsh/zle, external to any module.  However, if the
user explicitly loads local/zle, I want zsh to discover that it does not
also need to load zsh/zle at the time zsh/deltochar is loaded.

An alternate solution has just occurred to me ... in the setup function in
local/zle, one could simply scan the module list for any module that has a
dependency on zsh/zle, and substitute the name local/zle in its place.
However, that would not help if `zmodload -d ... zsh/zle' was used after
the local/zle setup function was called, so some additional magic would
also be needed.

} There's another issue that on some systems name clashes between symbols
} exported by different modules isn't permitted, so every module's boot
} function has to be named differently, so we need to know the module's
} internal name in order to call any code in it.  Loading under a different
} name just won't work on that kind of system (as I mentioned upthread).

Under what circumstances would it be necessary to call any code without
knowing the actual name of the shared object?

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

RE: Module hierarchies, aliasing, and dependencies

[Andrej Borsenkow]

>
> On Sep 10, 10:35pm, Zefram wrote:
> } Subject: Re: Module hierarchies, aliasing, and dependencies
> }
> } [E.g.] zsh/deltochar depends on zsh/zle: with dependencies set up right,
> } zsh/zle will be loaded automatically before zsh/deltochar.  But if you
> } load zsh/deltochar under the name "local/deltochar" (or just "deltochar",
> } or anything else) zsh/zle won't be loaded first, and so the load will fail
> } (unless zsh/zle was already loaded).
>
> I don't think this is really a problem -- the issue is not automatically
> discovering that local/deltochar depends on zsh/zle, it's discovering that
> local/zle has already substituted for zsh/zle when loading zsh/deltochar.
>
> Put another way, I'm quite happy to have to explicitly state dependency
> of local/deltochar on zsh/zle, external to any module.  However, if the
> user explicitly loads local/zle, I want zsh to discover that it does not
> also need to load zsh/zle at the time zsh/deltochar is loaded.
>

It looks, like both posters still assume that "file name" == "module name". I
do not see why it must be so. Module dependencies are stated in term of
"module" not "file". So, if I have my own  *module* zsh/deltochar (or zsh/zle)
that happen to reside in *file* /what/ever/local/deltochar.so or
/some/other/path/zle.so then I just need some way to say

zmodload -f /path/to/my/module.so zsh/zle

Of course, zmodload must check, that /path/to/my/module.so really defines
zsh/zle, but after that no changes are really needed to current module system
(but see below). So, what is needed

- add some module identification (this may be one more entry point - it is
probably the only way to do it, unless we want to reimplement runtime loader).

- separate module name and file name. Hey! we could even implement files that
define more than one module! :-)

> An alternate solution has just occurred to me ... in the setup function in
> local/zle, one could simply scan the module list for any module that has a
> dependency on zsh/zle, and substitute the name local/zle in its place.
> However, that would not help if `zmodload -d ... zsh/zle' was used after
> the local/zle setup function was called, so some additional magic would
> also be needed.
>

Unless I'm mistaken, this is completely redundant. The initial wish was to
load the *same* module from different place, not to load module under
different name?

> } There's another issue that on some systems name clashes between symbols
> } exported by different modules isn't permitted, so every module's boot
> } function has to be named differently, so we need to know the module's
> } internal name in order to call any code in it.  Loading under a different
> } name just won't work on that kind of system (as I mentioned upthread).
>
> Under what circumstances would it be necessary to call any code without
> knowing the actual name of the shared object?
>

The unfortunate problem is, that we may *need* the name of shared object. The
current state is, that modules have many cross-references. This works under
most modern Unix with ELF, because run-time linker (normally) does not care
where symbol is located (normally, because I have options to bind symbols to
specific objects at compile time). It means, that after
/what/ever/local/myzle.so was loaded it will satisfy implicit request for any
symbol from zle module.

But we have to deal with Cygwin. Import list for any module includes names off
DLL from which symbols are imported. And I got impression (still have not
confirmed), that this name is the same as DLL file name (at least, I was not
able to change it. It may be a problem of Cygwin tools). It means, that it may
not be possible to substitute zle with myzle at all.

The general solution is to get rid of all implicit cross references and use
only dlsym()'d objects. This will correctly work under Cygwin but it means
substantial rewrite (it may be automated. For every module get list of all
mod_exported symbols from another modules and put dlsym() calls for them into
setup function. Of course, we need some dlsym() wrapper to get real file name
from module name. Actually, sounds quite reasonable :-)

To summarize:

- I agree with one additional entry point for module name (it may just as well
be a simple string)
- we must distinguish between module name and file name. I suggest extending
zmodload to allow optional file name. It must check, of course, that this file
really defines requested module.
- all internal dependencies are in terms of mode names. After module has been
loaded, it is known under it's module name, and it's file name is of small (or
no) interest at all. That covers both Zefram and Bart's arguments.
- well, Cygwin case remains ... any ideas? Anybody knows internals of
Microsoft run-time linker?

-andrej

Re: Module hierarchies, aliasing, and dependencies

[Bart Schaefer]

On Sep 14,  7:18pm, Andrej Borsenkow wrote:
}
} > On Sep 10, 10:35pm, Zefram wrote:
} > }
} > } [E.g.] zsh/deltochar depends on zsh/zle: with dependencies set up
} > } right, zsh/zle will be loaded automatically before zsh/deltochar.
} > } But if you load zsh/deltochar under the name "local/deltochar" (or
} > } just "deltochar", or anything else) zsh/zle won't be loaded first,
} > } and so the load will fail (unless zsh/zle was already loaded).
} >
} > I don't think this is really a problem -- the issue is not
} > automatically discovering that local/deltochar depends on zsh/zle,
} > it's discovering that local/zle has already substituted for zsh/zle
} > when loading zsh/deltochar.
}
} It looks, like both posters still assume that "file name" == "module
} name".  I do not see why it must be so.

It was not my intention to imply that "file name" == "module name" --
though I think the problems you point out with Cygwin may be a reason to
keep the ${filename:t} == ${modulename:t} behavior for at least a while
yet.

It *was* my assumption that the file name relative to $module_path would
be the single argument to zmodload when loading the module, but of course
there's no particular reason to assume that, either.

} zmodload -f /path/to/my/module.so zsh/zle

That would be fine, except that it requires you to know the full path
to the module and the shared object extension, so it won't work in a
cross-platform way.  How about instead something like

zmodload -N zsh/zle my/module

or perhaps

zmodload -N zsh/zle=my/module

which would mean the same as your suggestion, except that /path/to would
have to be added to $module_path.  (-N for "name as", roughly.)

This could interact with the alias mechanism, so that if you wrote:

zmodload -A zle=zsh/zle
zmodload -N zsh/zle=my/module
zmodload zle

Then it would load the file $^module_path/my/module(.so)(N) and enter the
string "zsh/zle" in the list of loaded modules.

} Of course, zmodload must check, that /path/to/my/module.so really
} defines zsh/zle, but after that no changes are really needed to
} current module system (but see below). So, what is needed
}
} - add some module identification (this may be one more entry point -
} it is probably the only way to do it, unless we want to reimplement
} runtime loader).
} 
} - separate module name and file name. Hey! we could even implement
} files that define more than one module! :-)

I suggest that, for the first pass, the "module identification" can be
limited to the base name of the shared object; unless I misunderstand
what you said about Windows DLLs, it won't work to load two DLLs with
the same base name anyway (at least not without a major rewrite, and
maybe not even then).

This means that we don't entirely separate the module name and the file
name, but we at least separate the module name from its load path.

} > An alternate solution has just occurred to me ... in the setup function in
} > local/zle, one could simply scan the module list for any module that has a
} > dependency on zsh/zle, and substitute the name local/zle in its place.
} 
} Unless I'm mistaken, this is completely redundant. The initial wish was to
} load the *same* module from different place, not to load module under
} different name?

The initial wish was to load the module from a different place and have
the "same name" part be automatically computed after the fact.  I've no
problem with giving up the automatic part for a simpler implementation.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

RE: Module hierarchies, aliasing, and dependencies

[Andrej Borsenkow]

>
> It *was* my assumption that the file name relative to $module_path would
> be the single argument to zmodload when loading the module, but of course
> there's no particular reason to assume that, either.
>
> } zmodload -f /path/to/my/module.so zsh/zle
>
> That would be fine, except that it requires you to know the full path
> to the module and the shared object extension, so it won't work in a
> cross-platform way.  How about instead something like
>

It depends on usage pattern. I think about it as developers aid in the first
place. In this case you definitely know path name and extension :-)

But nothing prevents relative pathnames like

zmodload -f relative/path zsh/zle

that would be interpreted in usual way (search through module_path adding
system-specific extension).

>
> I suggest that, for the first pass, the "module identification" can be
> limited to the base name of the shared object; unless I misunderstand
> what you said about Windows DLLs, it won't work to load two DLLs with
> the same base name anyway (at least not without a major rewrite, and
> maybe not even then).
>

The worst case is that currently even zsh/foo and bar/foo may clash. But this
happens to my best knowledge only if we use implicit linking (I believe,
Microsoft calls it compile-time). We can load two DLLs with the same base
names using dlopen() and dlsym() will work correctly. IIRC :-) I must check.
What may not work, if zsh/bar depends on (is linked with) zsh/foo and my/bar
depends on my/foo. I am afraid, that Windows will search only one of */foo
DLLs even if we explicitly load both.


-andrej

RE: Module hierarchies, aliasing, and dependencies

[Andrej Borsenkow]

>
> It *was* my assumption that the file name relative to $module_path would
> be the single argument to zmodload when loading the module, but of course
> there's no particular reason to assume that, either.
>
> } zmodload -f /path/to/my/module.so zsh/zle
>
> That would be fine, except that it requires you to know the full path
> to the module and the shared object extension, so it won't work in a
> cross-platform way.  How about instead something like
>
> zmodload -N zsh/zle my/module
>

Thinking more about it, I was wrong. The problem is linked-in modules. We
obviously cannot "overload" them with external file. It means, that the above
command must fail with some error message. This creates ugly special case. The
same applies to your idea.

If we use just module name + path search, it works nicely for linked-in
modules as well. Just assume (and document) that linked-in modules are
implicitly prepended to module_path and so always "found" first.

-andrej