Getting dynamic loading to work on cygwin

Getting dynamic loading to work on cygwin

[Peter Stephenson]

[-- Attachment #1: Type: text/plain, Size: 1139 bytes --]

In case anyone has a chance to play around with it, here's a minimal
example of getting dynamically loaded libraries to work on cygwin.  The
changes to what we have already should be fairly mechanical but will still
require a bit of work.  Zefram's and Oliver's work on export files will
come in handy, since it looks like cygwin will need a very similar set to
AIX (there are tools to generate them automatically, but we have enough
in place already to get it right).  Just unpacking these and running make
should create a main.exe and a dl.dll; running the first loads the second.
I actually found out about the dllwrap incantation (which does all the hard
work for you) from the perl dynamic loading configuration.

There may be some issues about search paths still to resolve, since there
are both .a (list of exports) and a .dll (the actual library) files; I
haven't tried moving the files into different directories.  It might also
be more efficient to use dlltool directly rather than rely on dllwrap, but
I for one don't have the patience.

In case anyone is any doubt, this will *not* be in the release which is
immediately due.


[-- Attachment #2: Makefile --]
[-- Type: text/plain, Size: 392 bytes --]

.SUFFIXES: .so .dll

CC = gcc
CFLAGS = -O -g -Wall
LIBS = # -ldl

all: main dl.dll

main: main.o
	$(CC) -o main main.o $(LIBS)

dl.o: dl.c
	gcc $(CFLAGS) -c dl.c

dl.dll : dl.o
	dllwrap --dllname $*.dll --driver-name gcc --dlltool dlltool --as as --def $*.def --output-lib lib$*.a dl.o

dl.so : dl.o
	gcc -G -o dl.so dl.o

clean:
	rm -f dl.o dl.so dl.dll main.o main libdl.a main.exe dl.base

[-- Attachment #3: main.c --]
[-- Type: text/plain, Size: 453 bytes --]

#include <stdio.h>
#include <dlfcn.h>

int main(int argc, char **argv)
{
    void *handle, *symbol;
    int ret;

    handle = dlopen("./dl.dll", RTLD_LAZY);
    if (handle == NULL) {
	fprintf(stderr, "dlopen failed.\n");
	return 1;
    }
    symbol = dlsym(handle, "module");
    if (symbol == NULL) {
	fprintf(stderr, "dlsym failed.\n");
	return 1;
    }
    ret = ((int (*)(void))symbol)();

    printf("module returned %d\n", ret);

    return 0;
}

[-- Attachment #4: dl.c --]
[-- Type: text/plain, Size: 94 bytes --]

#include <stdio.h>

int module(void)
{
    printf("This is the module.\n");
    return 42;
}


[-- Attachment #5: dl.def --]
[-- Type: text/plain, Size: 16 bytes --]

EXPORTS
	module

Re: Getting dynamic loading to work on cygwin

[Peter Stephenson]

> The
> changes to what we have already should be fairly mechanical but will still
> require a bit of work.

Hmm, looks like I spoke too soon.  I don't see a way of exporting symbols
from the main executables which DLL's will recognize.  However, the old
system V trick of making the main body of the code a DLL and zsh.exe a
boot-loader ought to work, and isn't that difficult (again, the hard part
is really just fiddling the scripts etc.).

Another thing I missed before was that the lib*.a libraries created
with the .dll files are only needed at link time, and then only when
importing stuff from a dependency (such as zsh.dll if we do the trick
above); we only need to install the .dll files themselves.

Maybe someone knows better.

-- 
Peter Stephenson <pws@cambridgesiliconradio.com>
Cambridge Silicon Radio, Unit 300, Science Park, Milton Road,
Cambridge, CB4 0XL, UK                          Tel: +44 (0)1223 392070

Use and abuse of dynamic loading RE: Getting dynamic loading to work on cygwin

[Andrej Borsenkow]

This is not directly related to Cygwin; I just use the occasion.

What are plans for dynamic loading future in Zsh? This feature as it
currently is may be interesting, but ...

- it does not save disk space. Modules take up at least the same amount
of disk space (and even more due to symbol tables).

- it does not save memory at runtime. Most modern systems share code in
any case; actually, dynamic loading requires slightly more memory for
houskeeping (and then, this memory really per process)

- it is usually slower because of indirection.

So, the sole advantage IMHO is the ability to extend main program
on-the-fly. Consider Perl or Tcl where you compile and install modules
independently of main program. But this is exactly what Zsh lacks (just
think about autoloading - all information is compiled into main
executable, so if I want another module be autoloaded I need to
recompile zsh).

Sorry, if it obvious, but I thought I ask anyway.

-andrej

RE: Getting dynamic loading to work on cygwin

[Andrej Borsenkow]

>
> Hmm, looks like I spoke too soon.  I don't see a way of
> exporting symbols
> from the main executables which DLL's will recognize.
> However, the old
> system V trick of making the main body of the code a DLL and zsh.exe a
> boot-loader ought to work, and isn't that difficult (again,
> the hard part
> is really just fiddling the scripts etc.).
>


If I recall correctly, there is no way to export data references; only
subroutines entry points can be exported. I.e. one DLL cannot directly
reference variable in other DLL. But I may be wrong.

Is there any cross-module data reference in Zsh? Including main
exectuable?

-andrej

Re: Use and abuse of dynamic loading RE: Getting dynamic loading to work on cygwin

[Peter Stephenson]

> Consider Perl or Tcl where you compile and install modules
> independently of main program. But this is exactly what Zsh lacks (just
> think about autoloading - all information is compiled into main
> executable, so if I want another module be autoloaded I need to
> recompile zsh).

I don't know what you mean by this.  In perl, you use `use Module' to
include a module, which may have compiled data, and may have autoload
capability.  In zsh, you use `zmodload -ab foo' and its friends to specify
that foo is loadable from a module; you don't need to recompile to do that.
What's the difference?  Do you mean that zsh doesn't come with a separate
kit to allow you to compile new modules without the original compilation
environment?

-- 
Peter Stephenson <pws@cambridgesiliconradio.com>
Cambridge Silicon Radio, Unit 300, Science Park, Milton Road,
Cambridge, CB4 0XL, UK                          Tel: +44 (0)1223 392070

RE: Use and abuse of dynamic loading RE: Getting dynamic loading towork on cygwin

[Andrej Borsenkow]

>
> I don't know what you mean by this.  In perl, you use `use Module' to
> include a module, which may have compiled data, and may have autoload
> capability.  In zsh, you use `zmodload -ab foo' and its
> friends to specify
> that foo is loadable from a module; you don't need to
> recompile to do that.
> What's the difference?

Yes, sorry, I overlooked it. I mostly thought about these magic modules
that are always autoloaded, like zle and completion, and file
bltinmods.list. This one is generated at compile time.

Do you mean that zsh doesn't come
> with a separate
> kit to allow you to compile new modules without the original
> compilation
> environment?
>

Yes, that's it.

-andrej

Re: Getting dynamic loading to work on cygwin

[Peter Stephenson]

> If I recall correctly, there is no way to export data references; only
> subroutines entry points can be exported. I.e. one DLL cannot directly
> reference variable in other DLL. But I may be wrong.

Yuk, it looks like your right.  Using dlsym to get the symbol works OK, but
simply linking against the DLL and loading it doesn't seem to pick up the
right value.  That will screw up a lot of things.

-- 
Peter Stephenson <pws@cambridgesiliconradio.com>
Cambridge Silicon Radio, Unit 300, Science Park, Milton Road,
Cambridge, CB4 0XL, UK                          Tel: +44 (0)1223 392070

Re: Getting dynamic loading to work on cygwin

[maurice s. barnum]

Peter Stephenson <pws@cambridgesiliconradio.com> writes:

: > If I recall correctly, there is no way to export data references; only
: > subroutines entry points can be exported. I.e. one DLL cannot directly
: > reference variable in other DLL. But I may be wrong.
: 
: Yuk, it looks like your right.  Using dlsym to get the symbol works OK, but
: simply linking against the DLL and loading it doesn't seem to pick up the
: right value.  That will screw up a lot of things.

Data references can be imported on Win32, but you need help from the
compiler to generate an extra indirection on access to the imported data.
This of course implies that the compiler needs to know what data is
being imported, which with normal Win32 compilers is indicated by
decorating the extern declaration with '__import' or '__declspec(dllimport)'.
I don't have cygwin or it's documentation handy, so I don't know if
gcc supports this or some alternate spelling.

 --xmsb

Re: Getting dynamic loading to work on cygwin

[Peter Stephenson]

Maurice Barnum wrote:
> Data references can be imported on Win32, but you need help from the
> compiler to generate an extra indirection on access to the imported data.
> This of course implies that the compiler needs to know what data is
> being imported, which with normal Win32 compilers is indicated by
> decorating the extern declaration with '__import' or '__declspec(dllimport)'.
> I don't have cygwin or it's documentation handy, so I don't know if
> gcc supports this or some alternate spelling.

Aha!

extern int module_value __attribute__((__dllimport__));

This fixes things up.  Thanks, that's exactly what we needed know --- this
can (eventually) be automated by dumping the appropriate declarations to
the prototype files.

-- 
Peter Stephenson <pws@cambridgesiliconradio.com>
Cambridge Silicon Radio, Unit 300, Science Park, Milton Road,
Cambridge, CB4 0XL, UK                          Tel: +44 (0)1223 392070

Re: Use and abuse of dynamic loading RE: Getting dynamic loading towork on cygwin

[Fletch]

>>>>> "Andrej" == Andrej Borsenkow <Andrej.Borsenkow@mow.siemens.ru> writes:

[...]

    >> Do you mean that zsh doesn't come
    >> with a separate kit to allow you to compile new modules without
    >> the original compilation environment?
    >> 

    Andrej> Yes, that's it.

        Something along the lines of perl's ExtUtils::MakeMaker or
even the Gtk+/glib foo-config style utility would be very nice to
have.  Does zsh even install the headers necessary to build a module
outside the zsh source tree?

-- 
Fletch                | "If you find my answers frightening,       __`'/|
fletch@phydeaux.org   |  Vincent, you should cease askin'          \ o.O'
678 443-6239(w)       |  scary questions." -- Jules                =(___)=
                      |                                               U

RE: Getting dynamic loading to work on cygwin

[Andrej Borsenkow]

> :
> : Yuk, it looks like your right.  Using dlsym to get the
> symbol works OK, but
> : simply linking against the DLL and loading it doesn't seem
> to pick up the
> : right value.  That will screw up a lot of things.
>
> Data references can be imported on Win32, but you need help from the
> compiler to generate an extra indirection on access to the
> imported data.
> This of course implies that the compiler needs to know what data is
> being imported, which with normal Win32 compilers is indicated by
> decorating the extern declaration with '__import' or
> '__declspec(dllimport)'.
> I don't have cygwin or it's documentation handy, so I don't know if
> gcc supports this or some alternate spelling.
>

Cygwin supports __declspec(dllimport/export) but it different matter.
The following case works (foo.dll is assumed to access variable in
bar.dll)

build bar.dll that defines and exports variable
build foo.dll with bar stub library bar.a that directly references this
variable (extern ...)

the following case does not work

build bar.dll that defines and exports variable
build foo.dll that loads bar.dll with dlopen() and lokos up variable
with dlsym().

Variable is found but seems to point to the wrong location.

I did define variable with __declspec(dllexport) but it did not help.

-andrej

Re: Getting dynamic loading to work on cygwin

[Peter Stephenson]

> I did define variable with __declspec(dllexport) but it did not help.

I replied to this at the time.  __attribute__((dllimport)) (presumably
equivalent to __declspec) does get the value correctly imported with
dlopen.  (At least, check the archive and see if I did something different,
but I think it's the same that you're talking about.)

-- 
Peter Stephenson <pws@cambridgesiliconradio.com>
Cambridge Silicon Radio, Unit 300, Science Park, Milton Road,
Cambridge, CB4 0XL, UK                          Tel: +44 (0)1223 392070