Adding (compiling) a module to already build Zsh

Adding (compiling) a module to already build Zsh

[Sebastian Gniazdowski]

Hello,
on IRC someone asked how to "add gdbm". I wonder if the module can be
easily added to already installed Zsh, if it's not already bundled?
There should be something like Linux kernel-headers-*.(deb,rpm,etc.),
however, I think gdbm and pcre add linkage to additional libraries to
the Zsh binary itself. I guess there will be more obstacles, but who
knows, maybe it's possible? It would be then possible to do following:
bundle module source together with plugin (yeah, the current state of
art Oh-My-Zsh plugin architecture's) and build the module at first
load of the plugin.

Best regards,
Sebastian Gniazdowski

Re: Adding (compiling) a module to already build Zsh

[Bart Schaefer]

On Sep 12,  8:12am, Sebastian Gniazdowski wrote:
}
} on IRC someone asked how to "add gdbm". I wonder if the module can be
} easily added to already installed Zsh, if it's not already bundled?

It should be possible to do this provided that
(a) the zsh source matching the installed binary is available to
compile against and
(b) the installed binary was compiled with dynamic linkage enabled.

What won't work is any automatic autoloads of the builtins etc. from
any module added later.  I can't think of a reason explicit "zmodload" 
wouldn't work, though, assuming the .so file was in a $module_path
directory.

Re: Adding (compiling) a module to already build Zsh

[Sebastian Gniazdowski]

On 12 September 2016 at 19:38, Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Sep 12,  8:12am, Sebastian Gniazdowski wrote:
> }
> } on IRC someone asked how to "add gdbm". I wonder if the module can be
> } easily added to already installed Zsh, if it's not already bundled?
>
> It should be possible to do this provided that
> (a) the zsh source matching the installed binary is available to
> compile against and

Could be something like the commonly occurring package
kernel-headers-*.(deb|rpm|etc.) created, with mostly header files
constituting a build environment (source files are rather not needed?)
? It would be cool thing from one other reason. For example, there is
gdbm module for Zsh. Its features make it capable of rising Zsh code
to another level. However, it is rarely available in distributed Zsh
packages (licensing issue? I'm not sure if license issue doesn't apply
only to pcre module). However, with easily available build environment
one would be able to distribute a gdbm module together with his plugin
and this way overcome licensing issue.

Best regards,
Sebastian Gniazdowski

Re: Adding (compiling) a module to already build Zsh

[Oliver Kiddle]

Sebastian Gniazdowski wrote:
> > On Sep 12,  8:12am, Sebastian Gniazdowski wrote:

> > } I think gdbm and pcre add linkage to additional libraries to
> > } the Zsh binary itself.

You can usually prevent that with something like
  LDFLAGS='-Wl,--as-needed'
Depending on you linker, the option might be -zignore.

Or you can retrospectively remove the library from the NEEDED list with
a tool like patchelf.

The limitation is really with autoconf or at least our use of it.
We add checks for libraries such as pcre and it adds -lpcre to the
global list of linker options. I don't know whether anything has changed
in autoconf to make it easier to limit that so it would only be used
when linking the pcre module - anyone?

> Could be something like the commonly occurring package
> kernel-headers-*.(deb|rpm|etc.) created, with mostly header files
> constituting a build environment (source files are rather not needed?)

That would amount to a make target named something like install-headers
to dump .h files in $prefix/include/zsh.

Except we'd need to decide between dumping them all there or first
stripping them down to a minimum.

Many of the prototypes are autogenerated with makepro.awk and that
mechanism ties in to aspects of the module mechanism. So, for example, an
obvious function to include in the module interface might be addhookdefs
but the prototype is in module.epro. This may not be a problem as such
but it'd need someone who understands what's going on in that area.
Perhaps, we could have another tag like mod_export for things we want
to be part of the public interface. It would then need someone to go
through the internal interfaces of zsh and decide what should be included
in the external module interface. We may want only part of zsh.h
included.

It'd also be good if things were stuctured so that a module wouldn't
need changes in order to be moved into the zsh tree or to be built with
include files taken from a zsh source tree instead of a -dev package
installation.

> ? It would be cool thing from one other reason. For example, there is
> gdbm module for Zsh. Its features make it capable of rising Zsh code
> to another level. However, it is rarely available in distributed Zsh

It might also benefit other things like the zython module that was
posted a while back.

Oliver

Re: Adding (compiling) a module to already build Zsh

[Lawrence Velázquez]

> On Sep 14, 2016, at 3:16 AM, Sebastian Gniazdowski <sgniazdowski@gmail.com> wrote:
> 
> it is rarely available in distributed Zsh packages (licensing issue?
> I'm not sure if license issue doesn't apply only to pcre module).
> However, with easily available build environment one would be able to
> distribute a gdbm module together with his plugin and this way
> overcome licensing issue.

Depends on the system vendor -- for instance, Linux distributions are
fine with the GPL requirement, but Apple won't ship GPL 3 software.

I also expect that package maintainers just don't think it's worth
adding the gdbm dependency, although I do ship zsh/gdbm (and zsh/pcre)
in MacPorts.

vq

Re: Adding (compiling) a module to already build Zsh

[Bart Schaefer]

On Sep 14, 10:54am, Oliver Kiddle wrote:
}
} Many of the prototypes are autogenerated with makepro.awk and that
} mechanism ties in to aspects of the module mechanism. So, for example, an
} obvious function to include in the module interface might be addhookdefs
} but the prototype is in module.epro. This may not be a problem as such
} but it'd need someone who understands what's going on in that area.

This.  It's difficult enough right now to add a module directly to the
full zsh tree, much less tease apart all the bits that would need to
be teased in order to build with only a few pre-existing headers.

Packages like libapreq (mod_perl) and the old X11 had similar problems
back in the day.  (I haven't built loadable modules against any of
their successors in many years, so I don't know the current state.)
The decisions we made with .epro files and awk scripts were all good
ones at the time.  If we could find an expert on the current state of
dynamic loading on a variety of platforms, it would be very helpful.

Re: Adding (compiling) a module to already build Zsh

[Mikael Magnusson]

On Wed, Sep 14, 2016 at 10:54 AM, Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
> Sebastian Gniazdowski wrote:
>> > On Sep 12,  8:12am, Sebastian Gniazdowski wrote:
>
>> > } I think gdbm and pcre add linkage to additional libraries to
>> > } the Zsh binary itself.
>
> You can usually prevent that with something like
>   LDFLAGS='-Wl,--as-needed'
> Depending on you linker, the option might be -zignore.
>
> Or you can retrospectively remove the library from the NEEDED list with
> a tool like patchelf.
>
> The limitation is really with autoconf or at least our use of it.
> We add checks for libraries such as pcre and it adds -lpcre to the
> global list of linker options. I don't know whether anything has changed
> in autoconf to make it easier to limit that so it would only be used
> when linking the pcre module - anyone?

http://cgit.mika.l3ib.org/cgit/zsh-cvs/commit/?h=a095c720fff0893b9f1ca3f96bb28de125aa155f

I did this at least 5 years ago so I didn't know as much then as I do
know about autoconf, but I think AC_CHECK_LIB is a convenience macro
for something that first checks if a lib is linkable, and then adds it
to the list of things to link to globally. We could just expand those
to the full thing that only does the check without adding, and since
the lib is always the same, the part of the patch that has the
unconditional lib names in the .mdd files should be fine?

https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/html_node/Libraries.html
ah, actually we can just provide the action-if-found action and it
won't add the name to LIBS for us, even better.

-- 
Mikael Magnusson

Re: Adding (compiling) a module to already build Zsh

[Sebastian Gniazdowski]

On 14 September 2016 at 21:03, Mikael Magnusson <mikachu@gmail.com> wrote:
> http://cgit.mika.l3ib.org/cgit/zsh-cvs/commit/?h=a095c720fff0893b9f1ca3f96bb28de125aa155f
>
> I did this at least 5 years ago so I didn't know as much then as I do
> know about autoconf, but I think AC_CHECK_LIB is a convenience macro
> for something that first checks if a lib is linkable, and then adds it
> to the list of things to link to globally. We could just expand those
> to the full thing that only does the check without adding, and since
> the lib is always the same, the part of the patch that has the
> unconditional lib names in the .mdd files should be fine?
>
> https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/html_node/Libraries.html
> ah, actually we can just provide the action-if-found action and it
> won't add the name to LIBS for us, even better.

I once proposed update of configure when I at Ubuntu 14.04 observed,
that when the terminal handling library is detected to be terminfo,
Zcurses is not being build. I was warned that the "link modules'
libraries to Zsh binary" is dictated by traits of some platforms, so
if Zsh binary doesn't have ncurses, then any module also can't have
it, and thus Zcurses is not build. I recall analyzing configure.ac and
seeing there a precedence list of terminal handling libraries, when I
wondered why configure ends with "-ldl -ltinfo -lrt -lm -lc", without
-lncurses (ran the configure on Ubuntu 14.04 laptop right now). Being
afraid about Zcurses future, I want to say that possibly altered
configure.ac should strive for always building Zcurses, because it is
an important module, maybe currently I'm the single person that's
going to prove that, but I will with my new plugin which will change
the way shell is used and put in shadow fish and bash. Zcurses is an
asset, together with loadable modules, rich flags (e.g. [[
${(M)${(f)"$(<$f)"}:#$query} ]] as replacement for forking for grep),
patterns, etc. One can code like in Ruby, what I proved with
zsh-navigation-tools that supports 120k-150k elements in the
selectable list function n-list. Of course Zsh coding style has its
long learning curve, I code for about 14 months and constantly
improve, and I must say that many other plugin authors devote say 1
month for a plugin, when I constantly updated ZNT for 10 months. In
short, Zsh can have developer community (already has, but they code
close to ABSGuide and don't devote enough time) writing software
thanks to well established platform, and "Zcurses everywhere" is
important part of that platform, and not requiring that Zsh binary is
linked to what modules are linked will allow to do "Gdbm everywhere"
etc.

Best regards,
Sebastian Gniazdowski