zsh/pcre module breakage when static build enabled

zsh/pcre module breakage when static build enabled

[Vadim A. Misbakh-Soloviov]

[-- Attachment #1: Type: text/plain, Size: 482 bytes --]

Hi there!
I faced strange issue: when I build zsh-5.4.2 as static binary and trying to 
run, for example, something like that:

> $  [[ 'foo→bar' =~ .([^[:ascii:]]). ]]; print $MATCH

I'm getting following errors:

> zsh: module `zsh/pcre' has no such feature: `C:pcre-match'
> zsh: -pcre-match not available for regex

Although, when I rebuild it as dynamically linked binary, pcre starting to 
work again

I think, it looks like a bug.

Isn't it a way to fix it?

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: zsh/pcre module breakage when static build enabled

[Phil Pennock]

On 2018-03-20 at 00:33 +0700, Vadim A. Misbakh-Soloviov wrote:
> Hi there!
> I faced strange issue: when I build zsh-5.4.2 as static binary and trying to 
> run, for example, something like that:
> 
> > $  [[ 'foo→bar' =~ .([^[:ascii:]]). ]]; print $MATCH
> 
> I'm getting following errors:
> 
> > zsh: module `zsh/pcre' has no such feature: `C:pcre-match'
> > zsh: -pcre-match not available for regex
> 
> Although, when I rebuild it as dynamically linked binary, pcre starting to 
> work again
> 
> I think, it looks like a bug.
> 
> Isn't it a way to fix it?

I went to do a static build on FreeBSD (Darwin doesn't support static
binaries) and when grabbing the configure options used by Ports, saw
this:

  ## Some modules can only be built as a shared library.
  ## If you enable STATIC, you may get strange errors if you, a script,
  ## or a plugin tries to use the regex module.

So someone has spotted this before, but I don't recall seeing a
bug-report.

I can't reproduce on FreeBSD, as long as I follow the 'INSTALL'
instructions.  Perhaps it could be made easier to build static shells
with variant features, but I suspect that if you really need a static
shell, you can follow the steps.

Here's what I did.  Did you perhaps skip the '--disable-dynamic' option,
required to change how zsh manages the module code when building
statically?

----------------8< zsh build zsh-5.4.2-198-gd2350a1e8 >8----------------
git pull
git status
git clean -fxd
./.preconfig
CPPFLAGS=-I/usr/local/include LDFLAGS='-L/usr/local/lib -static' \
  ./configure --prefix=$HOME/uzsh \
    --enable-etcdir=/etc --with-tcsetpgrp \
    --enable-function-subdirs --enable-maildir-support \
    --enable-multibyte --enable-zsh-mem --enable-zsh-secure-free \
    --sysconfdir=/usr/local \
    --disable-dynamic --with-term-lib="tinfow tinfo" \
    --enable-pcre --disable-dynamic
sed -Ei '' -e '/name=zsh\/(pcre|regex)/s/link=no/link=static/' config.modules
make && make install.bin install.modules install.fns

~/uzsh/bin/zsh -f

tower% [[ 'foo→bar' =~ .([^[:ascii:]]). ]]; print $MATCH
zsh: failed to compile regex: invalid character class

tower% setopt rematchpcre
tower% [[ 'foo→bar' =~ .([^[:ascii:]]). ]]; print $MATCH
o→b
----------------8< zsh build zsh-5.4.2-198-gd2350a1e8 >8----------------

I just rebuilt with 'config.modules' saying 'link=static' for every
module except: zsh/attr, zsh/db/gdbm, zsh/example

I can use the resulting shell quite well.  It even functions well enough
to display my prompt, which is a bit of a torture test. :^)

-Phil

Re: zsh/pcre module breakage when static build enabled

[Vadim A. Misbakh-Soloviov]

[-- Attachment #1: Type: text/plain, Size: 2784 bytes --]

В письме от вторник, 20 марта 2018 г. 6:42:36 +07 пользователь Phil Pennock 
написал:
> On 2018-03-20 at 00:33 +0700, Vadim A. Misbakh-Soloviov wrote:
> > Hi there!
> > I faced strange issue: when I build zsh-5.4.2 as static binary and trying
> > to> 
> > run, for example, something like that:
> > > $  [[ 'foo→bar' =~ .([^[:ascii:]]). ]]; print $MATCH
> > 
> > I'm getting following errors:
> > > zsh: module `zsh/pcre' has no such feature: `C:pcre-match'
> > > zsh: -pcre-match not available for regex
> > 
> > Although, when I rebuild it as dynamically linked binary, pcre starting to
> > work again
> > 
> > I think, it looks like a bug.
> > 
> > Isn't it a way to fix it?
> 
> I went to do a static build on FreeBSD (Darwin doesn't support static
> binaries) and when grabbing the configure options used by Ports, saw
> this:
> 
>   ## Some modules can only be built as a shared library.
>   ## If you enable STATIC, you may get strange errors if you, a script,
>   ## or a plugin tries to use the regex module.
> 
> So someone has spotted this before, but I don't recall seeing a
> bug-report.

Well, yup. Even some of my friends said they faced that too.

> Here's what I did.  Did you perhaps skip the '--disable-dynamic' option,
> required to change how zsh manages the module code when building
> statically?

Well, actually, I'm using Gentoo (and it's "static" use-flag). And final 
configure-string is (for experiment clearance I also took version from git 
HEAD):

> ./configure --prefix=/usr --build=x86_64-pc-linux-gnu --host=x86_64-pc-
linux-gnu --mandir=/usr/share/man --infodir=/usr/share/info --datadir=/usr/
share --sysconfdir=/etc --localstatedir=/var/lib --docdir=/usr/share/doc/
zsh-9999 --htmldir=/usr/share/doc/zsh-9999/html --bindir=/bin --libdir=/usr/
lib64 --enable-etcdir=/etc/zsh --enable-runhelpdir=/usr/share/zsh/9999/help --
enable-fndir=/usr/share/zsh/9999/functions --enable-site-fndir=/usr/share/zsh/
site-functions --enable-function-subdirs --with-tcsetpgrp --enable-maildir-
support --enable-pcre --enable-cap --enable-multibyte --enable-unicode9 --
enable-gdbm --disable-dynamic

And there is also seddig similar to yours, and, as result:

> name=zsh/pcre modfile=Src/Modules/pcre.mdd link=static auto=yes load=no
> name=zsh/regex modfile=Src/Modules/regex.mdd link=static auto=yes load=no


Although, result still the same as in release version.


> make install.bin install.modules install.fns

Although, instead of this, ebuild just doing `make install install.info`, and 
then copies completions and so on by itself.

But as far as I understand principles of static build, it shouldn't interfere 
in any way, since the binary (zsh)

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: zsh/pcre module breakage when static build enabled

[Vadim A. Misbakh-Soloviov]

[-- Attachment #1: Type: text/plain, Size: 979 bytes --]

Well, I just tried to repeat your steps:

---------

mkdir /tmp/zsh
cd /tmp/zsh
git pull git://git.code.sf.net/p/zsh/code
./.preconfig
CPPFLAGS=-I/usr/include LDFLAGS='-L/usr/lib64 -static' \
./configure --prefix=/tmp/zsh/uzsh  --enable-etcdir=/etc --with-tcsetpgrp \
--enable-function-subdirs --enable-maildir-support --enable-multibyte \
--enable-zsh-mem --enable-zsh-secure-free  --sysconfdir=/usr \
--disable-dynamic --with-term-lib="tinfow tinfo" --enable-pcre \
--disable-dynamic

# yeah, even with double `disable-dynamic like in yours)

sed -i -e '/name=zsh\/(pcre|regex)/s/link=no/link=static/' config.modules
make

----------

Then I called ./Src/zsh -f, and...

note%  setopt rematchpcre
note%  [[ 'foo→bar' =~ .([^[:ascii:]]). ]]; print $MATCH
zsh: module `zsh/pcre' has no such feature: `C:pcre-match'
zsh: -pcre-match not available for regex

So, it seems, it is either somehow related to linux only, or to libpcre 
version, maybe...

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: zsh/pcre module breakage when static build enabled

[Phil Pennock]

On 2018-03-20 at 12:16 +0700, Vadim A. Misbakh-Soloviov wrote:
> # yeah, even with double `disable-dynamic like in yours)

Sorry, mistake caused by tiredness.

The separate install steps were because I couldn't be bothered to
install the YODL suite of tools in that jail, so skipped the doc stages.

> Then I called ./Src/zsh -f, and...
> 
> note%  setopt rematchpcre
> note%  [[ 'foo→bar' =~ .([^[:ascii:]]). ]]; print $MATCH
> zsh: module `zsh/pcre' has no such feature: `C:pcre-match'
> zsh: -pcre-match not available for regex
> 
> So, it seems, it is either somehow related to linux only, or to libpcre 
> version, maybe...

I just built zsh on Ubuntu Xenial with PCRE packages from 2:8.38-3.1,
and could not reproduce.

I think the only way to get this failure is if PCRE support wasn't
actually built in, and the pcre module became a stub.  Easiest way to
check for that is to look at the HAVE_* values in config.h:

    % grep HAVE_PCRE_ config.h
    #define HAVE_PCRE_COMPILE 1
    #define HAVE_PCRE_EXEC 1
    #define HAVE_PCRE_H 1
    #define HAVE_PCRE_STUDY 1

The build flow requires that you enable the flag, and that
`pcre-config --version` exits successfully.

Are you missing a build dependency?

If you examine the file `config.log` and look for `pcre` then you should
see what checks the configure system ran and what the results were,
which may help with tracing down when things have gone wrong.

There may still be a bug in zsh, or some failure mode which can be
handled more gracefully, but we'll need to know what's actually failing
first.

Hrm, perhaps `--build-require-all-enables` or something for the
`configure` script could have the build system go "hey, you enabled
PCRE, but I'm not going to enable it for $reasons, so that's a fatal
error".  Mechanism: config.h would #define `SHOULD_ERROR_IF_ENABLED` and
`ENABLE_REQUESTED_FOO` for each FOO, and then the #if guard in/around
the relevant modules would just have an `#else if
defined(SHOULD_ERROR_IF_ENABLED) && defined(ENABLE_REQUESTED_FOO)`
followed by `# error whatever` just at the end.

-Phil

Re: zsh/pcre module breakage when static build enabled

[Vadim A. Misbakh-Soloviov]

[-- Attachment #1: Type: text/plain, Size: 4077 bytes --]

>     % grep HAVE_PCRE_ config.h
>     #define HAVE_PCRE_COMPILE 1
>     #define HAVE_PCRE_EXEC 1
>     #define HAVE_PCRE_H 1
>     #define HAVE_PCRE_STUDY 1
> 

Sorry for delay, I had some troubles offline...

So, here is where our output differs. I've the following here:

> #define HAVE_PCRE_COMPILE 1
> /* #undef HAVE_PCRE_EXEC */
> #define HAVE_PCRE_H 1
> /* #undef HAVE_PCRE_STUDY */

Although, `pcre-config` works fine:

> % pcre-config --version
> 8.42

So, I looked in config.log and found following:
> configure:8361: checking for pcre_study
> configure:8361: gcc -o conftest  -Wall -Wmissing-prototypes -O2 -I/usr/
include  -L/usr/lib64 -static conftest.c -lgdbm -L/usr/lib64 -lpcre -ltinfow -
lrt -lm  -lc >&5
> /usr/lib64/libpcre.a(libpcre_la-pcre_jit_compile.o): In function 
`sljit_free_exec':
> pcre_jit_compile.c:(.text+0x190c): undefined reference to 
`pthread_mutex_lock'
> /usr/lib64/libpcre.a(libpcre_la-pcre_jit_compile.o): In function 
`pcre_jit_free_unused_memory':
> pcre_jit_compile.c:(.text+0x23359): undefined reference to 
`pthread_mutex_lock'
> /usr/lib64/libpcre.a(libpcre_la-pcre_jit_compile.o): In function 
`sljit_free_exec':
> pcre_jit_compile.c:(.text+0x1974): undefined reference to 
`pthread_mutex_unlock'
> /usr/lib64/libpcre.a(libpcre_la-pcre_jit_compile.o): In function 
`pcre_jit_free_unused_memory':
> pcre_jit_compile.c:(.text+0x233c9): undefined reference to 
`pthread_mutex_unlock'
> /usr/lib64/libpcre.a(libpcre_la-pcre_jit_compile.o): In function 
`sljit_generate_code':
> pcre_jit_compile.c:(.text.unlikely+0x1eb): undefined reference to 
`pthread_mutex_lock'
> pcre_jit_compile.c:(.text.unlikely+0x287): undefined reference to 
`pthread_mutex_unlock'
> pcre_jit_compile.c:(.text.unlikely+0x2d9): undefined reference to 
`pthread_mutex_unlock'
> pcre_jit_compile.c:(.text.unlikely+0x36f): undefined reference to 
`pthread_mutex_unlock'
> /usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/libgcc_eh.a(unwind-dw2-fde-dip.o): In 
function `__gthread_mutex_lock':
> /var/tmp/portage/sys-devel/gcc-7.3.0-r1/work/build/x86_64-pc-linux-gnu/
libgcc/./gthr-default.h:748: undefined reference to `pthread_mutex_lock'
> /var/tmp/portage/sys-devel/gcc-7.3.0-r1/work/build/x86_64-pc-linux-gnu/
libgcc/./gthr-default.h:748: undefined reference to `pthread_mutex_lock'
> /var/tmp/portage/sys-devel/gcc-7.3.0-r1/work/build/x86_64-pc-linux-gnu/
libgcc/./gthr-default.h:748: undefined reference to `pthread_mutex_lock'
> /usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/libgcc_eh.a(unwind-dw2-fde-dip.o): In 
function `__gthread_mutex_unlock':
> /var/tmp/portage/sys-devel/gcc-7.3.0-r1/work/build/x86_64-pc-linux-gnu/
libgcc/./gthr-default.h:778: undefined reference to `pthread_mutex_unlock'
> /var/tmp/portage/sys-devel/gcc-7.3.0-r1/work/build/x86_64-pc-linux-gnu/
libgcc/./gthr-default.h:778: undefined reference to `pthread_mutex_unlock'
> /usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/libgcc_eh.a(unwind-dw2-fde-dip.o): In 
function `__gthread_mutex_lock':
> /var/tmp/portage/sys-devel/gcc-7.3.0-r1/work/build/x86_64-pc-linux-gnu/
libgcc/./gthr-default.h:748: undefined reference to `pthread_mutex_lock'
> /usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/libgcc_eh.a(unwind-dw2-fde-dip.o): In 
function `__gthread_mutex_unlock':
> /var/tmp/portage/sys-devel/gcc-7.3.0-r1/work/build/x86_64-pc-linux-gnu/
libgcc/./gthr-default.h:778: undefined reference to `pthread_mutex_unlock'
> /var/tmp/portage/sys-devel/gcc-7.3.0-r1/work/build/x86_64-pc-linux-gnu/
libgcc/./gthr-default.h:778: undefined reference to `pthread_mutex_unlock'
> /var/tmp/portage/sys-devel/gcc-7.3.0-r1/work/build/x86_64-pc-linux-gnu/
libgcc/./gthr-default.h:778: undefined reference to `pthread_mutex_unlock'
> /var/tmp/portage/sys-devel/gcc-7.3.0-r1/work/build/x86_64-pc-linux-gnu/
libgcc/./gthr-default.h:778: undefined reference to `pthread_mutex_unlock'
> collect2: error: ld returned 1 exit status


Although, I already tried to add `-lpthread` to LDFLAGS, but it didn't fixed 
that error :'(
And also, shared version of libpcre is linked to pthread, so I guess, static 
version should be as well. Isn't it?

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: zsh/pcre module breakage when static build enabled

[Vadim A. Misbakh-Soloviov]

[-- Attachment #1: Type: text/plain, Size: 181 bytes --]

Well, quick-fix: I've changed `LDFLAGS=... -lpthread` to `LIBS=-lpthread` and 
now all works like a charm.

So, I guess, that `-lpthread` should be somehow upstreamed now, isn't it?

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: zsh/pcre module breakage when static build enabled

[Bart Schaefer]

On Sat, Mar 24, 2018 at 11:25 PM, Vadim A. Misbakh-Soloviov
<mva@mva.name> wrote:
>
> So, I guess, that `-lpthread` should be somehow upstreamed now, isn't it?

This is a somewhat strange situation, because zsh process/signal
handling is not compatible with POSIX threading.

Why is pcre linking to this?  If pcre is actually starting threads,
there may be unexpected side-effects.

At the very least we'd need an additional configure test to detect
this pcre dependency.

Re: zsh/pcre module breakage when static build enabled

[Vadim A. Misbakh-Soloviov]

[-- Attachment #1: Type: text/plain, Size: 700 bytes --]

> Why is pcre linking to this?  If pcre is actually starting threads,
> there may be unexpected side-effects.

I don't know :(
But I know that dynamic version of libpcre links to pthread as well:

> $ ldd /lib64/libpcre.so.1
>        linux-vdso.so.1 (0x00007ffd63be6000)
>        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f7e8015e000)
>        libc.so.6 => /lib64/libc.so.6 (0x00007f7e7fd9a000)
>        /lib64/ld-linux-x86-64.so.2 (0x00007f7e805f0000)

// And even libpcre2 links to it.

And I don't see any kind of `--enable-threads` or `--enable-pthread` in their 
ebuilds (package manager build scripts), so, I guess, they links to pthread on 
their own, and not on maintainers decision.

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: zsh/pcre module breakage when static build enabled

[Phil Pennock]

On 2018-03-25 at 12:50 -0700, Bart Schaefer wrote:
> Why is pcre linking to this?  If pcre is actually starting threads,
> there may be unexpected side-effects.

With PCRE1, it means that the JIT was enabled, and that requires
threads.  With PCRE2, I'm less sure.

> At the very least we'd need an additional configure test to detect
> this pcre dependency.

The PCRE ChangeLog includes:
-------------------------8< PCRE1 ChangeLog >8--------------------------
Version 8.37 28-April-2015
--------------------------
[ ... ]
40. Static linking against the PCRE library using the pkg-config module was
    failing on missing pthread symbols.
-------------------------8< PCRE1 ChangeLog >8--------------------------

The same ChangeLog text is in PCRE2 as item 25 for version 10.20
released 30-June-2015.

Vadim has `pcre-config --version` reporting 8.42 though.  Perhaps a
regression, perhaps a system mismatch between libraries and tools.

Regardless, all the relevant errors from Vadim's post are in compiling
JIT stuff, which we're not using.  So it's bloat for unused
functionality but shouldn't be affecting our operation.

-Phil

Re: zsh/pcre module breakage when static build enabled

[Vadim A. Misbakh-Soloviov]

[-- Attachment #1: Type: text/plain, Size: 1809 bytes --]

В письме от понедельник, 26 марта 2018 г. 12:37:15 +07 пользователь Phil 
Pennock написал:
> 40. Static linking against the PCRE library using the pkg-config module was
>     failing on missing pthread symbols.

Well, I've looked in commit history, and found that:
https://github.com/svn2github/pcre/commit/
2765d0933715e6b2c78a4e74fa8d5cad27273f29

And my `libpcre.pc` have
> Libs.private: -pthread
at that place.

Although, I'm not sure if it is okay for it to be `-pthread`, and not `-
lpthread`.

> Vadim has `pcre-config --version` reporting 8.42 though.  Perhaps a
> regression, perhaps a system mismatch between libraries and tools.

I bet on regression (since I'm using source-based distro and it's pretty 
unlikely to have that mismatch here.

> 
> Regardless, all the relevant errors from Vadim's post are in compiling
> JIT stuff, which we're not using.  So it's bloat for unused
> functionality but shouldn't be affecting our operation.

Well, I'll try to check if manual fixing of libpcre's pkg-config file would 
help. Now it really looks like libpcre issue.

By the way,
> $   ldd /usr/lib64/zsh/5.4.2/zsh/pcre.so
>        linux-vdso.so.1 (0x00007ffd8469b000)
>        libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f368efaa000)
>        libc.so.6 => /lib64/libc.so.6 (0x00007f368ebe6000)
>        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f368e9c6000)
>        /lib64/ld-linux-x86-64.so.2 (0x00007f368f420000)

Here, libpthread is inherited from dynamic libpcre, and (I hope) it doesn't 
interfere with any zsh's internal work. So, I guess, it shouldn't interfere in 
static too.

But I'm not sure if it should be worked around in zsh at all (in case, if I'll 
confirm that is libpcre's pc-file bug).

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: zsh/pcre module breakage when static build enabled

[Vadim A. Misbakh-Soloviov]

[-- Attachment #1: Type: text/plain, Size: 617 bytes --]

> Although, I'm not sure if it is okay for it to be `-pthread`, and not `-
> lpthread`.

Well, my test shows that it is okay, and result is pretty similar.

> Well, I'll try to check if manual fixing of libpcre's pkg-config file would
> help. Now it really looks like libpcre issue.

Also my tests shows that, well, actually it may be an issue of how zsh uses 
pcre's pkg-config file.

I don't know how to get such a "private" libs using `pcre-config` tool, but 
pkg-config itself is fine about that:

> $ pkg-config --libs libpcre # dynamic
> -lpcre
> $ pkg-config --libs --static libpcre # static
> -lpcre -pthread

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

zsh/pcre module breakage when static build enabled

[Vadim A. Misbakh-Soloviov]

[-- Attachment #1: Type: text/plain, Size: 482 bytes --]

Hi there!
I faced strange issue: when I build zsh-5.4.2 as static binary and trying to 
run, for example, something like that:

> $  [[ 'foo→bar' =~ .([^[:ascii:]]). ]]; print $MATCH

I'm getting following errors:

> zsh: module `zsh/pcre' has no such feature: `C:pcre-match'
> zsh: -pcre-match not available for regex

Although, when I rebuild it as dynamically linked binary, pcre starting to 
work again

I think, it looks like a bug.

Isn't it a way to fix it?

[-- Attachment #2: This is a digitally signed message part. --]
[-- Type: application/pgp-signature, Size: 228 bytes --]