PATCH: zsh/pcre module

PATCH: zsh/pcre module

[Clint Adams]

This adds via builtins in the zsh/pcre module, linked with a PCRE
library rudimentary support for perl5-compatible regular expressions.
I imagine that further augmentation would be of use, including
the capturing of substrings (and putting them in positional parameters?)

The module should probably complain on systems without pcre_*.

Index: zshconfig.ac
===================================================================
RCS file: /cvsroot/zsh/zsh/zshconfig.ac,v
retrieving revision 1.13
diff -u -r1.13 zshconfig.ac
--- zshconfig.ac	2001/07/02 11:33:26	1.13
+++ zshconfig.ac	2001/07/02 19:27:06
@@ -475,7 +475,7 @@
 		 limits.h fcntl.h libc.h sys/utsname.h sys/resource.h \
 		 locale.h errno.h stdlib.h unistd.h sys/capability.h \
 		 utmp.h utmpx.h sys/types.h pwd.h grp.h poll.h sys/mman.h \
-		 netinet/in_systm.h)
+		 netinet/in_systm.h, pcre.h)
 if test $dynamic = yes; then
   AC_CHECK_HEADERS(dlfcn.h)
   AC_CHECK_HEADERS(dl.h)
@@ -643,6 +643,9 @@
 
 AC_CHECK_LIB(socket, socket)
 
+dnl pcre-config should probably be employed here
+AC_CHECK_LIB(pcre, pcre_compile)
+
 dnl ---------------------
 dnl CHECK TERMCAP LIBRARY
 dnl ---------------------
@@ -933,7 +936,8 @@
 	       putenv getenv \
 	       brk sbrk \
 	       pathconf sysconf \
-	       tgetent tigetflag tigetnum tigetstr setupterm)
+	       tgetent tigetflag tigetnum tigetstr setupterm \
+	       pcre_compile pcre_study pcre_exec)
 AC_FUNC_STRCOLL
 
 dnl  Check if tgetent accepts NULL (and will allocate its own termcap buffer)
Index: Doc/Zsh/mod_pcre.yo
===================================================================
RCS file: mod_pcre.yo
diff -N mod_pcre.yo
--- /dev/null	Thu May 24 22:33:05 2001
+++ mod_pcre.yo	Mon Jul  2 12:27:06 2001
@@ -0,0 +1,22 @@
+COMMENT(!MOD!zsh/pcre
+Interface to the PCRE library.
+!MOD!)
+cindex(regular expressions, perl-compatible)
+The tt(zsh/pcre) module makes some commands available as builtins:
+
+startitem()
+findex(pcre_compile)
+item(tt(pcre_compile) [ tt(-aimx) ] var(PCRE))(
+Compiles a perl-compatible regular expression.
+)
+findex(pcre_study)
+item(tt(pcre_study))(
+Studies the previously-compiled PCRE which may result in faster
+matching.
+)
+findex(pcre_match)
+item(tt(pcre_match) var(string))(
+Returns successfully if tt(string) matches the previously-compiled
+PCRE.
+)
+enditem()
Index: Functions/Example/zpgrep
===================================================================
RCS file: zpgrep
diff -N zpgrep
--- /dev/null	Thu May 24 22:33:05 2001
+++ zpgrep	Mon Jul  2 12:27:06 2001
@@ -0,0 +1,25 @@
+# Usage: zpgrep <perl5-compatible regex> <file1> <file2> ... <fileN>
+#
+
+zpgrep() {
+local file pattern
+
+pattern=$1
+shift
+
+if ((! ARGC)) then
+	set -- -
+fi
+
+pcre_compile $pattern
+pcre_study
+
+for file
+do
+	if [[ "$file" == - ]] then
+		while read -u0 buf; do pcre_match $buf && print $buf; done
+	else
+		while read -u0 buf; do pcre_match $buf && print $buf; done < "$file"
+	fi
+done
+}
Index: Src/Modules/pcre.c
===================================================================
RCS file: pcre.c
diff -N pcre.c
--- /dev/null	Thu May 24 22:33:05 2001
+++ pcre.c	Mon Jul  2 12:27:07 2001
@@ -0,0 +1,134 @@
+/*
+ * pcre.c - interface to the PCRE library
+ *
+ * This file is part of zsh, the Z shell.
+ *
+ * Copyright (c) 2001 Clint Adams
+ * All rights reserved.
+ *
+ * Permission is hereby granted, without written agreement and without
+ * license or royalty fees, to use, copy, modify, and distribute this
+ * software and to distribute modified versions of this software for any
+ * purpose, provided that the above copyright notice and the following
+ * two paragraphs appear in all copies of this software.
+ *
+ * In no event shall Clint Adams or the Zsh Development Group be liable
+ * to any party for direct, indirect, special, incidental, or consequential
+ * damages arising out of the use of this software and its documentation,
+ * even if Andrew Main and the Zsh Development Group have been advised of
+ * the possibility of such damage.
+ *
+ * Clint Adams and the Zsh Development Group specifically disclaim any
+ * warranties, including, but not limited to, the implied warranties of
+ * merchantability and fitness for a particular purpose.  The software
+ * provided hereunder is on an "as is" basis, and Andrew Main and the
+ * Zsh Development Group have no obligation to provide maintenance,
+ * support, updates, enhancements, or modifications.
+ *
+ */
+
+/**/
+#if defined(HAVE_PCRE_COMPILE) && defined(HAVE_PCRE_EXEC)
+
+#include "pcre.mdh"
+#include "pcre.pro"
+#include <pcre.h>
+
+static pcre *pcre_pattern;
+static pcre_extra *pcre_hints;
+
+/**/
+static int
+bin_pcre_compile(char *nam, char **args, char *ops, int func)
+{
+    int pcre_opts = 0, pcre_errptr;
+    const char *pcre_error;
+    
+    if(ops['a']) pcre_opts |= PCRE_ANCHORED;
+    if(ops['i']) pcre_opts |= PCRE_CASELESS;
+    if(ops['m']) pcre_opts |= PCRE_MULTILINE;
+    if(ops['x']) pcre_opts |= PCRE_EXTENDED;
+    
+    pcre_pattern = pcre_compile(*args, pcre_opts, &pcre_error, &pcre_errptr, NULL);
+    
+    if (pcre_pattern == NULL)
+    {
+	zwarnnam(nam, "error in regex: %s", pcre_error, 0);
+	return 1;
+    }
+    
+    return 0;
+}
+
+/**/
+#ifdef HAVE_PCRE_STUDY
+
+/**/
+static int
+bin_pcre_study(char *nam, char **args, char *ops, int func)
+{
+    const char *pcre_error;
+    
+    pcre_hints = pcre_study(pcre_pattern, 0, &pcre_error);
+    if (pcre_error != NULL)
+    {
+	zwarnnam(nam, "error while studying regex: %s", pcre_error, 0);
+	return 1;
+    }
+    
+    return 0;
+}
+
+/**/
+#endif /* HAVE_PCRE_STUDY */
+
+/**/
+static int
+bin_pcre_match(char *nam, char **args, char *ops, int func)
+{
+#define PCRE_OVEC_SIZE 50
+    
+    int ovec[PCRE_OVEC_SIZE];   /* throwing this away now, but will be useful someday */
+    
+    return !(pcre_exec(pcre_pattern, pcre_hints, *args, strlen(*args), 0, 0, ovec, PCRE_OVEC_SIZE) >= 0);
+}
+
+static struct builtin bintab[] = {
+    BUILTIN("pcre_compile", 0, bin_pcre_compile, 1, 1, 0, "aimx",  NULL),
+#ifdef HAVE_PCRE_STUDY
+    BUILTIN("pcre_study",   0, bin_pcre_study,   0, 0, 0, NULL,    NULL),
+#endif
+    BUILTIN("pcre_match",   0, bin_pcre_match,   1, 1, 0, NULL,    NULL)
+};
+
+/**/
+int
+setup_(Module m)
+{
+    return 0;
+}
+
+/**/
+int
+boot_(Module m)
+{
+    return !addbuiltins(m->nam, bintab, sizeof(bintab)/sizeof(*bintab));
+}
+
+/**/
+int
+cleanup_(Module m)
+{
+    deletebuiltins(m->nam, bintab, sizeof(bintab)/sizeof(*bintab));
+    return 0;
+}
+
+/**/
+int
+finish_(Module m)
+{
+    return 0;
+}
+
+/**/
+#endif /* HAVE_PCRE_COMPILE && HAVE_PCRE_EXEC */
Index: Src/Modules/pcre.mdd
===================================================================
RCS file: pcre.mdd
diff -N pcre.mdd
--- /dev/null	Thu May 24 22:33:05 2001
+++ pcre.mdd	Mon Jul  2 12:27:07 2001
@@ -0,0 +1,7 @@
+name=zsh/pcre
+link=either
+load=no
+
+autobins="pcre_compile pcre_study pcre_match"
+
+objects="pcre.o"

Re: PATCH: zsh/pcre module

[Bart Schaefer]

On Jul 2,  3:35pm, Clint Adams wrote:
} Subject: PATCH: zsh/pcre module
}
} Index: zshconfig.ac
} +		 netinet/in_systm.h, pcre.h)

Should we create a separate .ac file for tests related only to modules,
and include it in the main .ac somehow?

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

RE: PATCH: zsh/pcre module

[Andrej Borsenkow]

> 
> Should we create a separate .ac file for tests related only to modules,
> and include it in the main .ac somehow?
> 

My long cherished idea is own configure for every module (that needs it).

-andrej

Re: PATCH: zsh/pcre module

[Bart Schaefer]

On Jul 3,  9:54am, Andrej Borsenkow wrote:
}
} My long cherished idea is own configure for every module (that needs it).

Incidentally, I'm not entirely sure I approve of adding this PCRE module
to the regular zsh distribution.

We now have no less than three regular expression packages -- the one
that implements globbing, the one from the zregex functions in the zutil
module, and PCRE.  Do we really need them all?

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

RE: PATCH: zsh/pcre module

[Andrej Borsenkow]

> 
> On Jul 3,  9:54am, Andrej Borsenkow wrote:
> }
> } My long cherished idea is own configure for every module (that 
> needs it).
> 
> Incidentally, I'm not entirely sure I approve of adding this PCRE module
> to the regular zsh distribution.
> 
> We now have no less than three regular expression packages -- the one
> that implements globbing, the one from the zregex functions in the zutil
> module, and PCRE.  Do we really need them all?
> 

What zregex does?

-andrej

RE: PATCH: zsh/pcre module

[Andrej Borsenkow]

> +AC_CHECK_LIB(pcre, pcre_compile)

I urge everybody to use AC_SEARCH_LIB. AC_CHECK_LIB has nasty habit of
adding library even when it is not needed; moreover, it does it in front,
thus sometimes preventing you from using your own library.

-andrej

Re: PATCH: zsh/pcre module

[Clint Adams]

> I urge everybody to use AC_SEARCH_LIB. AC_CHECK_LIB has nasty habit of
> adding library even when it is not needed; moreover, it does it in front,
> thus sometimes preventing you from using your own library.

This switches AC_CHECK_LIB to AC_SEARCH_LIBS.  It sets
positional parameters with the contents of capture buffers.

Index: zshconfig.ac
===================================================================
RCS file: /cvsroot/zsh/zsh/zshconfig.ac,v
retrieving revision 1.15
diff -u -r1.15 zshconfig.ac
--- zshconfig.ac	2001/07/03 05:59:55	1.15
+++ zshconfig.ac	2001/07/03 16:04:11
@@ -644,7 +644,7 @@
 AC_CHECK_LIB(socket, socket)
 
 dnl pcre-config should probably be employed here
-AC_CHECK_LIB(pcre, pcre_compile)
+AC_SEARCH_LIBS(pcre_compile, pcre)
 
 dnl ---------------------
 dnl CHECK TERMCAP LIBRARY
Index: Src/Modules/pcre.c
===================================================================
RCS file: /cvsroot/zsh/zsh/Src/Modules/pcre.c,v
retrieving revision 1.1
diff -u -r1.1 pcre.c
--- Src/Modules/pcre.c	2001/07/02 19:39:35	1.1
+++ Src/Modules/pcre.c	2001/07/03 16:04:11
@@ -27,11 +27,12 @@
  *
  */
 
-/**/
-#if defined(HAVE_PCRE_COMPILE) && defined(HAVE_PCRE_EXEC)
 
 #include "pcre.mdh"
 #include "pcre.pro"
+
+/**/
+#if defined(HAVE_PCRE_COMPILE) && defined(HAVE_PCRE_EXEC)
 #include <pcre.h>
 
 static pcre *pcre_pattern;
@@ -49,6 +50,8 @@
     if(ops['m']) pcre_opts |= PCRE_MULTILINE;
     if(ops['x']) pcre_opts |= PCRE_EXTENDED;
     
+    pcre_hints = NULL;  /* Is this necessary? */
+    
     pcre_pattern = pcre_compile(*args, pcre_opts, &pcre_error, &pcre_errptr, NULL);
     
     if (pcre_pattern == NULL)
@@ -86,11 +89,38 @@
 static int
 bin_pcre_match(char *nam, char **args, char *ops, int func)
 {
-#define PCRE_OVEC_SIZE 50
+    int ret, capcount, *ovec, ovecsize;
+    char **captures;
     
-    int ovec[PCRE_OVEC_SIZE];   /* throwing this away now, but will be useful someday */
+    if (pcre_fullinfo(pcre_pattern, pcre_hints, PCRE_INFO_CAPTURECOUNT, &capcount))
+    {
+	zwarnnam(nam, "error in fullinfo", NULL, 0);
+	return 1;
+    }
+    
+    ovecsize = (capcount+1)*3;
+    ovec = zalloc(ovecsize*sizeof(int));
+    
+    ret = pcre_exec(pcre_pattern, pcre_hints, *args, strlen(*args), 0, 0, ovec, ovecsize);
+    
+    if (ret==0) return 0;
+    else if (ret==PCRE_ERROR_NOMATCH) return 1; /* no match */
+    else if (ret>0) {
+	if(!pcre_get_substring_list(*args, ovec, ret, (const char ***)&captures)) {
+
+	    freearray(pparams);
+	    pparams = zarrdup(&captures[1]); /* first one would be entire string */
+	    
+	    pcre_free_substring_list((const char **)captures);
+	    return 0;
+      	}
+    }
+    else {
+	zwarnnam(nam, "error in pcre_exec", NULL, 0);
+	return 1;
+    }
     
-    return !(pcre_exec(pcre_pattern, pcre_hints, *args, strlen(*args), 0, 0, ovec, PCRE_OVEC_SIZE) >= 0);
+    return 1;
 }
 
 static struct builtin bintab[] = {

Re: PATCH: zsh/pcre module

[Bart Schaefer]

On Jul 3, 10:30am, Andrej Borsenkow wrote:
}
} What zregex does?

It's accessed via zregexparse and implements some internals of the
_regex_arguments function that is used for by few extremely complex
completion functions written by Tanaka Akira.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: PATCH: zsh/pcre module

[Clint Adams]

> Incidentally, I'm not entirely sure I approve of adding this PCRE module
> to the regular zsh distribution.
> 
> We now have no less than three regular expression packages -- the one
> that implements globbing, the one from the zregex functions in the zutil
> module, and PCRE.  Do we really need them all?

I don't know that it's necessary, but it's an optional module that provides an
alternate syntax that is perhaps more familiar to many users than that
of filename generation.

The documentation for the zregex stuff is lacking if it's meant to be used
outside of completion.

Re: PATCH: zsh/pcre module

[Oliver Kiddle]

Bart Schaefer wrote:
> 
> Incidentally, I'm not entirely sure I approve of adding this PCRE module
> to the regular zsh distribution.

I agree with Bart here in some respects.

How easy is it to package up modules separately in such a way that they
could be built and installed without needing to be unpacked over the zsh
source and compiled with it. Ideally, we would have a set of include
files which was all anyone needed to build a separate zsh module.

Does anyone remember what happened to the perl module someone posted
quite a while back?

I was going to moan about the name (pcre) because it isn't obvious what
it stands for but as it is the name of the library it uses, I suppose
I'll have to put up with it.

Oliver

Re: PATCH: zsh/pcre module

[Clint Adams]

> How easy is it to package up modules separately in such a way that they
> could be built and installed without needing to be unpacked over the zsh
> source and compiled with it. Ideally, we would have a set of include
> files which was all anyone needed to build a separate zsh module.

Also, there would need to be a mechanism for dynamically scanning for
autobins/autoparams/&c.  One could scan for module dependencies at the
same time.

RE: PATCH: zsh/pcre module

[Andrej Borsenkow]

>
> Bart Schaefer wrote:
> >
> > Incidentally, I'm not entirely sure I approve of adding this PCRE module
> > to the regular zsh distribution.
>
> I agree with Bart here in some respects.
>
> How easy is it to package up modules separately in such a way that they
> could be built and installed without needing to be unpacked over the zsh
> source and compiled with it. Ideally, we would have a set of include
> files which was all anyone needed to build a separate zsh module.
>

That is what I'm still thinking about.

I'm afraid currently there is no easy way to separate modules and main zsh.
We do not have well defined interfaces; even worse, with modules
cross-references you'd need extra stuff for every dependent module that is
needed to compile a module (ough ...). Also, chained dynamic loading is not
well supported on many plattforms.

What I am really considering is more Apache-like scheme. Modules can be
packaged separately but still need to be included in sources (registered?).
Every module is to large extend independent; you unpack it, do something
like

configure --with-zsh-dir=...

and it will modify zsh configuration to include this module (whatever it
means). It must not be configure (it is well possible that this should be
pre-configure task). After that you just do make and it will compile module
*and* recompile zsh main if needed.

But it is still very vague; and there are some other more urgent things
(module dependencies in the first place).

-andrej

RE: PATCH: zsh/pcre module

[Andrej Borsenkow]

>
> > How easy is it to package up modules separately in such a way that they
> > could be built and installed without needing to be unpacked over the zsh
> > source and compiled with it. Ideally, we would have a set of include
> > files which was all anyone needed to build a separate zsh module.
>
> Also, there would need to be a mechanism for dynamically scanning for
> autobins/autoparams/&c.  One could scan for module dependencies at the
> same time.
>
>

That is exactly what I have suggested - scan *.zmd files on startup and
register all auto*s found there.

-andrej

Re: PATCH: zsh/pcre module

[Bart Schaefer]

On Jul 3,  5:55pm, Oliver Kiddle wrote:
} Subject: Re: PATCH: zsh/pcre module
}
} How easy is it to package up modules separately in such a way that they
} could be built and installed without needing to be unpacked over the zsh
} source and compiled with it.

Not very easy, unfortunately.  The zsh.mdh file includes all the .epro
files, which are generated during the zsh build and may differ among
operating systems.

} Ideally, we would have a set of include
} files which was all anyone needed to build a separate zsh module.

That could work, but only for modules that didn't need to reference any
of the external symbols other than addbuiltin() etc.  That's a pretty
small subset of the current crop of modules.
 
} Does anyone remember what happened to the perl module someone posted
} quite a while back?

The URL was

    http://www.phydeaux.org/perl/zperl-0.01.patch.gz

I haven't looked to see if it's still there.

-- 
Bart Schaefer                                 Brass Lantern Enterprises
http://www.well.com/user/barts              http://www.brasslantern.com

Zsh: http://www.zsh.org | PHPerl Project: http://phperl.sourceforge.net   

Re: PATCH: zsh/pcre module

[Clint Adams]

>     http://www.phydeaux.org/perl/zperl-0.01.patch.gz
> 
> I haven't looked to see if it's still there.

www.phydeaux.org doesn't seem to exist anymore, but I managed to grab something from

 http://phydeaux.org/perl/zperl-0.01.patch.gz

Re: PATCH: zsh/pcre module

[Fletch]

>>>>> "Clint" == Clint Adams <clint@zsh.org> writes:

        [ Sorry this is a bit delayed, but the wife just gave
          birth to kid #2 and for some reason I haven't had as
          much time to keep up with email recently . . . :)    ]

    >> http://www.phydeaux.org/perl/zperl-0.01.patch.gz
    >> 
    >> I haven't looked to see if it's still there.

    Clint> www.phydeaux.org doesn't seem to exist anymore, but I

        Carp, I moved to djbdns and apparently missed some things.

    Clint> managed to grab something from

    Clint>  http://phydeaux.org/perl/zperl-0.01.patch.gz

        That should be the same thing.  I haven't touched that in a
long, long while (since 3.1.6 days), so I have no idea if it'll
cooperate with 4.0.2.


        And I like Oliver's sugguestion about an infrastructure for
building modules outside the zsh tree propper.  If there was a
framework along the same lines as perl's ExtUtils::MakeMaker module or
apache's apxs utility, then it'd be more feasable to setup a CZAN (a
la CPAN or CTAN) with seperately downloadable modules (my zperl, the
pcre module that was submitted, et al).

-- 
Fletch                | "If you find my answers frightening,       __`'/|
fletch@phydeaux.org   |  Vincent, you should cease askin'          \ o.O'
770 933-0600 x211(w)  |  scary questions." -- Jules                =(___)=
                      |                                               U