Re: call for testing: mandoc-1.13.4 release candidate

Re: call for testing: mandoc-1.13.4 release candidate

[Thomas Klausner]

[-- Attachment #1: Type: text/plain, Size: 301 bytes --]

I've tested 1.13.4rc1 on NetBSD-7.99.33 and it built fine, thank you.

There are a couple of patches in pkgsrc by joerg that, it seems, have
not been sent upstream. They remove the inclusion of getopt.h. At
least on NetBSD, the getopt(3) man page mentions that getopt is in
unistd.h.

Cheers,
 Thomas

[-- Attachment #2: patch-demandoc.c --]
[-- Type: text/plain, Size: 280 bytes --]

$NetBSD: patch-demandoc.c,v 1.1 2015/12/10 02:50:19 joerg Exp $

--- demandoc.c.orig	2015-12-07 15:20:07.413956576 +0000
+++ demandoc.c
@@ -20,7 +20,6 @@
 
 #include <assert.h>
 #include <ctype.h>
-#include <getopt.h>
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>

[-- Attachment #3: patch-mandocdb.c --]
[-- Type: text/plain, Size: 277 bytes --]

$NetBSD: patch-mandocdb.c,v 1.7 2015/12/10 02:50:19 joerg Exp $

--- mandocdb.c.orig	2015-12-07 15:20:09.513956486 +0000
+++ mandocdb.c
@@ -30,7 +30,6 @@
 #else
 #include "compat_fts.h"
 #endif
-#include <getopt.h>
 #include <limits.h>
 #include <stddef.h>
 #include <stdio.h>

[-- Attachment #4: patch-manpage.c --]
[-- Type: text/plain, Size: 281 bytes --]

$NetBSD: patch-manpage.c,v 1.1 2015/12/10 02:50:19 joerg Exp $

--- manpage.c.orig	2015-12-07 15:20:30.565955584 +0000
+++ manpage.c
@@ -20,7 +20,6 @@
 #include <sys/types.h>
 
 #include <assert.h>
-#include <getopt.h>
 #include <limits.h>
 #include <stdint.h>
 #include <stdio.h>

[-- Attachment #5: patch-mansearch.c --]
[-- Type: text/plain, Size: 286 bytes --]

$NetBSD: patch-mansearch.c,v 1.1 2015/12/10 02:50:19 joerg Exp $

--- mansearch.c.orig	2016-07-09 11:15:54.000000000 +0000
+++ mansearch.c
@@ -26,7 +26,6 @@
 #endif
 #include <errno.h>
 #include <fcntl.h>
-#include <getopt.h>
 #include <glob.h>
 #include <limits.h>
 #include <regex.h>

Re: call for testing: mandoc-1.13.4 release candidate

[Baptiste Daroussin FreeBSD]

[-- Attachment #1: Type: text/plain, Size: 2617 bytes --]

On Sat, Jul 09, 2016 at 03:11:59PM +0200, Ingo Schwarze wrote:
> Hi,
> 
> i'm planning to release mandoc-1.13.4 during the next week.
> 
> Since you are listed as downstream port/package maintainers
> and/or have sent in major contributions or many and/or important
> bug reports lately, i'm giving you the chance of testing
> the non-public release candidate on your favourite platform(s)
> in advance if you want to.  Please do not publicly update your
> ports/packages just yet.
> 
> To avoid flooding people's mailboxes, please don't reply to the
> full To: list, but rather publicly to <tech@mdocml.bsd.lv>
> (requires subscription) or privately to <devel@mdocml.bsd.lv>.
> 
> Both reports of successes and failures are welcome!

No issues at all on FreeBSD
> 
> I already did some testing myself on OpenBSD-6.0-current,
> OpenBSD-5.9-stable, OpenBSD-5.8-stable, Debian GNU/Linux, and Solaris
> 9, 10, and 11, but tests on all other supported platforms are still
> missing, and of course nothing is wrong with giving OpenBSD and
> Solaris another try if you want to.  I'll do the public release as
> soon as i have test reports from a sufficiently diverse set of
> systems.
> 
> Mandoc 1.13.4 is a regular maintenance release and also introduces
> a small number of new features; here is an extract from the NEWS
> file in the release candidate tarball:
> 
>  * man.conf(5): Design and implement a simpler configuration file format.
>  * man(1): Leverage less(1) -T and :t in a way resembling ctags(1)
>    to jump to the definitions of various terms inside manual pages.
>  * soelim(1): New implementation by Baptiste Daroussin.
>  * pledge(2): Use the OpenBSD privilege limitation syscall when available.
>  * man.cgi(8): Support short URIs like http://man.openbsd.org/mdoc .
>  * mandoc.css: Use one unified stylesheet rather than three different ones.
> 
> For a detailed list of changes, see the NEWS file
> inside the release candidate tarball:
> 
>   http://mdocml.bsd.lv/snapshots/mdocml-1.13.4rc1.tar.gz
> 
> Thank you for your support!
>   Ingo
> 
> 
> P.S.
> It seems likely that this will be the last release needing SQLite
> for database support.  I have started work on removing the dependency
> on SQLite without loss of functionality and without needing any
> replacement, relying on POSIX interfaces only.  So it is likely
> that we will have a 1.14.1 release this autumn that no longer needs
> SQLite.

Tell me more, I'm really interestedin that, is there a place were we can look at
this work in progress code?

Best regards,
Bapt

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: call for testing: mandoc-1.13.4 release candidate

[Svyatoslav Mishyn Crux]

[-- Attachment #1.1: Type: text/plain, Size: 2887 bytes --]

(Sat, 09 Jul 15:11) Ingo Schwarze:
> Hi,
> 
> i'm planning to release mandoc-1.13.4 during the next week.
> 
> Since you are listed as downstream port/package maintainers
> and/or have sent in major contributions or many and/or important
> bug reports lately, i'm giving you the chance of testing
> the non-public release candidate on your favourite platform(s)
> in advance if you want to.  Please do not publicly update your
> ports/packages just yet.
> 
> To avoid flooding people's mailboxes, please don't reply to the
> full To: list, but rather publicly to <tech@mdocml.bsd.lv>
> (requires subscription) or privately to <devel@mdocml.bsd.lv>.
> 
> Both reports of successes and failures are welcome!
> 
> I already did some testing myself on OpenBSD-6.0-current,
> OpenBSD-5.9-stable, OpenBSD-5.8-stable, Debian GNU/Linux, and Solaris
> 9, 10, and 11, but tests on all other supported platforms are still
> missing, and of course nothing is wrong with giving OpenBSD and
> Solaris another try if you want to.  I'll do the public release as
> soon as i have test reports from a sufficiently diverse set of
> systems.
> 
> Mandoc 1.13.4 is a regular maintenance release and also introduces
> a small number of new features; here is an extract from the NEWS
> file in the release candidate tarball:
> 
>  * man.conf(5): Design and implement a simpler configuration file format.
>  * man(1): Leverage less(1) -T and :t in a way resembling ctags(1)
>    to jump to the definitions of various terms inside manual pages.
>  * soelim(1): New implementation by Baptiste Daroussin.
>  * pledge(2): Use the OpenBSD privilege limitation syscall when available.
>  * man.cgi(8): Support short URIs like http://man.openbsd.org/mdoc .
>  * mandoc.css: Use one unified stylesheet rather than three different ones.
> 
> For a detailed list of changes, see the NEWS file
> inside the release candidate tarball:
> 
>   http://mdocml.bsd.lv/snapshots/mdocml-1.13.4rc1.tar.gz
> 
> Thank you for your support!
>   Ingo
> 
> 
> P.S.
> It seems likely that this will be the last release needing SQLite
> for database support.  I have started work on removing the dependency
> on SQLite without loss of functionality and without needing any
> replacement, relying on POSIX interfaces only.  So it is likely
> that we will have a 1.14.1 release this autumn that no longer needs
> SQLite.
> 
> Then again, if there is substantial demand, i might maintain the
> 1.13 branch for some time in parallel.  In either case, it seems
> important to me to get 1.13.4 as good and stable as possible to
> have a solid foundation to build on.  The code base itself is
> definitely very solid, it is quite similar to what we will ship in
> the OpenBSD 6.0 release this fall.

more(1) doesn't have "-T" option.
Patch attached.


-- 
https://www.juef.tk/

[-- Attachment #1.2: mdocml-do_not_use_tag_file_more.patch --]
[-- Type: text/plain, Size: 579 bytes --]

diff -Naur mdocml-1.13.4-a/main.c mdocml-1.13.4-b/main.c
--- mdocml-1.13.4-a/main.c	2016-07-09 14:15:54.000000000 +0300
+++ mdocml-1.13.4-b/main.c	2016-07-09 17:19:16.138769535 +0300
@@ -1055,11 +1055,11 @@
 			break;
 	}
 
-	/* For more(1) and less(1), use the tag file. */
+	/* For less(1), use the tag file. */
 
 	if ((cmdlen = strlen(argv[0])) >= 4) {
 		cp = argv[0] + cmdlen - 4;
-		if (strcmp(cp, "less") == 0 || strcmp(cp, "more") == 0) {
+		if (strcmp(cp, "less") == 0) {
 			argv[argc++] = mandoc_strdup("-T");
 			argv[argc++] = tag_files->tfn;
 		}

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: call for testing: mandoc-1.13.4 release candidate

[Anthony J. Bentley]

Baptiste Daroussin FreeBSD writes:
> > P.S.
> > It seems likely that this will be the last release needing SQLite
> > for database support.  I have started work on removing the dependency
> > on SQLite without loss of functionality and without needing any
> > replacement, relying on POSIX interfaces only.  So it is likely
> > that we will have a 1.14.1 release this autumn that no longer needs
> > SQLite.
> 
> Tell me more, I'm really interestedin that, is there a place were we can
> look at this work in progress code?

Ingo sent an initial patch to the OpenBSD tech@ list a few days ago:

https://marc.info/?l=openbsd-tech&m=146734380814620&w=2

-- 
Anthony J. Bentley
--
 To unsubscribe send an email to tech+unsubscribe@mdocml.bsd.lv

New db format (was Re: call for testing: mandoc-1.13.4 release candidate)

[Baptiste Daroussin]

[-- Attachment #1: Type: text/plain, Size: 1851 bytes --]

On Sat, Jul 09, 2016 at 08:31:58AM -0600, Anthony J. Bentley wrote:
> Baptiste Daroussin FreeBSD writes:
> > > P.S.
> > > It seems likely that this will be the last release needing SQLite
> > > for database support.  I have started work on removing the dependency
> > > on SQLite without loss of functionality and without needing any
> > > replacement, relying on POSIX interfaces only.  So it is likely
> > > that we will have a 1.14.1 release this autumn that no longer needs
> > > SQLite.
> > 
> > Tell me more, I'm really interestedin that, is there a place were we can
> > look at this work in progress code?
> 
> Ingo sent an initial patch to the OpenBSD tech@ list a few days ago:
> 
> https://marc.info/?l=openbsd-tech&m=146734380814620&w=2
> 
Thank you,

Beside minor portability issues (that I will provide fixes for once this patch
is official) I can say it works fine on FreeBSD :)

Tested on FreeBSD 12-CURRENT:

The 3 main "bad" feedbacks I got about mandoc was:
- time to create the db was long with sqlite with the new code it is longer:
  does from 4s to 11s in general for the base manpages but I do not consider
  that as a big issue given it is not run that often and the code is quite new
  and there are probably room of improvements

- size of the db (for embedded). With the new code:
  3.8M    /usr/share/man/mandoc.db
  1.6M    /usr/share/man/mandoc.new.db

  This is a big improvement

- apropos can be very slow before showing anything, with the new code, it is
  instant!
  old 'apropos ls':
          8.67 real         3.63 user         5.03 sys
  new 'apropos ls':
          0.02 real         0.02 user         0.00 sys


I notice some memory corruption in the ouput of apropos, I haven't dig into it
yet

Great work, I'm eager to see that code in!

Best regards,
Bapt

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: call for testing: mandoc-1.13.4 release candidate

[Ingo Schwarze]

Hi Thomas & Joerg,

Thomas Klausner NetBSD wrote on Sat, Jul 09, 2016 at 03:18:28PM +0200:

> I've tested 1.13.4rc1 on NetBSD-7.99.33 and it built fine, thank you.

Thank you!

> There are a couple of patches in pkgsrc by joerg that, it seems, have
> not been sent upstream. They remove the inclusion of getopt.h. At
> least on NetBSD, the getopt(3) man page mentions that getopt is in
> unistd.h.

OpenBSD manuals and header files agree, and more importantly,
so does POSIX.  Nothing break on Linux, Solaris 9, 10, or 11
either, so i committed the patch.

Thank you,
  Ingo


> $NetBSD: patch-demandoc.c,v 1.1 2015/12/10 02:50:19 joerg Exp $
> 
> --- demandoc.c.orig	2015-12-07 15:20:07.413956576 +0000
> +++ demandoc.c
> @@ -20,7 +20,6 @@
>  
>  #include <assert.h>
>  #include <ctype.h>
> -#include <getopt.h>
>  #include <stdio.h>
>  #include <stdlib.h>
>  #include <string.h>

> $NetBSD: patch-mandocdb.c,v 1.7 2015/12/10 02:50:19 joerg Exp $
> 
> --- mandocdb.c.orig	2015-12-07 15:20:09.513956486 +0000
> +++ mandocdb.c
> @@ -30,7 +30,6 @@
>  #else
>  #include "compat_fts.h"
>  #endif
> -#include <getopt.h>
>  #include <limits.h>
>  #include <stddef.h>
>  #include <stdio.h>

> $NetBSD: patch-manpage.c,v 1.1 2015/12/10 02:50:19 joerg Exp $
> 
> --- manpage.c.orig	2015-12-07 15:20:30.565955584 +0000
> +++ manpage.c
> @@ -20,7 +20,6 @@
>  #include <sys/types.h>
>  
>  #include <assert.h>
> -#include <getopt.h>
>  #include <limits.h>
>  #include <stdint.h>
>  #include <stdio.h>

> $NetBSD: patch-mansearch.c,v 1.1 2015/12/10 02:50:19 joerg Exp $
> 
> --- mansearch.c.orig	2016-07-09 11:15:54.000000000 +0000
> +++ mansearch.c
> @@ -26,7 +26,6 @@
>  #endif
>  #include <errno.h>
>  #include <fcntl.h>
> -#include <getopt.h>
>  #include <glob.h>
>  #include <limits.h>
>  #include <regex.h>
--
 To unsubscribe send an email to tech+unsubscribe@mdocml.bsd.lv

Re: New db format

[Ingo Schwarze]

Hi Baptiste,

Baptiste Daroussin FreeBSD wrote on Sat, Jul 09, 2016 at 05:29:27PM +0200:

> Beside minor portability issues (that I will provide fixes for once
> this patch is official)

You can already provide them now if you want to, and i'll include
them in my work-in-progress version and testing.

> I can say it works fine on FreeBSD :)
> Tested on FreeBSD 12-CURRENT:

Great to hear that.

My main worry is that i screwed alignment somewhere.  Did you test
on any machines with unusual alignment requirements?  Or is keeping
int32_t aligned to 4-byte boundaries guaranteed to be safe everywhere?

> The 3 main "bad" feedbacks I got about mandoc was:
> - time to create the db was long with sqlite with the new code
>   it is longer: does from 4s to 11s in general for the base manpages
>   but I do not consider that as a big issue given it is not run that
>   often and the code is quite new and there are probably room of
>   improvements

The SQLite version was optimized with gprof(1).  I can try again
for the new code.  But mandoc's makewhatis(8) will always remain
much slower than traditional makewhatis(8) because it parses the
complete manuals rather than just the NAME sections.

> - size of the db (for embedded). With the new code:
>   3.8M    /usr/share/man/mandoc.db
>   1.6M    /usr/share/man/mandoc.new.db
> 
>   This is a big improvement
> 
> - apropos can be very slow before showing anything, with the new code,
>   it is instant!
>   old 'apropos ls':
>           8.67 real         3.63 user         5.03 sys

That is surprising.  Yes, mandoc's SQLite based apropos(1) is slower
than the traditional BSD apropos(1) - no big surprise given the
much larger database.  But i have never seen it to be *that* slow
on OpenBSD, not even for complex queries accessing multiple macro
keys and using -a and -o; certainly not for a plain Nm,Nd access
like 'apropos ls' - that was always reasonably fast.  But now isn't
the time for making apropos-1.13 faster any longer...  :-)

schwarze@fantadrom $ dmesg
OpenBSD 5.8-stable (GENERIC) #1: Fri Oct 30 18:55:29 EST 2015
cpu0: Intel(R) Pentium(R) 4 CPU 2.40GHz ("GenuineIntel" 686-class) 2.41 GHz
real mem  = 3203874816 (3055MB)
[...]
schwarze@fantadrom $ time apropos ls
[...]
    0m00.19s real     0m00.10s user     0m00.08s system

schwarze@man $ dmesg
OpenBSD 5.9 (GENERIC.MP) #1888: Fri Feb 26 01:20:19 MST 2016
cpu0: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, 3093.47 MHz
cpu1: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, 3092.98 MHz
cpu2: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, 3092.98 MHz
cpu3: Intel(R) Xeon(R) CPU E31220 @ 3.10GHz, 3092.98 MHz
real mem = 8541536256 (8145MB)
[...]
schwarze@man $ time apropos ls
[...]
    0m00.02s real     0m00.01s user     0m00.01s system

schwarze@isnote $ dmesg
OpenBSD 6.0-beta (GENERIC.MP) #1875: Sun Jun 19 11:51:07 MDT 2016
cpu0: Genuine Intel(R) CPU T2300 @ 1.66GHz ("GenuineIntel" 686-class) 1.67 GHz
cpu1: Genuine Intel(R) CPU T2300 @ 1.66GHz ("GenuineIntel" 686-class) 1.67 GHz
real mem  = 3211083776 (3062MB)
[...]
schwarze@isnote $ time apropos ls
[...]
    0m00.22s real     0m00.09s user     0m00.07s system

>   new 'apropos ls':
>           0.02 real         0.02 user         0.00 sys
> 
> I notice some memory corruption in the ouput of apropos,
> I haven't dig into it yet

That might already be fixed; i just uploaded the current version
of the patch here:

  http://mdocml.bsd.lv/snapshots/mdocml-1.14.0.06.patch
  http://mdocml.bsd.lv/snapshots/mdocml-1.14.0.06.regress.tgz
  http://mdocml.bsd.lv/snapshots/mdocml-1.14.0.06.log.txt

THIS IS NOT A RELEASE AND NOT INTENDED FOR PRODUCTION.
This code is likely to be still buggy.

> Great work, I'm eager to see that code in!

Thanks for your support!

Yours,
  Ingo
--
 To unsubscribe send an email to tech+unsubscribe@mdocml.bsd.lv

Re: call for testing: mandoc-1.13.4 release candidate

[Svyatoslav Mishyn]

[-- Attachment #1: Type: text/plain, Size: 203 bytes --]

(Sat, 09 Jul 15:11) Ingo Schwarze:
>  * man.cgi(8): Support short URIs like http://man.openbsd.org/mdoc .

But why doesn't the link http://man.openbsd.org/mandoc work?


-- 
https://www.juef.tk/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: call for testing: mandoc-1.13.4 release candidate

[Svyatoslav Mishyn Crux]

[-- Attachment #1: Type: text/plain, Size: 780 bytes --]

(Sat, 09 Jul 15:11) Ingo Schwarze:
> Both reports of successes and failures are welcome!

Also some manual pages I cannot open.

/home/juef: echo $MANPATH
/usr/share/man:/usr/man

/home/juef: sudo /usr/sbin/makewhatis -a /usr/share/man
/home/juef: sudo /usr/sbin/makewhatis -a /usr/man

/home/juef: ls -l /usr/share/man/man2/open.2.gz
-rw-r--r-- 1 root root 15K Чтв, 12 Май 2016 11:55:29 /usr/share/man/man2/open.2.gz

/home/juef: man 2 open > tmp/e
man: man2/open.2: ERROR: No such file or directory

/home/juef: cat tmp/e
()                                                                          ()



See the file man2/open.2.




                                 July 9, 2016                               ()


-- 
https://www.juef.tk/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: call for testing: mandoc-1.13.4 release candidate

[Ingo Schwarze]

Hi Svyatoslav,

Svyatoslav Mishyn Crux wrote on Sat, Jul 09, 2016 at 09:21:39PM +0300:
> (Sat, 09 Jul 15:11) Ingo Schwarze:

>>  * man.cgi(8): Support short URIs like http://man.openbsd.org/mdoc .

> But why doesn't the link http://man.openbsd.org/mandoc work?

Thanks for reminding me, fix committed and installed, see below.
  Ingo


Log Message:
-----------
Do not treat PATH_INFO as a complete path if it doesn't contain
a manpath.  For example, this makes http://man.openbsd.org/mandoc
work as expected.
Bug reported by tb@, reminded by Svyatoslav Mishyn.

Modified Files:
--------------
    mdocml:
        cgi.c

Revision Data
-------------
Index: cgi.c
===================================================================
RCS file: /home/cvs/mdocml/mdocml/cgi.c,v
retrieving revision 1.132
retrieving revision 1.133
diff -Lcgi.c -Lcgi.c -u -p -r1.132 -r1.133
--- cgi.c
+++ cgi.c
@@ -1026,7 +1026,7 @@ main(void)
 
 	if (*path != '\0') {
 		parse_path_info(&req, path);
-		if (access(path, F_OK) == -1)
+		if (req.q.manpath == NULL || access(path, F_OK) == -1)
 			path = "";
 	} else if ((querystring = getenv("QUERY_STRING")) != NULL)
 		parse_query_string(&req, querystring);
--
 To unsubscribe send an email to tech+unsubscribe@mdocml.bsd.lv

Re: call for testing: mandoc-1.13.4 release candidate

[Svyatoslav Mishyn]

[-- Attachment #1: Type: text/plain, Size: 1107 bytes --]

(Sat, 09 Jul 21:55) Svyatoslav Mishyn:
> (Sat, 09 Jul 15:11) Ingo Schwarze:
> > Both reports of successes and failures are welcome!
> 
> Also some manual pages I cannot open.
> 
> /home/juef: echo $MANPATH
> /usr/share/man:/usr/man
> 
> /home/juef: sudo /usr/sbin/makewhatis -a /usr/share/man
> /home/juef: sudo /usr/sbin/makewhatis -a /usr/man
> 
> /home/juef: ls -l /usr/share/man/man2/open.2.gz
> -rw-r--r-- 1 root root 15K Чтв, 12 Май 2016 11:55:29 /usr/share/man/man2/open.2.gz
> 
> /home/juef: man 2 open > tmp/e
> man: man2/open.2: ERROR: No such file or directory
> 
> /home/juef: cat tmp/e
> ()                                                                          ()
> 
> 
> 
> See the file man2/open.2.
> 
> 
> 
> 
>                                  July 9, 2016                               ()

If I remove openat.2.gz and creat.2.gz, it works.

/home/juef: zcat /usr/share/man/man2/openat.2.gz
.so man2/open.2
/home/juef: zcat /usr/share/man/man2/creat.2.gz
.so man2/open.2


There wasn't such problem in v1.13.3.


-- 
https://www.juef.tk/

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 801 bytes --]

Re: call for testing: mandoc-1.13.4 release candidate

[Ingo Schwarze]

Hi Svyatoslav,

Svyatoslav Mishyn Crux wrote on Sun, Jul 10, 2016 at 01:46:00PM +0300:
> (Sat, 09 Jul 21:55) Svyatoslav Mishyn:

>> Also some manual pages I cannot open.
>> 
>> /home/juef: man 2 open > tmp/e
>> man: man2/open.2: ERROR: No such file or directory
>> 
>> /home/juef: cat tmp/e
>> ()                                                               ()
>> 
>> See the file man2/open.2.
>> 
>>                           July 9, 2016                           ()

> If I remove openat.2.gz and creat.2.gz, it works.
> 
> /home/juef: zcat /usr/share/man/man2/openat.2.gz
> .so man2/open.2
> 
> There wasn't such problem in v1.13.3.

Whoa, good catch, thanks!
Fixed by the commit below.

Yours,
  Ingo


Log Message:
-----------
Fix a nasty typo that prevented .so links to gziped manuals 
from working in the absence of a mandoc.db(5) database.
Found the hard way by Svyatoslav Mishyn on Crux Linux.

Modified Files:
--------------
    mdocml:
        read.c

Revision Data
-------------
Index: read.c
===================================================================
RCS file: /home/cvs/mdocml/mdocml/read.c,v
retrieving revision 1.148
retrieving revision 1.149
diff -Lread.c -Lread.c -u -p -r1.148 -r1.149
--- read.c
+++ read.c
@@ -798,7 +798,7 @@ mparse_open(struct mparse *curp, const c
 
 	if ( ! curp->gzip) {
 		mandoc_asprintf(&cp, "%s.gz", file);
-		fd = open(file, O_RDONLY);
+		fd = open(cp, O_RDONLY);
 		free(cp);
 		if (fd != -1) {
 			curp->gzip = 1;
--
 To unsubscribe send an email to tech+unsubscribe@mdocml.bsd.lv

Re: call for testing: mandoc-1.13.4 release candidate

[Ingo Schwarze]

Hi Svyatoslav,

Svyatoslav Mishyn Crux wrote on Sat, Jul 09, 2016 at 05:28:20PM +0300:

> more(1) doesn't have "-T" option.
> Patch attached.

Committed, thanks!
  Ingo


> diff -Naur mdocml-1.13.4-a/main.c mdocml-1.13.4-b/main.c
> --- mdocml-1.13.4-a/main.c	2016-07-09 14:15:54.000000000 +0300
> +++ mdocml-1.13.4-b/main.c	2016-07-09 17:19:16.138769535 +0300
> @@ -1055,11 +1055,11 @@
>  			break;
>  	}
>  
> -	/* For more(1) and less(1), use the tag file. */
> +	/* For less(1), use the tag file. */
>  
>  	if ((cmdlen = strlen(argv[0])) >= 4) {
>  		cp = argv[0] + cmdlen - 4;
> -		if (strcmp(cp, "less") == 0 || strcmp(cp, "more") == 0) {
> +		if (strcmp(cp, "less") == 0) {
>  			argv[argc++] = mandoc_strdup("-T");
>  			argv[argc++] = tag_files->tfn;
>  		}
--
 To unsubscribe send an email to tech+unsubscribe@mdocml.bsd.lv

Re: call for testing: mandoc-1.13.4 release candidate

[Christian Neukirchen]

Ingo Schwarze <schwarze@usta.de> writes:

> Both reports of successes and failures are welcome!

The RC works fine on Void Linux (glibc and musl) and addresses many
bugs found since the last release, thanks!

Two small nits:

- It would be nice to support the LDFLAGS variable in the Makefile
  (soelim doesn't use any variable at all, else LDADD could be misused
  for that...)

- It would be nice to make the default MANPAGER configurable at
  compile time, since more(1) on non-BSD systems is an unusable
  default (no back scrolling etc.)

Thanks,
-- 
Christian Neukirchen  <chneukirchen@gmail.com>  http://chneukirchen.org
--
 To unsubscribe send an email to tech+unsubscribe@mdocml.bsd.lv

Re: call for testing: mandoc-1.13.4 release candidate

[Ingo Schwarze]

Hi Christian,

Christian Neukirchen wrote on Sun, Jul 10, 2016 at 04:27:14PM +0200:

> The RC works fine on Void Linux (glibc and musl) and addresses many
> bugs found since the last release, thanks!

Thanks for testing!

> Two small nits:
> 
> - It would be nice to support the LDFLAGS variable in the Makefile

What is that needed for?  I don't like adding gratuitious features
and would prefer to understand the actual purpose in order to find
a solution that is as simple as possible.

>   (soelim doesn't use any variable at all, else LDADD could be misused
>   for that...)

Maybe adding ${LDADD} to the soelim ${CC} line is a good solution?
It may depend on what the actual problem is.

> - It would be nice to make the default MANPAGER configurable at
>   compile time, since more(1) on non-BSD systems is an unusable
>   default (no back scrolling etc.)

Hmm, POSIX slightly favours having more(1) as the default in man(1)
but explicitly allows a system to use a different pager by default.
So maybe.  But implementing it is a bit of a bother because it
requires patching the file man.1 at configuration time.  I'm not
sure yet whether i want to do that.  I won't do it for the 1.13.4
release, it would be too intrusive at this point.

If you prepare a port for an operating system where the system more(1)
is unusable, i'd recommend adding local patches to the appropriate
places in main.c and man.1 for now.

Yours,
  Ingo
--
 To unsubscribe send an email to tech+unsubscribe@mdocml.bsd.lv

Re: call for testing: mandoc-1.13.4 release candidate

[Christian Neukirchen]

Ingo Schwarze <schwarze@usta.de> writes:

> Hi Christian,
>
> Christian Neukirchen wrote on Sun, Jul 10, 2016 at 04:27:14PM +0200:
>
>> The RC works fine on Void Linux (glibc and musl) and addresses many
>> bugs found since the last release, thanks!
>
> Thanks for testing!
>
>> Two small nits:
>> 
>> - It would be nice to support the LDFLAGS variable in the Makefile
>
> What is that needed for?  I don't like adding gratuitious features
> and would prefer to understand the actual purpose in order to find
> a solution that is as simple as possible.

We build programs on Void using hardening, and need to pass special
linker flags for that (add RELRO and BIND_NOW).

>>   (soelim doesn't use any variable at all, else LDADD could be misused
>>   for that...)
>
> Maybe adding ${LDADD} to the soelim ${CC} line is a good solution?
> It may depend on what the actual problem is.

I can live with that.

Else we patch the Makefile, not a big deal either.

>> - It would be nice to make the default MANPAGER configurable at
>>   compile time, since more(1) on non-BSD systems is an unusable
>>   default (no back scrolling etc.)
>
> Hmm, POSIX slightly favours having more(1) as the default in man(1)
> but explicitly allows a system to use a different pager by default.
> So maybe.  But implementing it is a bit of a bother because it
> requires patching the file man.1 at configuration time.  I'm not
> sure yet whether i want to do that.  I won't do it for the 1.13.4
> release, it would be too intrusive at this point.

Perhaps a macro could be used to refer to the default pager, then you
just need to change one line?

> If you prepare a port for an operating system where the system more(1)
> is unusable, i'd recommend adding local patches to the appropriate
> places in main.c and man.1 for now.

I currently use

        sed -i 's,bin/more,bin/less,g' main.c *.1
        sed -i 's,.Xr more,.Xr less,g' *.1
        sed -i 's,"more","less",g' manpage.c

Thanks,
-- 
Christian Neukirchen  <chneukirchen@gmail.com>  http://chneukirchen.org
--
 To unsubscribe send an email to tech+unsubscribe@mdocml.bsd.lv

Re: call for testing: mandoc-1.13.4 release candidate

[Ingo Schwarze]

Hi Christian,

Christian Neukirchen wrote on Sun, Jul 10, 2016 at 06:48:40PM +0200:
> Ingo Schwarze <schwarze@usta.de> writes:
>> Christian Neukirchen wrote on Sun, Jul 10, 2016 at 04:27:14PM +0200:

>>> - It would be nice to support the LDFLAGS variable in the Makefile

>> What is that needed for?  I don't like adding gratuitious features
>> and would prefer to understand the actual purpose in order to find
>> a solution that is as simple as possible.

> We build programs on Void using hardening, and need to pass special
> linker flags for that (add RELRO and BIND_NOW).

Actually, that seems like a useful idea to me, certainly not
gratuitious.  Supporting that by real LDFLAGS support in ./configure.local
and in the Makefile seemed cleanest to me, so i just added it, see
the commit below.

>>> - It would be nice to make the default MANPAGER configurable at
>>>   compile time, since more(1) on non-BSD systems is an unusable
>>>   default (no back scrolling etc.)

>> Hmm, POSIX slightly favours having more(1) as the default in man(1)
>> but explicitly allows a system to use a different pager by default.
>> So maybe.  But implementing it is a bit of a bother because it
>> requires patching the file man.1 at configuration time.  I'm not
>> sure yet whether i want to do that.  I won't do it for the 1.13.4
>> release, it would be too intrusive at this point.

> Perhaps a macro could be used to refer to the default pager, then you
> just need to change one line?

Macros in mdoc(7)?  That's possible, but ugly.

>> If you prepare a port for an operating system where the system more(1)
>> is unusable, i'd recommend adding local patches to the appropriate
>> places in main.c and man.1 for now.

> I currently use
> 
>         sed -i 's,bin/more,bin/less,g' main.c *.1
>         sed -i 's,.Xr more,.Xr less,g' *.1
>         sed -i 's,"more","less",g' manpage.c

Sure, it's fine if you do something like that locally for now.

Yours,
  Ingo


Log Message:
-----------
support LDFLAGS;
suggested by Christian Neukirchen <chneukirchen at gmail dot com>

Modified Files:
--------------
    mdocml:
        Makefile
        configure
        configure.local.example

Revision Data
-------------
Index: configure
===================================================================
RCS file: /home/cvs/mdocml/mdocml/configure,v
retrieving revision 1.38
retrieving revision 1.39
diff -Lconfigure -Lconfigure -u -p -r1.38 -r1.39
--- configure
+++ configure
@@ -37,6 +37,7 @@ OSNAME=
 CC=`printf "all:\\n\\t@echo \\\$(CC)\\n" | make -f -`
 CFLAGS="-g -W -Wall -Wstrict-prototypes -Wno-unused-parameter -Wwrite-strings"
 LDADD=
+LDFLAGS=
 LD_OHASH=
 LD_SQLITE3=
 STATIC="-static"
@@ -428,6 +429,7 @@ INSTALL_TARGETS	= ${INSTALL_TARGETS}
 CC		= ${CC}
 CFLAGS		= ${CFLAGS}
 LDADD		= ${LDADD}
+LDFLAGS		= ${LDFLAGS}
 STATIC		= ${STATIC}
 PREFIX		= ${PREFIX}
 BINDIR		= ${BINDIR}
Index: configure.local.example
===================================================================
RCS file: /home/cvs/mdocml/mdocml/configure.local.example,v
retrieving revision 1.11
retrieving revision 1.12
diff -Lconfigure.local.example -Lconfigure.local.example -u -p -r1.11 -r1.12
--- configure.local.example
+++ configure.local.example
@@ -131,10 +131,17 @@ BINM_SOELIM=msoelim		# default is "soeli
 
 LD_OHASH="-lutil"
 
-# Some platforms may need additional linker flags that are not autodetected.
+# Some platforms may need additional linker flags to link against libmandoc
+# that are not autodetected.
 # For example, Solaris 9 and 10 need -lrt for nanosleep(2).
 
 LDADD="-lrt"
+
+# Some systems may want to set additional linker flags for all the
+# binaries, not only for those using libmandoc, for example for
+# hardening options.
+
+LDFLAGS="-Wl,-z,relro"
 
 # It is possible to change the utility program used for installation
 # and the modes files are installed with.  The defaults are:
Index: Makefile
===================================================================
RCS file: /home/cvs/mdocml/mdocml/Makefile,v
retrieving revision 1.486
retrieving revision 1.487
diff -LMakefile -LMakefile -u -p -r1.486 -r1.487
--- Makefile
+++ Makefile
@@ -408,19 +408,19 @@ libmandoc.a: $(COMPAT_OBJS) $(LIBMANDOC_
 	ar rs $@ $(COMPAT_OBJS) $(LIBMANDOC_OBJS)
 
 mandoc: $(MAIN_OBJS) libmandoc.a
-	$(CC) -o $@ $(MAIN_OBJS) libmandoc.a $(LDADD)
+	$(CC) -o $@ $(LDFLAGS) $(MAIN_OBJS) libmandoc.a $(LDADD)
 
 manpage: $(MANPAGE_OBJS) libmandoc.a
-	$(CC) -o $@ $(MANPAGE_OBJS) libmandoc.a $(LDADD)
+	$(CC) -o $@ $(LDFLAGS) $(MANPAGE_OBJS) libmandoc.a $(LDADD)
 
 man.cgi: $(CGI_OBJS) libmandoc.a
-	$(CC) $(STATIC) -o $@ $(CGI_OBJS) libmandoc.a $(LDADD)
+	$(CC) $(STATIC) -o $@ $(LDFLAGS) $(CGI_OBJS) libmandoc.a $(LDADD)
 
 demandoc: $(DEMANDOC_OBJS) libmandoc.a
-	$(CC) -o $@ $(DEMANDOC_OBJS) libmandoc.a $(LDADD)
+	$(CC) -o $@ $(LDFLAGS) $(DEMANDOC_OBJS) libmandoc.a $(LDADD)
 
 soelim: $(SOELIM_OBJS)
-	$(CC) -o $@ $(SOELIM_OBJS)
+	$(CC) -o $@ $(LDFLAGS) $(SOELIM_OBJS)
 
 # --- maintainer targets ---
 
--
 To unsubscribe send an email to tech+unsubscribe@mdocml.bsd.lv

Re: New db format

[Baptiste Daroussin]

[-- Attachment #1: Type: text/plain, Size: 1577 bytes --]

On Sat, Jul 09, 2016 at 06:12:51PM +0200, Ingo Schwarze wrote:
> Hi Baptiste,
> 
> Baptiste Daroussin FreeBSD wrote on Sat, Jul 09, 2016 at 05:29:27PM +0200:
> 
> > Beside minor portability issues (that I will provide fixes for once
> > this patch is official)
> 
> You can already provide them now if you want to, and i'll include
> them in my work-in-progress version and testing.
> 
> > I can say it works fine on FreeBSD :)
> > Tested on FreeBSD 12-CURRENT:
> 
> Great to hear that.
> 
> My main worry is that i screwed alignment somewhere.  Did you test
> on any machines with unusual alignment requirements?  Or is keeping
> int32_t aligned to 4-byte boundaries guaranteed to be safe everywhere?
> 
> > The 3 main "bad" feedbacks I got about mandoc was:
> > - time to create the db was long with sqlite with the new code
> >   it is longer: does from 4s to 11s in general for the base manpages
> >   but I do not consider that as a big issue given it is not run that
> >   often and the code is quite new and there are probably room of
> >   improvements
> 
> The SQLite version was optimized with gprof(1).  I can try again
> for the new code.  But mandoc's makewhatis(8) will always remain
> much slower than traditional makewhatis(8) because it parses the
> complete manuals rather than just the NAME sections.

After some profiling of the new code, it appears that most of the time is spent
on dba_array_get and dba_array_next

The flamegraph of makewhatis

https://people.freebsd.org/~bapt/makewhatis.svg

Best regards,
Bapt

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]