* Portability of fts() functions @ 2014-08-09 10:38 Paul Onyschuk 2014-08-09 15:49 ` Ingo Schwarze 0 siblings, 1 reply; 10+ messages in thread From: Paul Onyschuk @ 2014-08-09 10:38 UTC (permalink / raw) To: discuss mandocdb.c in v1.13.x introduced dependency on fts() family of functions. This will bite on non-BSD systems. Musl libc doesn't provide it at all, neither does systems of Solaris origin I guess, but then glibc does something even worse [1], relevant fts.h header [2]. AFAIK uClibc replicated behavior of glibc. [1] https://sourceware.org/bugzilla/show_bug.cgi?id=15838 [2] https://sourceware.org/git/?p=glibc.git;a=blob;f=io/fts.h -- Paul Onyschuk -- To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Portability of fts() functions 2014-08-09 10:38 Portability of fts() functions Paul Onyschuk @ 2014-08-09 15:49 ` Ingo Schwarze 2014-08-09 17:09 ` Paul Onyschuk 2014-08-10 10:53 ` Portability of fts() functions Paul Onyschuk 0 siblings, 2 replies; 10+ messages in thread From: Ingo Schwarze @ 2014-08-09 15:49 UTC (permalink / raw) To: Paul Onyschuk; +Cc: discuss Hi Paul, Paul Onyschuk wrote on Sat, Aug 09, 2014 at 12:38:27PM +0200: > mandocdb.c in v1.13.x introduced dependency on fts() family of > functions. This will bite on non-BSD systems. Ouch. I dimly remember that was mentioned before, but then it seems i forgot about it. :-( > Musl libc doesn't provide it at all, neither does systems of Solaris > origin I guess, but then glibc does something even worse [1], relevant > fts.h header [2]. AFAIK uClibc replicated behavior of glibc. > > [1] https://sourceware.org/bugzilla/show_bug.cgi?id=15838 > [2] https://sourceware.org/git/?p=glibc.git;a=blob;f=io/fts.h Uh-oh. Not pretty at all. I guess what is needed is a compat_fts.h/compat_fts.c just like for ohash(3). I fear that won't be something that can be done in a hurry, though. So it looks like for the 1.13.1 release, it's probably to late to fix the fts(3) issue, and systems not having it will have the choice of either running 1.13.1 with "BUILD_TARGETS += db-build" disabled (that is, without apropos/makewhatis) or stay with 1.12.4 until 1.13.2 comes out. Do you think that would be tolerable? Yours, Ingo -- To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Portability of fts() functions 2014-08-09 15:49 ` Ingo Schwarze @ 2014-08-09 17:09 ` Paul Onyschuk 2014-08-09 21:59 ` Ingo Schwarze 2014-08-10 1:23 ` man(1) replacement Ingo Schwarze 2014-08-10 10:53 ` Portability of fts() functions Paul Onyschuk 1 sibling, 2 replies; 10+ messages in thread From: Paul Onyschuk @ 2014-08-09 17:09 UTC (permalink / raw) To: Ingo Schwarze; +Cc: discuss On Sat, 9 Aug 2014 17:49:28 +0200 Ingo Schwarze wrote: > I guess what is needed is a compat_fts.h/compat_fts.c just like > for ohash(3). I fear that won't be something that can be done > in a hurry, though. > > So it looks like for the 1.13.1 release, it's probably to late > to fix the fts(3) issue, and systems not having it will have the > choice of either running 1.13.1 with "BUILD_TARGETS += db-build" > disabled (that is, without apropos/makewhatis) > or stay with 1.12.4 until 1.13.2 comes out. > > Do you think that would be tolerable? For systems missing fts(3) I would say yes. I would worry more about glibc scenario, especially if mdocml is packaged e.g. clean build on x86_64 system used by packager, where it could break otherwise, wasting someone's time. Using occasion: In file included from mansearch_const.c:20:0: manpath.h:36:1: error: unknown type name '__END_DECLS' mansearch_const.c is not including config.h before manpath.h, patch: XXX --- mansearch_const.c.orig +++ mansearch_const.c @@ -14,6 +14,10 @@ * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. */ +#ifdef HAVE_CONFIG_H +#include "config.h" +#endif + #include <sys/types.h> #include <stdint.h> XXX I think configure script should be guarded against standalone execution. Right now you can do "./configure && make" expecting usual behavior (if you forget about looking at INSTALL). Since in this case ${CC} won't be defined, script will try running commands starting with "-Wno-unused -Werror". All tests will fail, enabling all compat functions. I also have additional question. Are there any plans for providing man(1) command also? This would make mdocml a possible, standalone replacement for groff and man-db combination (typical in Linux distributions). -- Paul Onyschuk -- To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Portability of fts() functions 2014-08-09 17:09 ` Paul Onyschuk @ 2014-08-09 21:59 ` Ingo Schwarze 2014-08-09 23:26 ` Paul Onyschuk 2014-08-10 1:23 ` man(1) replacement Ingo Schwarze 1 sibling, 1 reply; 10+ messages in thread From: Ingo Schwarze @ 2014-08-09 21:59 UTC (permalink / raw) To: Paul Onyschuk; +Cc: discuss Hi Paul, Paul Onyschuk wrote on Sat, Aug 09, 2014 at 07:09:23PM +0200: > Ingo Schwarze wrote: >> I guess what is needed is a compat_fts.h/compat_fts.c just like >> for ohash(3). I fear that won't be something that can be done >> in a hurry, though. >> >> So it looks like for the 1.13.1 release, it's probably to late >> to fix the fts(3) issue, and systems not having it will have the >> choice of either running 1.13.1 with "BUILD_TARGETS += db-build" >> disabled (that is, without apropos/makewhatis) >> or stay with 1.12.4 until 1.13.2 comes out. >> >> Do you think that would be tolerable? > For systems missing fts(3) I would say yes. I would worry more about > glibc scenario, especially if mdocml is packaged e.g. clean build on > x86_64 system used by packager, where it could break otherwise, wasting > someone's time. Oh you mean if mandoc is compiled on a 32bit glibc platform with 32bit off_t but more than 2 billion files in one file system or files larger than 2 GB inside a manual tree, mandoc will compile all right, but i may crash at runtime due to the broken fts(3) implementation contained in glibc? Did i get that right? I'm not sure what i could do about that - both in the long term and as a quick fix for this release. What would you suggest? > Using occasion: > > In file included from mansearch_const.c:20:0: > manpath.h:36:1: error: unknown type name '__END_DECLS' > > mansearch_const.c is not including config.h before manpath.h, patch: > > XXX > --- mansearch_const.c.orig > +++ mansearch_const.c > @@ -14,6 +14,10 @@ > * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF > * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. > */ > +#ifdef HAVE_CONFIG_H > +#include "config.h" > +#endif > + > #include <sys/types.h> > #include <stdint.h> > > XXX Yes, Thomas Klausner reported that one, too, and i fixed it with exactly that patch in rc3. > I think configure script should be guarded against standalone > execution. Right now you can do "./configure && make" expecting usual > behavior (if you forget about looking at INSTALL). Since in this case > ${CC} won't be defined, script will try running commands starting with > "-Wno-unused -Werror". All tests will fail, enabling all compat > functions. Indeed, that sounds bad. I guess i won't change that for this release, though. I worry that if we are very unlucky, whatever guard i put in there might break on another system that was already tested and may not be tested again. For the next release, i have to keep this in mind. I might do some more cleanup related to configure, anyway. > I also have additional question. Are there any plans for providing > man(1) command also? This would make mdocml a possible, standalone > replacement for groff and man-db combination (typical in Linux > distributions). I will address that separately. Yours, Ingo -- To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Portability of fts() functions 2014-08-09 21:59 ` Ingo Schwarze @ 2014-08-09 23:26 ` Paul Onyschuk 2014-08-10 2:46 ` Ingo Schwarze 0 siblings, 1 reply; 10+ messages in thread From: Paul Onyschuk @ 2014-08-09 23:26 UTC (permalink / raw) To: Ingo Schwarze; +Cc: discuss On Sat, 9 Aug 2014 23:59:19 +0200 Ingo Schwarze wrote: > Oh you mean if mandoc is compiled on a 32bit glibc platform with 32bit > off_t but more than 2 billion files in one file system or files larger > than 2 GB inside a manual tree, mandoc will compile all right, > but i may crash at runtime due to the broken fts(3) implementation > contained in glibc? Did i get that right? I think you got it right. > I'm not sure what i could do about that - both in the long term > and as a quick fix for this release. What would you suggest? There isn't much you to do about for time now I guess. Explanation in readme and that "BUILD_TARGETS += db-build" won't compile on systems without fts(3) would do. > Indeed, that sounds bad. > > I guess i won't change that for this release, though. I worry that > if we are very unlucky, whatever guard i put in there might break > on another system that was already tested and may not be tested > again. > > For the next release, i have to keep this in mind. > I might do some more cleanup related to configure, anyway. What about something like that before 'set -e': echo "/* RUNNING ./CONFIGURE - SHOULD BE USED ONLY VIA MAKE, READ INSTALL */" Maybe that would be enough? -- Paul Onyschuk -- To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Portability of fts() functions 2014-08-09 23:26 ` Paul Onyschuk @ 2014-08-10 2:46 ` Ingo Schwarze 0 siblings, 0 replies; 10+ messages in thread From: Ingo Schwarze @ 2014-08-10 2:46 UTC (permalink / raw) To: Paul Onyschuk; +Cc: discuss Hi Paul, Paul Onyschuk wrote on Sun, Aug 10, 2014 at 01:26:53AM +0200: > On Sat, 9 Aug 2014 23:59:19 +0200 Ingo Schwarze wrote: >> Oh you mean if mandoc is compiled on a 32bit glibc platform with 32bit >> off_t but more than 2 billion files in one file system or files larger >> than 2 GB inside a manual tree, mandoc will compile all right, >> but i may crash at runtime due to the broken fts(3) implementation >> contained in glibc? Did i get that right? > I think you got it right. >> I'm not sure what i could do about that - both in the long term >> and as a quick fix for this release. What would you suggest? > There isn't much you to do about for time now I guess. Explanation in > readme and that "BUILD_TARGETS += db-build" won't compile on systems > without fts(3) would do. I've added a comment to the relevant place in the Makefile. >> Indeed, that sounds bad. >> >> I guess i won't change that for this release, though. I worry that >> if we are very unlucky, whatever guard i put in there might break >> on another system that was already tested and may not be tested >> again. >> >> For the next release, i have to keep this in mind. >> I might do some more cleanup related to configure, anyway. > What about something like that before 'set -e': > > echo "/* RUNNING ./CONFIGURE - SHOULD BE USED ONLY VIA MAKE, READ INSTALL */" > > Maybe that would be enough? Sounds reasonable for now and certainly not dangerous; done. Thanks, Ingo Index: Makefile =================================================================== RCS file: /usr/vhosts/mdocml.bsd.lv/cvs/mdocml/Makefile,v retrieving revision 1.434 diff -u -p -r1.434 Makefile --- Makefile 8 Aug 2014 23:47:21 -0000 1.434 +++ Makefile 10 Aug 2014 02:42:13 -0000 @@ -52,9 +52,10 @@ INSTALL_MAN = $(INSTALL_DATA) # --- user settings related to database support ------------------------ -# If you want to build without database support, for example to avoid -# the dependency on SQLite3, comment the following line. -# However, you won't get apropos(1) and makewhatis(8) in that case. +# Building apropos(1) and makewhatis(8) requires both SQLite3 and fts(3). +# To avoid those dependencies, comment the following line. +# Be careful: the fts(3) implementation in glibc is broken on 32bit +# machines, see: https://sourceware.org/bugzilla/show_bug.cgi?id=15838 # BUILD_TARGETS += db-build Index: configure =================================================================== RCS file: /usr/vhosts/mdocml.bsd.lv/cvs/mdocml/configure,v retrieving revision 1.7 diff -u -p -r1.7 configure --- configure 8 Aug 2014 23:47:21 -0000 1.7 +++ configure 10 Aug 2014 02:42:13 -0000 @@ -14,6 +14,8 @@ # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. +echo "/* RUNNING ./CONFIGURE - SHOULD BE USED ONLY VIA MAKE, READ INSTALL */" + set -e exec > config.h 2> config.log -- To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv ^ permalink raw reply [flat|nested] 10+ messages in thread
* man(1) replacement 2014-08-09 17:09 ` Paul Onyschuk 2014-08-09 21:59 ` Ingo Schwarze @ 2014-08-10 1:23 ` Ingo Schwarze 1 sibling, 0 replies; 10+ messages in thread From: Ingo Schwarze @ 2014-08-10 1:23 UTC (permalink / raw) To: Paul Onyschuk; +Cc: discuss Hi Paul, Paul Onyschuk wrote on Sat, Aug 09, 2014 at 07:09:23PM +0200: > I also have additional question. Are there any plans for providing > man(1) command also? This would make mdocml a possible, standalone > replacement for groff and man-db combination (typical in Linux > distributions). Well, that's almost ready, actually; not for 1.13.1, though. The file implementing it is manpage.c. Try this: make clean make sudo ./makewhatis make manpage ./manpage setitimer Ironically, there isn't a manpage for manpage(1). ;-) What it does is basically this: It takes the same options and arguments as apropos(1), except that the -O option is not available. If it finds exactly one result, it shows that manual, just like the traditional man(1). If it finds more than one result, it shows the list of manuals, similar to apropos(1), but with serial numbers in front. When running on a terminal, it interactively asks you to select one of the numbers, defaulting to number 1, then shows that manual. I dislike the interactive prompting so much that i never enabled it in the build, but when you say "make manpage", it does build and run. Obviously, commands like manpage ls are almost useless, and manpage Nm~^ls\$ which is equivalent to "man ls", is hard to type. Having reconsidered all this, here is what i will do for 1.13.2: * Provide one single binary program, integrating all the functionality of mandoc(1), man(1), apropos(1), and whatis(1). * Provide four argument input modes: - filenames: ["mandoc"] Each argument is a relative or absolute pathname to a file, like in traditional mandoc(1). - names: ["man"] Each argument is a complete, literal name of a manual, like in traditional man(1). - words: ["man -f"] Each argument is a literal string and will be matched against whole words in manual names, like in traditional whatis(1). - expressions: ["man -k"] Arguments are full apropos(1) expressions. As a special case, arguments containing neither '=' nor '~' are literal strings and will be matched against substrings in manual names and descriptions, like in traditional apropos(1). * Provide five output modes: - one: ["man"] Among the matching files, one will be selected according to traditional precedence rules, and this one file will be formatted and shown. If on a terminal, the command is man, and -c was not given, a pager is used. - all: ["man -a"] All matching files will be formatted and shown one after the other, using a pager as above. - filenames: ["man -w"] Only the filenames of all matching files are shown. - list: ["man -k", "man -f"] The names and descriptions of all matching files will be listed like in traditional apropos(1) and whatis(1). - ask: ["man -i"] If there is exactly one matching file, it will be formatted and shown. If there is more than one, the names and descriptions will be listed, and if on a terminal, the user will be asked for a choice, like in manpage(1). The following combinations are useful: arguments output command alias comment --------- ------ ------- ----- ------- filenames all mandoc no change names one man new in mandoc names ask man -i completely new names all man -a new in mandoc names filenames man -w new in mandoc words list man -f whatis no change words ask man -fi whatis -i completely new words all man -fa whatis -a completely new words filenames man -fw whatis -w completely new expressions list man -k apropos no change expressions ask man -ki apropos -i replaces manpage(1) expressions all man -ka apropos -a completely new expressions filenames man -kw apropos -w completely new That looks like a compact, consistent, and feature-complete interface: * man(1) defaults to arguments:names, output:one as before * -f switches to arguments:words, output:list as before * -k switches to arguments:expressions, output:list as before * -i switches to output:ask in all three cases (new) * -a switches to output:all in all three cases (new for -fk) * -w switches to output:filenames in all three cases (new for -fk) * output:one would be useless with -f and -k * all except output:all would be useless with mandoc(1) I'm not likely to turn into an avid user of -i, but in that complete feature set, i no longer dislike it so much that i feel unwilling to integrate it: in that set, it looks like a reasonable companion of the names/one standard mode. Going from names to expressions, the following naturally correspond to each other: man -w -> man -kw man -a -> man -ka man -> man -ki Without -i, there would be an ugly hole in the lower right corner of the table, no sane way to get from an expression search to the display of one single page... Arguably, -i alone is rarely useful, nobody will type "man -i chmod" instead of "man 2 chmod" (when remembering chmod as ambiguous) or instead of just "man chmod" (when forgetting about the ambiguity). So -i could be made to imply -k, making life easier for people who want to use man -ki. Then again, that makes the interface less consistent, and some people may wish to set "alias man='man -i'" as long as that doesn't imply -k; but nobody is likely to want -k by default. So i tend to have -i not imply -k. Implementing all this looks nearly trivial: Basically, all the required code is already available in main.c, apropos.c, and manpage.c, it merely needs some reshuffling and polishing. So, thanks for asking that question. :-) If you switch to that system when 1.13.2 comes out, you just have to remember one thing: It strictly requires keeping the mandoc.db(5) files up-to-date in all your manual trees. If you manually copy a new page to /usr/local/man/man1/foo.1 and forget to run "makewhatis -d /usr/local/man man1/foo.1" then "man foo" will *not* find your new page! And, admittedly, it causes some slowdown. Querying a database is not for free. On my notebook: $ time sh -c 'for i in `jot 100`; do man -w ksh; done > /dev/null' 0m1.47s real 0m0.32s user 0m1.12s system $ time sh -c 'for i in `jot 100`; do whatis ksh; done > /dev/null' 0m2.53s real 0m1.74s user 0m0.78s system $ time sh -c 'for i in `jot 100`; do mandoc cat.1; done > /dev/null' 0m0.58s real 0m0.24s user 0m0.29s system $ time sh -c 'for i in `jot 100`; do mandoc ksh.1; done > /dev/null' 0m5.12s real 0m4.08s user 0m0.97s system $ time sh -c 'for i in `jot 100`; do man ksh; done > /dev/null' 0m5.89s real 0m4.23s user 0m1.58s system So: - find a manual with traditional man: 15 milliseconds - find a manual in the SQL database: 25 milliseconds - format a small manual: 5 milliseconds - format a huge manual: 50 milliseconds That looks like roughly a 50% increase in man page serving time, 15+5=20 -> 25+5=30 milliseconds, very imprecisely. Not relevant on modern hardware, but likely noticable on a VAX... Yours, Ingo -- To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Portability of fts() functions 2014-08-09 15:49 ` Ingo Schwarze 2014-08-09 17:09 ` Paul Onyschuk @ 2014-08-10 10:53 ` Paul Onyschuk 2014-08-11 3:28 ` Ingo Schwarze 1 sibling, 1 reply; 10+ messages in thread From: Paul Onyschuk @ 2014-08-10 10:53 UTC (permalink / raw) To: Ingo Schwarze; +Cc: discuss On Sat, 9 Aug 2014 17:49:28 +0200 Ingo Schwarze wrote: > I guess what is needed is a compat_fts.h/compat_fts.c just like > for ohash(3). I fear that won't be something that can be done > in a hurry, though. This may be helpful in future. Libnbcompat from pkgsrc is providing portable fts(3) implementation under reasonable conditions (3-clause BSD license). http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/pkgtools/libnbcompat/files/nbcompat/ -- Paul Onyschuk -- To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: Portability of fts() functions 2014-08-10 10:53 ` Portability of fts() functions Paul Onyschuk @ 2014-08-11 3:28 ` Ingo Schwarze 0 siblings, 0 replies; 10+ messages in thread From: Ingo Schwarze @ 2014-08-11 3:28 UTC (permalink / raw) To: Paul Onyschuk; +Cc: discuss Hi Paul, Paul Onyschuk wrote on Sun, Aug 10, 2014 at 12:53:40PM +0200: > On Sat, 9 Aug 2014 17:49:28 +0200 Ingo Schwarze wrote: >> I guess what is needed is a compat_fts.h/compat_fts.c just like >> for ohash(3). I fear that won't be something that can be done >> in a hurry, though. > This may be helpful in future. Libnbcompat from pkgsrc is providing > portable fts(3) implementation under reasonable conditions (3-clause > BSD license). > > http://cvsweb.netbsd.org/bsdweb.cgi/ \ > pkgsrc/pkgtools/libnbcompat/files/nbcompat/ Thanks for the pointer! While i didn't use the NetBSD implementation itself - it seems to lack some bugfixes that the OpenBSD implementation has - the libnbcompat file was quite helpful for inspiration how to work around the lack of d_namlen and ALIGN/ALIGNBYTES on Linux. Anyway, the mdocml.bsd.lv repo now contains a fallback for fts(3) that works for me on OpenBSD and Linux. It will be contained in the future mandoc 1.13.2 release. Yours, Ingo -- To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv ^ permalink raw reply [flat|nested] 10+ messages in thread
* Portability of fts() functions @ 2014-08-09 9:54 Paul Onyschuk 0 siblings, 0 replies; 10+ messages in thread From: Paul Onyschuk @ 2014-08-09 9:54 UTC (permalink / raw) To: discuss mandocdb.c in v1.13.x introduced dependency on fts() family of functions. This will bite on non-BSD systems. Musl libc doesn't provide it at all, neither does systems of Solaris origin I guess, but then glibc does something even worse [1], relevant fts.h header [2]. AFAIK uClibc replicated behavior of glibc. [1] https://sourceware.org/bugzilla/show_bug.cgi?id=15838 [2] https://sourceware.org/git/?p=glibc.git;a=blob;f=io/fts.h -- Paul Onyschuk -- To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2014-08-11 3:29 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-08-09 10:38 Portability of fts() functions Paul Onyschuk 2014-08-09 15:49 ` Ingo Schwarze 2014-08-09 17:09 ` Paul Onyschuk 2014-08-09 21:59 ` Ingo Schwarze 2014-08-09 23:26 ` Paul Onyschuk 2014-08-10 2:46 ` Ingo Schwarze 2014-08-10 1:23 ` man(1) replacement Ingo Schwarze 2014-08-10 10:53 ` Portability of fts() functions Paul Onyschuk 2014-08-11 3:28 ` Ingo Schwarze -- strict thread matches above, loose matches on Subject: below -- 2014-08-09 9:54 Paul Onyschuk
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).