Portability of fts() functions

Portability of fts() functions

[Paul Onyschuk]

mandocdb.c in v1.13.x introduced dependency on fts() family of
functions.  This will bite on non-BSD systems.

Musl libc doesn't provide it at all, neither does systems of Solaris
origin I guess, but then glibc does something even worse [1], relevant
fts.h header [2].  AFAIK uClibc replicated behavior of glibc.

[1] https://sourceware.org/bugzilla/show_bug.cgi?id=15838
[2] https://sourceware.org/git/?p=glibc.git;a=blob;f=io/fts.h

-- 
Paul Onyschuk
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: Portability of fts() functions

[Ingo Schwarze]

Hi Paul,

Paul Onyschuk wrote on Sun, Aug 10, 2014 at 12:53:40PM +0200:
> On Sat, 9 Aug 2014 17:49:28 +0200 Ingo Schwarze wrote:

>> I guess what is needed is a compat_fts.h/compat_fts.c just like
>> for ohash(3).  I fear that won't be something that can be done
>> in a hurry, though.

> This may be helpful in future.  Libnbcompat from pkgsrc is providing
> portable fts(3) implementation under reasonable conditions (3-clause
> BSD license).
> 
> http://cvsweb.netbsd.org/bsdweb.cgi/ \
> pkgsrc/pkgtools/libnbcompat/files/nbcompat/

Thanks for the pointer!

While i didn't use the NetBSD implementation itself - it seems to
lack some bugfixes that the OpenBSD implementation has - the
libnbcompat file was quite helpful for inspiration how to work around
the lack of d_namlen and ALIGN/ALIGNBYTES on Linux.

Anyway, the mdocml.bsd.lv repo now contains a fallback for fts(3)
that works for me on OpenBSD and Linux.  It will be contained in
the future mandoc 1.13.2 release.

Yours,
  Ingo
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: Portability of fts() functions

[Paul Onyschuk]

On Sat, 9 Aug 2014 17:49:28 +0200
Ingo Schwarze wrote:

> I guess what is needed is a compat_fts.h/compat_fts.c just like
> for ohash(3).  I fear that won't be something that can be done
> in a hurry, though.

This may be helpful in future.  Libnbcompat from pkgsrc is providing
portable fts(3) implementation under reasonable conditions (3-clause
BSD license).

http://cvsweb.netbsd.org/bsdweb.cgi/pkgsrc/pkgtools/libnbcompat/files/nbcompat/

-- 
Paul Onyschuk
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: Portability of fts() functions

[Ingo Schwarze]

Hi Paul,

Paul Onyschuk wrote on Sun, Aug 10, 2014 at 01:26:53AM +0200:
> On Sat, 9 Aug 2014 23:59:19 +0200 Ingo Schwarze wrote:

>> Oh you mean if mandoc is compiled on a 32bit glibc platform with 32bit
>> off_t but more than 2 billion files in one file system or files larger
>> than 2 GB inside a manual tree, mandoc will compile all right,
>> but i may crash at runtime due to the broken fts(3) implementation
>> contained in glibc?  Did i get that right?

> I think you got it right.
 
>> I'm not sure what i could do about that - both in the long term
>> and as a quick fix for this release.  What would you suggest?

> There isn't much you to do about for time now I guess.  Explanation in
> readme and  that "BUILD_TARGETS += db-build" won't compile on systems
> without fts(3) would do.

I've added a comment to the relevant place in the Makefile.

>> Indeed, that sounds bad.
>> 
>> I guess i won't change that for this release, though.  I worry that
>> if we are very unlucky, whatever guard i put in there might break
>> on another system that was already tested and may not be tested
>> again.
>> 
>> For the next release, i have to keep this in mind.
>> I might do some more cleanup related to configure, anyway.

> What about something like that before 'set -e':
> 
> echo "/* RUNNING ./CONFIGURE - SHOULD BE USED ONLY VIA MAKE, READ INSTALL */"
> 
> Maybe that would be enough? 

Sounds reasonable for now and certainly not dangerous; done.

Thanks,
  Ingo


Index: Makefile
===================================================================
RCS file: /usr/vhosts/mdocml.bsd.lv/cvs/mdocml/Makefile,v
retrieving revision 1.434
diff -u -p -r1.434 Makefile
--- Makefile	8 Aug 2014 23:47:21 -0000	1.434
+++ Makefile	10 Aug 2014 02:42:13 -0000
@@ -52,9 +52,10 @@ INSTALL_MAN	 = $(INSTALL_DATA)
 
 # --- user settings related to database support ------------------------
 
-# If you want to build without database support, for example to avoid
-# the dependency on SQLite3, comment the following line.
-# However, you won't get apropos(1) and makewhatis(8) in that case.
+# Building apropos(1) and makewhatis(8) requires both SQLite3 and fts(3).
+# To avoid those dependencies, comment the following line.
+# Be careful: the fts(3) implementation in glibc is broken on 32bit
+# machines, see: https://sourceware.org/bugzilla/show_bug.cgi?id=15838
 #
 BUILD_TARGETS	+= db-build
 
Index: configure
===================================================================
RCS file: /usr/vhosts/mdocml.bsd.lv/cvs/mdocml/configure,v
retrieving revision 1.7
diff -u -p -r1.7 configure
--- configure	8 Aug 2014 23:47:21 -0000	1.7
+++ configure	10 Aug 2014 02:42:13 -0000
@@ -14,6 +14,8 @@
 # ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
 # OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 
+echo "/* RUNNING ./CONFIGURE - SHOULD BE USED ONLY VIA MAKE, READ INSTALL */"
+
 set -e
 exec > config.h 2> config.log
 
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: Portability of fts() functions

[Paul Onyschuk]

On Sat, 9 Aug 2014 23:59:19 +0200
Ingo Schwarze wrote:

> Oh you mean if mandoc is compiled on a 32bit glibc platform with 32bit
> off_t but more than 2 billion files in one file system or files larger
> than 2 GB inside a manual tree, mandoc will compile all right,
> but i may crash at runtime due to the broken fts(3) implementation
> contained in glibc?  Did i get that right?

I think you got it right.

> I'm not sure what i could do about that - both in the long term
> and as a quick fix for this release.  What would you suggest?

There isn't much you to do about for time now I guess.  Explanation in
readme and  that "BUILD_TARGETS += db-build" won't compile on systems
without fts(3) would do.

> Indeed, that sounds bad.
> 
> I guess i won't change that for this release, though.  I worry that
> if we are very unlucky, whatever guard i put in there might break
> on another system that was already tested and may not be tested
> again.
> 
> For the next release, i have to keep this in mind.
> I might do some more cleanup related to configure, anyway.

What about something like that before 'set -e':

echo "/* RUNNING ./CONFIGURE - SHOULD BE USED ONLY VIA MAKE, READ INSTALL */"

Maybe that would be enough? 

-- 
Paul Onyschuk
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: Portability of fts() functions

[Ingo Schwarze]

Hi Paul,

Paul Onyschuk wrote on Sat, Aug 09, 2014 at 07:09:23PM +0200:
> Ingo Schwarze wrote:

>> I guess what is needed is a compat_fts.h/compat_fts.c just like
>> for ohash(3).  I fear that won't be something that can be done
>> in a hurry, though.
>> 
>> So it looks like for the 1.13.1 release, it's probably to late
>> to fix the fts(3) issue, and systems not having it will have the
>> choice of either running 1.13.1 with "BUILD_TARGETS += db-build"
>> disabled (that is, without apropos/makewhatis)
>> or stay with 1.12.4 until 1.13.2 comes out.
>> 
>> Do you think that would be tolerable?

> For systems missing fts(3) I would say yes.  I would worry more about
> glibc scenario, especially if mdocml is packaged e.g. clean build on
> x86_64 system used by packager, where it could break otherwise, wasting
> someone's time.

Oh you mean if mandoc is compiled on a 32bit glibc platform with 32bit
off_t but more than 2 billion files in one file system or files larger
than 2 GB inside a manual tree, mandoc will compile all right,
but i may crash at runtime due to the broken fts(3) implementation
contained in glibc?  Did i get that right?

I'm not sure what i could do about that - both in the long term
and as a quick fix for this release.  What would you suggest?

> Using occasion:
> 
> In file included from mansearch_const.c:20:0:
> manpath.h:36:1: error: unknown type name '__END_DECLS'
> 
> mansearch_const.c is not including config.h before manpath.h, patch:
> 
> XXX
> --- mansearch_const.c.orig
> +++ mansearch_const.c
> @@ -14,6 +14,10 @@
>   * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
>   * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
>   */
> +#ifdef HAVE_CONFIG_H
> +#include "config.h"
> +#endif
> +
>  #include <sys/types.h>
>  #include <stdint.h>
>  
> XXX

Yes, Thomas Klausner reported that one, too, and i fixed it with
exactly that patch in rc3.

> I think configure script should be guarded against standalone
> execution.  Right now you can do "./configure && make" expecting usual
> behavior (if you forget about looking at INSTALL).  Since in this case
> ${CC} won't be defined, script will try running commands starting with
> "-Wno-unused -Werror".  All tests will fail, enabling all compat
> functions.

Indeed, that sounds bad.

I guess i won't change that for this release, though.  I worry that
if we are very unlucky, whatever guard i put in there might break
on another system that was already tested and may not be tested
again.

For the next release, i have to keep this in mind.
I might do some more cleanup related to configure, anyway.

> I also have additional question.  Are there any plans for providing
> man(1) command also?  This would make mdocml a possible, standalone
> replacement for groff and man-db combination (typical in Linux
> distributions).

I will address that separately.

Yours,
  Ingo
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: Portability of fts() functions

[Paul Onyschuk]

On Sat, 9 Aug 2014 17:49:28 +0200
Ingo Schwarze wrote:

> I guess what is needed is a compat_fts.h/compat_fts.c just like
> for ohash(3).  I fear that won't be something that can be done
> in a hurry, though.
> 
> So it looks like for the 1.13.1 release, it's probably to late
> to fix the fts(3) issue, and systems not having it will have the
> choice of either running 1.13.1 with "BUILD_TARGETS += db-build"
> disabled (that is, without apropos/makewhatis)
> or stay with 1.12.4 until 1.13.2 comes out.
> 
> Do you think that would be tolerable?

For systems missing fts(3) I would say yes.  I would worry more about
glibc scenario, especially if mdocml is packaged e.g. clean build on
x86_64 system used by packager, where it could break otherwise, wasting
someone's time.

Using occasion:

In file included from mansearch_const.c:20:0:
manpath.h:36:1: error: unknown type name '__END_DECLS'

mansearch_const.c is not including config.h before manpath.h, patch:

XXX
--- mansearch_const.c.orig
+++ mansearch_const.c
@@ -14,6 +14,10 @@
  * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
  * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
  */
+#ifdef HAVE_CONFIG_H
+#include "config.h"
+#endif
+
 #include <sys/types.h>
 #include <stdint.h>
 
XXX

I think configure script should be guarded against standalone
execution.  Right now you can do "./configure && make" expecting usual
behavior (if you forget about looking at INSTALL).  Since in this case
${CC} won't be defined, script will try running commands starting with
"-Wno-unused -Werror".  All tests will fail, enabling all compat
functions.

I also have additional question.  Are there any plans for providing
man(1) command also?  This would make mdocml a possible, standalone
replacement for groff and man-db combination (typical in Linux
distributions).

-- 
Paul Onyschuk
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: Portability of fts() functions

[Ingo Schwarze]

Hi Paul,

Paul Onyschuk wrote on Sat, Aug 09, 2014 at 12:38:27PM +0200:

> mandocdb.c in v1.13.x introduced dependency on fts() family of
> functions.  This will bite on non-BSD systems.

Ouch.  I dimly remember that was mentioned before, but then
it seems i forgot about it.  :-(

> Musl libc doesn't provide it at all, neither does systems of Solaris
> origin I guess, but then glibc does something even worse [1], relevant
> fts.h header [2].  AFAIK uClibc replicated behavior of glibc.
> 
> [1] https://sourceware.org/bugzilla/show_bug.cgi?id=15838
> [2] https://sourceware.org/git/?p=glibc.git;a=blob;f=io/fts.h

Uh-oh.  Not pretty at all.

I guess what is needed is a compat_fts.h/compat_fts.c just like
for ohash(3).  I fear that won't be something that can be done
in a hurry, though.

So it looks like for the 1.13.1 release, it's probably to late
to fix the fts(3) issue, and systems not having it will have the
choice of either running 1.13.1 with "BUILD_TARGETS += db-build"
disabled (that is, without apropos/makewhatis)
or stay with 1.12.4 until 1.13.2 comes out.

Do you think that would be tolerable?

Yours,
  Ingo
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Portability of fts() functions

[Paul Onyschuk]

mandocdb.c in v1.13.x introduced dependency on fts() family of
functions.  This will bite on non-BSD systems.

Musl libc doesn't provide it at all, neither does systems of Solaris
origin I guess, but then glibc does something even worse [1], relevant
fts.h header [2].  AFAIK uClibc replicated behavior of glibc.

[1] https://sourceware.org/bugzilla/show_bug.cgi?id=15838
[2] https://sourceware.org/git/?p=glibc.git;a=blob;f=io/fts.h

-- 
Paul Onyschuk
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv