[musl] Cross-compiling and -D_LARGEFILE64_SOURCE

[musl] Cross-compiling and -D_LARGEFILE64_SOURCE

[Sebastian Kemper]

Hello list,

I'm working on updating apr in OpenWrt. During configure it uses TRY_RUN
a lot, so we have a lot of configure variables defined to point it into
the right direction.

With one variable I have some trouble: apr_cv_use_lfs64=yes. The trouble
is I don't know whether or not to set it.

It basically forces -D_LARGEFILE64_SOURCE to be included in the
CPPFLAGS. So my first impulse was to force it to "yes". But then I saw
some build logs from Alpine Linux:

https://build.alpinelinux.org/buildlogs/build-edge-x86/main/apr/apr-1.7.0-r0.log

<snip>
checking whether the compiler provides atomic builtins... yes
checking whether to enable -D_LARGEFILE64_SOURCE... no
configure: Configured for Linux 4.14
<snip>

So Alpine is not cross-compiling here, and the result is "no". And this
particular log is for i586 (I thought - before seeing this log - that
maybe -D_LARGEFILE64_SOURCE is needed for non-64-bit platforms).

The TRY_RUN that apr uses to find out whether to add
-D_LARGEFILE64_SOURCE is pasted at the bottom of this mail. If I run
configure on my x86_64 glibc laptop it says "no". In the Alpine logs it
also says "no". If I search the Internet I find some old pastes where
the outcome was "yes", though. It kind of looks like the answer over
time changed from "yes" to "no", for no apparent reason (at least for
me).

If it sets -D_LARGEFILE64_SOURCE it propagates the same flag to its
users via apr-1-config. In the apr sources I see the define also being
used occasionally.

include/arch/unix/apr_arch_file_io.h:

#if APR_HAS_LARGE_FILES && defined(_LARGEFILE64_SOURCE)
#define stat(f,b) stat64(f,b)
#define lstat(f,b) lstat64(f,b)
#define fstat(f,b) fstat64(f,b)
#define lseek(f,o,w) lseek64(f,o,w)
#define ftruncate(f,l) ftruncate64(f,l)
typedef struct stat64 struct_stat;
#else
typedef struct stat struct_stat;
#endif

file_io/unix/open.c:

#if APR_HAS_LARGE_FILES && defined(_LARGEFILE64_SOURCE)
    oflags |= O_LARGEFILE;
#elif defined(O_LARGEFILE)
    if (flag & APR_FOPEN_LARGEFILE) {
        oflags |= O_LARGEFILE;
    }
#endif

I obviously would like LFS support, but I don't know about this
configure variable. I guess the correct answer must be one of these:

"Don't use it"
"Use it"
"Use it if condition X applies"

I guess this is a bit off-topic maybe. But I'm really running out of
ideas :) So if anybody can share some insight I'd really appreciate it.

Kind regards,
Seb

   AC_CACHE_CHECK([whether to enable -D_LARGEFILE64_SOURCE], [apr_cv_use_lfs64], [
   apr_save_CPPFLAGS=$CPPFLAGS
   CPPFLAGS="$CPPFLAGS -D_LARGEFILE64_SOURCE"
   AC_TRY_RUN([
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>

void main(void)
{
    int fd, ret = 0;
    struct stat64 st;
    off64_t off = 4242;

    if (sizeof(off64_t) != 8 || sizeof(off_t) != 4)
       exit(1);
    if ((fd = open("conftest.lfs", O_LARGEFILE|O_CREAT|O_WRONLY, 0644)) < 0)
       exit(2);
    if (ftruncate64(fd, off) != 0)
       ret = 3;
    else if (fstat64(fd, &st) != 0 || st.st_size != off)
       ret = 4;
    else if (lseek64(fd, off, SEEK_SET) != off)
       ret = 5;
    else if (close(fd) != 0)
       ret = 6;
    else if (lstat64("conftest.lfs", &st) != 0 || st.st_size != off)
       ret = 7;
    else if (stat64("conftest.lfs", &st) != 0 || st.st_size != off)
       ret = 8;
    unlink("conftest.lfs");

    exit(ret);
}], [apr_cv_use_lfs64=yes], [apr_cv_use_lfs64=no], [apr_cv_use_lfs64=no])
   CPPFLAGS=$apr_save_CPPFLAGS])
   if test "$apr_cv_use_lfs64" = "yes"; then
      APR_ADDTO(CPPFLAGS, [-D_LARGEFILE64_SOURCE])
   fi

Re: [musl] Cross-compiling and -D_LARGEFILE64_SOURCE

[Rich Felker]

On Mon, Feb 03, 2020 at 09:44:17PM +0100, Sebastian Kemper wrote:
> Hello list,
> 
> I'm working on updating apr in OpenWrt. During configure it uses TRY_RUN
> a lot, so we have a lot of configure variables defined to point it into
> the right direction.
> 
> With one variable I have some trouble: apr_cv_use_lfs64=yes. The trouble
> is I don't know whether or not to set it.

Can you clarify where that came from? It sounds like it wasn't
detected by configure but something OpenWrt's build scripts are
forcing. Is that right?

> It basically forces -D_LARGEFILE64_SOURCE to be included in the
> CPPFLAGS. So my first impulse was to force it to "yes". But then I saw
> some build logs from Alpine Linux:

_LARGEFILE64_SOURCE is a horrible horrible thing you never want, so if
you have the opportunity to omit it, do. We're actually trying to get
rid of this stuff completely from the headers. It was originally put
there just as a workaround for a lot of bad GNU software trying to use
the "64"-suffixed symbols based on invalid configure tests (link-only)
which broke things badly when no declaration for the functions
existed.

> https://build.alpinelinux.org/buildlogs/build-edge-x86/main/apr/apr-1.7.0-r0.log
> 
> <snip>
> checking whether the compiler provides atomic builtins... yes
> checking whether to enable -D_LARGEFILE64_SOURCE... no
> configure: Configured for Linux 4.14
> <snip>
> 
> So Alpine is not cross-compiling here, and the result is "no". And this
> particular log is for i586 (I thought - before seeing this log - that
> maybe -D_LARGEFILE64_SOURCE is needed for non-64-bit platforms).

It's not. It's legacy junk that should never be used. It's to make
visible the "64"-suffixed functions on systems where the standard
functions are limited to 32-bit offsets, which is never the case on
musl.

> The TRY_RUN that apr uses to find out whether to add
> -D_LARGEFILE64_SOURCE is pasted at the bottom of this mail. If I run
> configure on my x86_64 glibc laptop it says "no". In the Alpine logs it
> also says "no". If I search the Internet I find some old pastes where
> the outcome was "yes", though. It kind of looks like the answer over
> time changed from "yes" to "no", for no apparent reason (at least for
> me).
> 
> If it sets -D_LARGEFILE64_SOURCE it propagates the same flag to its
> users via apr-1-config. In the apr sources I see the define also being
> used occasionally.
> 
> include/arch/unix/apr_arch_file_io.h:
> 
> #if APR_HAS_LARGE_FILES && defined(_LARGEFILE64_SOURCE)
> #define stat(f,b) stat64(f,b)
> #define lstat(f,b) lstat64(f,b)
> #define fstat(f,b) fstat64(f,b)
> #define lseek(f,o,w) lseek64(f,o,w)
> #define ftruncate(f,l) ftruncate64(f,l)
> typedef struct stat64 struct_stat;
> #else
> typedef struct stat struct_stat;
> #endif

OK, this is going to break badly. It's also completely bogus (bug in
APR). *All* feature test macros are conveyed *from* the application
*to* libc. #define of them belongs only in the application and
#ifdef/#if defined() of them only in libc headers.

> file_io/unix/open.c:
> 
> #if APR_HAS_LARGE_FILES && defined(_LARGEFILE64_SOURCE)
>     oflags |= O_LARGEFILE;
> #elif defined(O_LARGEFILE)
>     if (flag & APR_FOPEN_LARGEFILE) {
>         oflags |= O_LARGEFILE;
>     }
> #endif
> 
> I obviously would like LFS support, but I don't know about this
> configure variable. I guess the correct answer must be one of these:
> 
> "Don't use it"
> "Use it"
> "Use it if condition X applies"

Don't use it. :-)

> I guess this is a bit off-topic maybe. But I'm really running out of
> ideas :) So if anybody can share some insight I'd really appreciate it.
> 
> Kind regards,
> Seb
> 
>    AC_CACHE_CHECK([whether to enable -D_LARGEFILE64_SOURCE], [apr_cv_use_lfs64], [
>    apr_save_CPPFLAGS=$CPPFLAGS
>    CPPFLAGS="$CPPFLAGS -D_LARGEFILE64_SOURCE"
>    AC_TRY_RUN([
> #include <sys/types.h>
> #include <sys/stat.h>
> #include <fcntl.h>
> #include <stdlib.h>
> #include <stdio.h>
> #include <unistd.h>
> 
> void main(void)
> {
>     int fd, ret = 0;
>     struct stat64 st;
>     off64_t off = 4242;
> 
>     if (sizeof(off64_t) != 8 || sizeof(off_t) != 4)
>        exit(1);
>     if ((fd = open("conftest.lfs", O_LARGEFILE|O_CREAT|O_WRONLY, 0644)) < 0)
>        exit(2);
>     if (ftruncate64(fd, off) != 0)
>        ret = 3;
>     else if (fstat64(fd, &st) != 0 || st.st_size != off)
>        ret = 4;
>     else if (lseek64(fd, off, SEEK_SET) != off)
>        ret = 5;
>     else if (close(fd) != 0)
>        ret = 6;
>     else if (lstat64("conftest.lfs", &st) != 0 || st.st_size != off)
>        ret = 7;
>     else if (stat64("conftest.lfs", &st) != 0 || st.st_size != off)
>        ret = 8;
>     unlink("conftest.lfs");
> 
>     exit(ret);
> }], [apr_cv_use_lfs64=yes], [apr_cv_use_lfs64=no], [apr_cv_use_lfs64=no])
>    CPPFLAGS=$apr_save_CPPFLAGS])
>    if test "$apr_cv_use_lfs64" = "yes"; then
>       APR_ADDTO(CPPFLAGS, [-D_LARGEFILE64_SOURCE])
>    fi

OK, the configure tests that need to run something are a big problem
if you're cross-compiling, so you probably need to force it to no. The
above should be factored into two tests, one compile-only and the
other execution, so that the compile-only one is all that's needed
when it's sufficient to give a conclusive result.

Rich

Re: [musl] Cross-compiling and -D_LARGEFILE64_SOURCE

[Sebastian Kemper]

On Mon, Feb 03, 2020 at 04:44:45PM -0500, Rich Felker wrote:
> On Mon, Feb 03, 2020 at 09:44:17PM +0100, Sebastian Kemper wrote:
> > Hello list,
> >
> > I'm working on updating apr in OpenWrt. During configure it uses TRY_RUN
> > a lot, so we have a lot of configure variables defined to point it into
> > the right direction.
> >
> > With one variable I have some trouble: apr_cv_use_lfs64=yes. The trouble
> > is I don't know whether or not to set it.
>
> Can you clarify where that came from? It sounds like it wasn't
> detected by configure but something OpenWrt's build scripts are
> forcing. Is that right?

Hi Rich!

Historically OpenWrt had

TARGET_CPPFLAGS += -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE

_and_ apr_cv_use_lfs64=yes as configure var. I think this was working
around some bug or the other, I don't know which. Then these got removed
(not by me). A few weeks ago I added myself as maintainer of apr and was
poking around, and saw that buildroot also sets apr_cv_use_lfs64=yes. So
I thought we should use it too. But afterwards I had some doubts about
it and poked around some more. And here we are :D

>
> > It basically forces -D_LARGEFILE64_SOURCE to be included in the
> > CPPFLAGS. So my first impulse was to force it to "yes". But then I saw
> > some build logs from Alpine Linux:
>
> _LARGEFILE64_SOURCE is a horrible horrible thing you never want, so if
> you have the opportunity to omit it, do. We're actually trying to get
> rid of this stuff completely from the headers. It was originally put
> there just as a workaround for a lot of bad GNU software trying to use
> the "64"-suffixed symbols based on invalid configure tests (link-only)
> which broke things badly when no declaration for the functions
> existed.

Remove it I shall, thank you very much!

>
> > https://build.alpinelinux.org/buildlogs/build-edge-x86/main/apr/apr-1.7.0-r0.log
> >
> > <snip>
> > checking whether the compiler provides atomic builtins... yes
> > checking whether to enable -D_LARGEFILE64_SOURCE... no
> > configure: Configured for Linux 4.14
> > <snip>
> >
> > So Alpine is not cross-compiling here, and the result is "no". And this
> > particular log is for i586 (I thought - before seeing this log - that
> > maybe -D_LARGEFILE64_SOURCE is needed for non-64-bit platforms).
>
> It's not. It's legacy junk that should never be used. It's to make
> visible the "64"-suffixed functions on systems where the standard
> functions are limited to 32-bit offsets, which is never the case on
> musl.
>
> > The TRY_RUN that apr uses to find out whether to add
> > -D_LARGEFILE64_SOURCE is pasted at the bottom of this mail. If I run
> > configure on my x86_64 glibc laptop it says "no". In the Alpine logs it
> > also says "no". If I search the Internet I find some old pastes where
> > the outcome was "yes", though. It kind of looks like the answer over
> > time changed from "yes" to "no", for no apparent reason (at least for
> > me).
> >
> > If it sets -D_LARGEFILE64_SOURCE it propagates the same flag to its
> > users via apr-1-config. In the apr sources I see the define also being
> > used occasionally.
> >
> > include/arch/unix/apr_arch_file_io.h:
> >
> > #if APR_HAS_LARGE_FILES && defined(_LARGEFILE64_SOURCE)
> > #define stat(f,b) stat64(f,b)
> > #define lstat(f,b) lstat64(f,b)
> > #define fstat(f,b) fstat64(f,b)
> > #define lseek(f,o,w) lseek64(f,o,w)
> > #define ftruncate(f,l) ftruncate64(f,l)
> > typedef struct stat64 struct_stat;
> > #else
> > typedef struct stat struct_stat;
> > #endif
>
> OK, this is going to break badly. It's also completely bogus (bug in
> APR). *All* feature test macros are conveyed *from* the application
> *to* libc. #define of them belongs only in the application and
> #ifdef/#if defined() of them only in libc headers.
>
> > file_io/unix/open.c:
> >
> > #if APR_HAS_LARGE_FILES && defined(_LARGEFILE64_SOURCE)
> >     oflags |= O_LARGEFILE;
> > #elif defined(O_LARGEFILE)
> >     if (flag & APR_FOPEN_LARGEFILE) {
> >         oflags |= O_LARGEFILE;
> >     }
> > #endif
> >
> > I obviously would like LFS support, but I don't know about this
> > configure variable. I guess the correct answer must be one of these:
> >
> > "Don't use it"
> > "Use it"
> > "Use it if condition X applies"
>
> Don't use it. :-)
>

Thanks Rich! Again I came here for final wisdom and wasn't disappointed
;)

Kind regards,
Seb

> > I guess this is a bit off-topic maybe. But I'm really running out of
> > ideas :) So if anybody can share some insight I'd really appreciate it.
> >
> > Kind regards,
> > Seb
> >
> >    AC_CACHE_CHECK([whether to enable -D_LARGEFILE64_SOURCE], [apr_cv_use_lfs64], [
> >    apr_save_CPPFLAGS=$CPPFLAGS
> >    CPPFLAGS="$CPPFLAGS -D_LARGEFILE64_SOURCE"
> >    AC_TRY_RUN([
> > #include <sys/types.h>
> > #include <sys/stat.h>
> > #include <fcntl.h>
> > #include <stdlib.h>
> > #include <stdio.h>
> > #include <unistd.h>
> >
> > void main(void)
> > {
> >     int fd, ret = 0;
> >     struct stat64 st;
> >     off64_t off = 4242;
> >
> >     if (sizeof(off64_t) != 8 || sizeof(off_t) != 4)
> >        exit(1);
> >     if ((fd = open("conftest.lfs", O_LARGEFILE|O_CREAT|O_WRONLY, 0644)) < 0)
> >        exit(2);
> >     if (ftruncate64(fd, off) != 0)
> >        ret = 3;
> >     else if (fstat64(fd, &st) != 0 || st.st_size != off)
> >        ret = 4;
> >     else if (lseek64(fd, off, SEEK_SET) != off)
> >        ret = 5;
> >     else if (close(fd) != 0)
> >        ret = 6;
> >     else if (lstat64("conftest.lfs", &st) != 0 || st.st_size != off)
> >        ret = 7;
> >     else if (stat64("conftest.lfs", &st) != 0 || st.st_size != off)
> >        ret = 8;
> >     unlink("conftest.lfs");
> >
> >     exit(ret);
> > }], [apr_cv_use_lfs64=yes], [apr_cv_use_lfs64=no], [apr_cv_use_lfs64=no])
> >    CPPFLAGS=$apr_save_CPPFLAGS])
> >    if test "$apr_cv_use_lfs64" = "yes"; then
> >       APR_ADDTO(CPPFLAGS, [-D_LARGEFILE64_SOURCE])
> >    fi
>
> OK, the configure tests that need to run something are a big problem
> if you're cross-compiling, so you probably need to force it to no. The
> above should be factored into two tests, one compile-only and the
> other execution, so that the compile-only one is all that's needed
> when it's sufficient to give a conclusive result.
>
> Rich