* musl and kernel headers [was Re: system-images 1.4.2: od is broken; bzip2 is missing] [not found] ` <5612925A.4070402@landley.net> @ 2015-10-06 1:44 ` Rich Felker 2015-10-06 2:24 ` Rob Landley 2015-10-06 16:09 ` Denys Vlasenko 0 siblings, 2 replies; 18+ messages in thread From: Rich Felker @ 2015-10-06 1:44 UTC (permalink / raw) To: Rob Landley; +Cc: Denys Vlasenko, Aboriginal Linux, musl On Mon, Oct 05, 2015 at 10:08:10AM -0500, Rob Landley wrote: > > Re musl: two cases of breakage: > > > > #include <netinet/in.h> > > #include <linux/if_bridge.h> > > results in: > > In file included from /usr/include/linux/if_bridge.h:18, > > from networking/brctl.c:67: > > /usr/include/linux/in6.h:32: error: redefinition of 'struct in6_addr' > > /usr/include/linux/in6.h:49: error: redefinition of 'struct sockaddr_in6' > > /usr/include/linux/in6.h:59: error: redefinition of 'struct ipv6_mreq' > > > > and > > > > #include <linux/ethtool.h> > > #include <net/ethernet.h> > > results in: > > In file included from /usr/include/net/ethernet.h:10, > > from networking/ifplugd.c:41: > > /usr/include/netinet/if_ether.h:96: error: redefinition of 'struct ethhdr' > > > > That I leave to Rich. :) Including kernel headers is just really problematic. These days they try to make it work on glibc with a complex dance between glibc's headers and the kernel headers. You're likely to have the best luck by including the libc headers first. I think Sabotage and possibly some other musl-based dists are patching the kernel headers to make them behave better; see: https://github.com/sabotage-linux/sabotage/blob/master/pkg/kernel-headers But it might be possible that we can just #define whatever macro the kernel headers expect to suppress redefinitions, if it's in a reserved namespace. This would only fix the case where the kernel headers are included _after_ the libc/userspace ones, but it's not too ugly. I probably won't get around to it myself but if anyone wants to check what macros we'd need to define (and where), let's discuss it (cc'ing musl list). The cleaner approach is just avoiding including both the kernel headers and libc/userspace headers for the same things in the same file. In theory this may be hard in some cases, but I find that I can almost always fix these sorts of errors during a build by commenting out one or two #include lines. Rich ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: musl and kernel headers [was Re: system-images 1.4.2: od is broken; bzip2 is missing] 2015-10-06 1:44 ` musl and kernel headers [was Re: system-images 1.4.2: od is broken; bzip2 is missing] Rich Felker @ 2015-10-06 2:24 ` Rob Landley 2015-10-06 11:01 ` Szabolcs Nagy 2015-10-06 14:30 ` Rich Felker 2015-10-06 16:09 ` Denys Vlasenko 1 sibling, 2 replies; 18+ messages in thread From: Rob Landley @ 2015-10-06 2:24 UTC (permalink / raw) To: Rich Felker; +Cc: Denys Vlasenko, Aboriginal Linux, musl On 10/05/2015 08:44 PM, Rich Felker wrote: > On Mon, Oct 05, 2015 at 10:08:10AM -0500, Rob Landley wrote: > The cleaner approach is just avoiding including both the kernel > headers and libc/userspace headers for the same things in the same > file. In theory this may be hard in some cases, but I find that I can > almost always fix these sorts of errors during a build by commenting > out one or two #include lines. I am _deeply_ curious how you'd get linux/loop.h on a platform where you need the 32 bit loopback structure definition without including the kernel header. I struggled with that one a lot back in the busybox days. (For the 64 bit version you can block copy the structure out of the header into your program, just like you can block copy everything out of headers and never actually #include anything, in the name of portability!) Sadly, the kernel headers are exported for a _reason_. If I need to syscall something you haven't wrapped, I need the _NR_ and it varies per target. It's a thing. Rob ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Re: musl and kernel headers [was Re: system-images 1.4.2: od is broken; bzip2 is missing] 2015-10-06 2:24 ` Rob Landley @ 2015-10-06 11:01 ` Szabolcs Nagy 2015-10-06 14:30 ` Rich Felker 1 sibling, 0 replies; 18+ messages in thread From: Szabolcs Nagy @ 2015-10-06 11:01 UTC (permalink / raw) To: musl; +Cc: Rich Felker, Denys Vlasenko, Aboriginal Linux * Rob Landley <rob@landley.net> [2015-10-05 21:24:01 -0500]: > On 10/05/2015 08:44 PM, Rich Felker wrote: > > On Mon, Oct 05, 2015 at 10:08:10AM -0500, Rob Landley wrote: > > The cleaner approach is just avoiding including both the kernel > > headers and libc/userspace headers for the same things in the same > > file. In theory this may be hard in some cases, but I find that I can > > almost always fix these sorts of errors during a build by commenting > > out one or two #include lines. > > I am _deeply_ curious how you'd get linux/loop.h on a platform where you > need the 32 bit loopback structure definition without including the > kernel header. I struggled with that one a lot back in the busybox days. in theory you can put all usage of the structure in a separate translation unit where you don't include any libc headers. (if you need libc api you can declare prototypes e.g. int ioctl(int, int,...); but if you need libc types there then you are in trouble). > (For the 64 bit version you can block copy the structure out of the > header into your program, just like you can block copy everything out of > headers and never actually #include anything, in the name of portability!) > > Sadly, the kernel headers are exported for a _reason_. If I need to > syscall something you haven't wrapped, I need the _NR_ and it varies per > target. It's a thing. > > Rob ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: musl and kernel headers [was Re: system-images 1.4.2: od is broken; bzip2 is missing] 2015-10-06 2:24 ` Rob Landley 2015-10-06 11:01 ` Szabolcs Nagy @ 2015-10-06 14:30 ` Rich Felker 2015-10-06 16:05 ` Denys Vlasenko 1 sibling, 1 reply; 18+ messages in thread From: Rich Felker @ 2015-10-06 14:30 UTC (permalink / raw) To: Rob Landley; +Cc: Denys Vlasenko, Aboriginal Linux, musl On Mon, Oct 05, 2015 at 09:24:01PM -0500, Rob Landley wrote: > On 10/05/2015 08:44 PM, Rich Felker wrote: > > On Mon, Oct 05, 2015 at 10:08:10AM -0500, Rob Landley wrote: > > The cleaner approach is just avoiding including both the kernel > > headers and libc/userspace headers for the same things in the same > > file. In theory this may be hard in some cases, but I find that I can > > almost always fix these sorts of errors during a build by commenting > > out one or two #include lines. > > I am _deeply_ curious how you'd get linux/loop.h on a platform where you > need the 32 bit loopback structure definition without including the > kernel header. Sorry I wasn't clear on this; the meaning I intended to convey was including both kernel and libc/userspace headers for the same things. Network is the main area affected here. The kernel headers have fixed up all the gratuitous conflicts with userspace, but the big remaining ones are places where they want to define types with the exact same names, which mostly happens in network. So what I was trying to say is that programs using kernel network headers (legitimately for linux-specific stuff) are going to best avoid the risk of clashes by not including libc network headers in the same files. > Sadly, the kernel headers are exported for a _reason_. If I need to > syscall something you haven't wrapped, I need the _NR_ and it varies per > target. It's a thing. For syscalls we export all SYS_*/__NR_* macros (sys/syscall.h), but you still might need macro and struct definitions for arguments to the syscalls or ioctls. Rich ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: musl and kernel headers [was Re: system-images 1.4.2: od is broken; bzip2 is missing] 2015-10-06 14:30 ` Rich Felker @ 2015-10-06 16:05 ` Denys Vlasenko 0 siblings, 0 replies; 18+ messages in thread From: Denys Vlasenko @ 2015-10-06 16:05 UTC (permalink / raw) To: Rich Felker; +Cc: Rob Landley, Aboriginal Linux, musl On Tue, Oct 6, 2015 at 4:30 PM, Rich Felker <dalias@aerifal.cx> wrote: > On Mon, Oct 05, 2015 at 09:24:01PM -0500, Rob Landley wrote: >> On 10/05/2015 08:44 PM, Rich Felker wrote: >> > The cleaner approach is just avoiding including both the kernel >> > headers and libc/userspace headers for the same things in the same >> > file. In theory this may be hard in some cases, but I find that I can >> > almost always fix these sorts of errors during a build by commenting >> > out one or two #include lines. >> >> I am _deeply_ curious how you'd get linux/loop.h on a platform where you >> need the 32 bit loopback structure definition without including the >> kernel header. > > Sorry I wasn't clear on this; the meaning I intended to convey was > including both kernel and libc/userspace headers for the same things. > Network is the main area affected here. The kernel headers have fixed > up all the gratuitous conflicts with userspace, but the big remaining > ones are places where they want to define types with the exact same > names, which mostly happens in network. So what I was trying to say is > that programs using kernel network headers (legitimately for > linux-specific stuff) are going to best avoid the risk of clashes by > not including libc network headers in the same files. For me as a libc user, the gist of the story is: I switch to musl, things break. Apart from linux/* includes, utmp/wtmp also broke. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: musl and kernel headers [was Re: system-images 1.4.2: od is broken; bzip2 is missing] 2015-10-06 1:44 ` musl and kernel headers [was Re: system-images 1.4.2: od is broken; bzip2 is missing] Rich Felker 2015-10-06 2:24 ` Rob Landley @ 2015-10-06 16:09 ` Denys Vlasenko 2015-10-08 16:58 ` Rich Felker 1 sibling, 1 reply; 18+ messages in thread From: Denys Vlasenko @ 2015-10-06 16:09 UTC (permalink / raw) To: Rich Felker; +Cc: Rob Landley, Aboriginal Linux, musl On Tue, Oct 6, 2015 at 3:44 AM, Rich Felker <dalias@aerifal.cx> wrote: >> > >> > #include <netinet/in.h> >> > #include <linux/if_bridge.h> >> > results in: >> > In file included from /usr/include/linux/if_bridge.h:18, >> > from networking/brctl.c:67: >> > /usr/include/linux/in6.h:32: error: redefinition of 'struct in6_addr' >> > /usr/include/linux/in6.h:49: error: redefinition of 'struct sockaddr_in6' >> > /usr/include/linux/in6.h:59: error: redefinition of 'struct ipv6_mreq' >> > >> > and >> > >> > #include <linux/ethtool.h> >> > #include <net/ethernet.h> >> > results in: >> > In file included from /usr/include/net/ethernet.h:10, >> > from networking/ifplugd.c:41: >> > /usr/include/netinet/if_ether.h:96: error: redefinition of 'struct ethhdr' >> > >> >> That I leave to Rich. :) > > Including kernel headers is just really problematic. These days they > try to make it work on glibc with a complex dance between glibc's > headers and the kernel headers. You're likely to have the best luck by > including the libc headers first. brctl.c was including <linux/if_bridge.h> after <netinet/in.h> ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Re: musl and kernel headers [was Re: system-images 1.4.2: od is broken; bzip2 is missing] 2015-10-06 16:09 ` Denys Vlasenko @ 2015-10-08 16:58 ` Rich Felker 2015-10-09 19:11 ` Denys Vlasenko 0 siblings, 1 reply; 18+ messages in thread From: Rich Felker @ 2015-10-08 16:58 UTC (permalink / raw) To: Denys Vlasenko; +Cc: Rob Landley, Aboriginal Linux, musl On Tue, Oct 06, 2015 at 06:09:15PM +0200, Denys Vlasenko wrote: > On Tue, Oct 6, 2015 at 3:44 AM, Rich Felker <dalias@aerifal.cx> wrote: > >> > > >> > #include <netinet/in.h> > >> > #include <linux/if_bridge.h> > >> > results in: > >> > In file included from /usr/include/linux/if_bridge.h:18, > >> > from networking/brctl.c:67: > >> > /usr/include/linux/in6.h:32: error: redefinition of 'struct in6_addr' > >> > /usr/include/linux/in6.h:49: error: redefinition of 'struct sockaddr_in6' > >> > /usr/include/linux/in6.h:59: error: redefinition of 'struct ipv6_mreq' > >> > > >> > and > >> > > >> > #include <linux/ethtool.h> > >> > #include <net/ethernet.h> > >> > results in: > >> > In file included from /usr/include/net/ethernet.h:10, > >> > from networking/ifplugd.c:41: > >> > /usr/include/netinet/if_ether.h:96: error: redefinition of 'struct ethhdr' > >> > > >> > >> That I leave to Rich. :) > > > > Including kernel headers is just really problematic. These days they > > try to make it work on glibc with a complex dance between glibc's > > headers and the kernel headers. You're likely to have the best luck by > > including the libc headers first. > > brctl.c was including <linux/if_bridge.h> after <netinet/in.h> The problem is linux/libc-compat.h, which should fix this, only works on glibc, by design. See: #ifndef _LIBC_COMPAT_H #define _LIBC_COMPAT_H /* We have included glibc headers... */ #if defined(__GLIBC__) /* Coordinate with glibc netinet/in.h header. */ #if defined(_NETINET_IN_H) If you patch it like this: -#if defined(__GLIBC__) +#if 1 then it should mostly work but it's still all a big hack. I think that's what distros are doing. The problem is that the same header is trying to do two different things: 1. Provide extra linux-kernel-API stuff that's not in the libc/userspace headers. 2. Provide definitions of the standard types and constants for uClibc and klibc, which don't have complete libc headers and rely on the kernel headers for definitions. These two uses really should be separated out into separate headers so that the latter only get included explicitly by uClibc and klibc and otherwise remain completely unused. But that would require coordinated changes/upgrades which are unlikely to happen. :( Rich ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Re: musl and kernel headers [was Re: system-images 1.4.2: od is broken; bzip2 is missing] 2015-10-08 16:58 ` Rich Felker @ 2015-10-09 19:11 ` Denys Vlasenko 2015-10-09 19:46 ` Rich Felker 0 siblings, 1 reply; 18+ messages in thread From: Denys Vlasenko @ 2015-10-09 19:11 UTC (permalink / raw) To: Rich Felker; +Cc: Rob Landley, Aboriginal Linux, musl On Thu, Oct 8, 2015 at 6:58 PM, Rich Felker <dalias@aerifal.cx> wrote: >> > Including kernel headers is just really problematic. These days they >> > try to make it work on glibc with a complex dance between glibc's >> > headers and the kernel headers. You're likely to have the best luck by >> > including the libc headers first. >> >> brctl.c was including <linux/if_bridge.h> after <netinet/in.h> > > The problem is linux/libc-compat.h, which should fix this, only works > on glibc, by design. See: > > #ifndef _LIBC_COMPAT_H > #define _LIBC_COMPAT_H > > /* We have included glibc headers... */ > #if defined(__GLIBC__) > > /* Coordinate with glibc netinet/in.h header. */ > #if defined(_NETINET_IN_H) > > If you patch it like this: > > -#if defined(__GLIBC__) > +#if 1 > > then it should mostly work but it's still all a big hack. I think > that's what distros are doing. The problem is that the same header is > trying to do two different things: > > 1. Provide extra linux-kernel-API stuff that's not in the > libc/userspace headers. > > 2. Provide definitions of the standard types and constants for uClibc > and klibc, which don't have complete libc headers and rely on the > kernel headers for definitions. > > These two uses really should be separated out into separate headers so > that the latter only get included explicitly by uClibc and klibc and > otherwise remain completely unused. But that would require coordinated > changes/upgrades which are unlikely to happen. :( Looking at kernel's libc-compat.h, it looks like you can get away with using __UAPI_DEF_foo's like this? #if defined(__UAPI_DEF_SOCKADDR_IN) && __UAPI_DEF_SOCKADDR_IN == 1 /* kernel already defined the struct, do nothing */ #else struct sockaddr_in { ... }; #undef __UAPI_DEF_SOCKADDR_IN /* tell kernel to not define the struct */ #define __UAPI_DEF_SOCKADDR_IN 0 #endif ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Re: musl and kernel headers [was Re: system-images 1.4.2: od is broken; bzip2 is missing] 2015-10-09 19:11 ` Denys Vlasenko @ 2015-10-09 19:46 ` Rich Felker 2015-10-10 4:56 ` Rob Landley 2015-10-13 12:10 ` Denys Vlasenko 0 siblings, 2 replies; 18+ messages in thread From: Rich Felker @ 2015-10-09 19:46 UTC (permalink / raw) To: Denys Vlasenko; +Cc: Rob Landley, Aboriginal Linux, musl On Fri, Oct 09, 2015 at 09:11:08PM +0200, Denys Vlasenko wrote: > On Thu, Oct 8, 2015 at 6:58 PM, Rich Felker <dalias@aerifal.cx> wrote: > >> > Including kernel headers is just really problematic. These days they > >> > try to make it work on glibc with a complex dance between glibc's > >> > headers and the kernel headers. You're likely to have the best luck by > >> > including the libc headers first. > >> > >> brctl.c was including <linux/if_bridge.h> after <netinet/in.h> > > > > The problem is linux/libc-compat.h, which should fix this, only works > > on glibc, by design. See: > > > > #ifndef _LIBC_COMPAT_H > > #define _LIBC_COMPAT_H > > > > /* We have included glibc headers... */ > > #if defined(__GLIBC__) > > > > /* Coordinate with glibc netinet/in.h header. */ > > #if defined(_NETINET_IN_H) > > > > If you patch it like this: > > > > -#if defined(__GLIBC__) > > +#if 1 > > > > then it should mostly work but it's still all a big hack. I think > > that's what distros are doing. The problem is that the same header is > > trying to do two different things: > > > > 1. Provide extra linux-kernel-API stuff that's not in the > > libc/userspace headers. > > > > 2. Provide definitions of the standard types and constants for uClibc > > and klibc, which don't have complete libc headers and rely on the > > kernel headers for definitions. > > > > These two uses really should be separated out into separate headers so > > that the latter only get included explicitly by uClibc and klibc and > > otherwise remain completely unused. But that would require coordinated > > changes/upgrades which are unlikely to happen. :( > > Looking at kernel's libc-compat.h, it looks like you can get away > with using __UAPI_DEF_foo's like this? > > > #if defined(__UAPI_DEF_SOCKADDR_IN) && __UAPI_DEF_SOCKADDR_IN == 1 > /* kernel already defined the struct, do nothing */ > #else > struct sockaddr_in { > ... > }; This would address the case where the kernel header is included first, but it's not a case I or most of the musl community wants to support, because there's no guarantee that the kernel's definitions of these structures will actually be compatible with use elsewhere in the libc headers, etc. The other direction, suppressing kernel headers' definition of the structs, is what we want to work, but they've restricted their logic for that to only work when __GLIBC__ is defined. :( > #undef __UAPI_DEF_SOCKADDR_IN > /* tell kernel to not define the struct */ > #define __UAPI_DEF_SOCKADDR_IN 0 > #endif We could do something like this but then we would need to keep up with the list of all the __UAPI defines we need to suppress unwanted kernel definitions. What if we could get the kernel to change the #if defined(__GLIBC__) to #if defined(__GLIBC__) || defined(__UAPI_DONTNEED_DEFS) or similar, so that there would only be one macro we need to define, and the kernel would then use the same logic it uses with glibc to suppress all of these. Rich ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Re: musl and kernel headers [was Re: system-images 1.4.2: od is broken; bzip2 is missing] 2015-10-09 19:46 ` Rich Felker @ 2015-10-10 4:56 ` Rob Landley 2015-10-13 12:10 ` Denys Vlasenko 1 sibling, 0 replies; 18+ messages in thread From: Rob Landley @ 2015-10-10 4:56 UTC (permalink / raw) To: Rich Felker, Denys Vlasenko; +Cc: Aboriginal Linux, musl On 10/09/2015 02:46 PM, Rich Felker wrote: > On Fri, Oct 09, 2015 at 09:11:08PM +0200, Denys Vlasenko wrote: >> #undef __UAPI_DEF_SOCKADDR_IN >> /* tell kernel to not define the struct */ >> #define __UAPI_DEF_SOCKADDR_IN 0 >> #endif > > We could do something like this but then we would need to keep up with > the list of all the __UAPI defines we need to suppress unwanted kernel > definitions. You have a bug report. This would fix it. Then wait for more bug reports? Rob ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Re: musl and kernel headers [was Re: system-images 1.4.2: od is broken; bzip2 is missing] 2015-10-09 19:46 ` Rich Felker 2015-10-10 4:56 ` Rob Landley @ 2015-10-13 12:10 ` Denys Vlasenko 2015-10-13 14:16 ` Ruben Winistörfer ` (2 more replies) 1 sibling, 3 replies; 18+ messages in thread From: Denys Vlasenko @ 2015-10-13 12:10 UTC (permalink / raw) To: Rich Felker; +Cc: Rob Landley, Aboriginal Linux, musl On Fri, Oct 9, 2015 at 9:46 PM, Rich Felker <dalias@aerifal.cx> wrote: >> Looking at kernel's libc-compat.h, it looks like you can get away >> with using __UAPI_DEF_foo's like this? >> >> >> #if defined(__UAPI_DEF_SOCKADDR_IN) && __UAPI_DEF_SOCKADDR_IN == 1 >> /* kernel already defined the struct, do nothing */ >> #else >> struct sockaddr_in { >> ... >> }; > > This would address the case where the kernel header is included first, > but it's not a case I or most of the musl community wants to support, > because there's no guarantee that the kernel's definitions of these > structures will actually be compatible with use elsewhere in the libc > headers, etc. If kernel's definition does not match yours, there is a much bigger problem than "includes do not compile": kernel and userspace definitions of these structs *must* match (modulo harmless things like different typedef names for field types). So in this case either kernel or libc would need to be fixed. > The other direction, suppressing kernel headers' definition of the > structs, is what we want to work, but they've restricted their logic > for that to only work when __GLIBC__ is defined. :( Yes, you will have to do by hand the thing which kernel automagically does for glibc - namely, define to 0: >> #undef __UAPI_DEF_SOCKADDR_IN >> /* tell kernel to not define the struct */ >> #define __UAPI_DEF_SOCKADDR_IN 0 >> #endif > We could do something like this but then we would need to keep up with > the list of all the __UAPI defines we need to suppress unwanted kernel > definitions. Looking at libc-compat.h, this list is at the moment only about 13 defines long: #define __UAPI_DEF_IN_ADDR 0 #define __UAPI_DEF_IN_IPPROTO 0 #define __UAPI_DEF_IN_PKTINFO 0 #define __UAPI_DEF_IP_MREQ 0 #define __UAPI_DEF_SOCKADDR_IN 0 #define __UAPI_DEF_IN_CLASS 0 #define __UAPI_DEF_IN6_ADDR 0 #define __UAPI_DEF_SOCKADDR_IN6 0 #define __UAPI_DEF_IPV6_MREQ 0 #define __UAPI_DEF_IPPROTO_V6 0 #define __UAPI_DEF_IPV6_OPTIONS 0 #define __UAPI_DEF_IN6_PKTINFO 0 #define __UAPI_DEF_IP6_MTUINFO 0 > What if we could get the kernel to change the #if defined(__GLIBC__) > to #if defined(__GLIBC__) || defined(__UAPI_DONTNEED_DEFS) or similar, > so that there would only be one macro we need to define, and the > kernel would then use the same logic it uses with glibc to suppress > all of these. Or ask kernel to remove "define to 0" glibc hack and ask glibc to do its own job. Why one libc should have preferential treatment? Or ask kernel to stop using structures with userspace names. This should not be that hard: struct __kernel_sockaddr_in {...} #if __KERNEL__ # define sockaddr_in __kernel_sockaddr_in #endif ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Re: musl and kernel headers [was Re: system-images 1.4.2: od is broken; bzip2 is missing] 2015-10-13 12:10 ` Denys Vlasenko @ 2015-10-13 14:16 ` Ruben Winistörfer 2015-10-13 14:53 ` Szabolcs Nagy 2015-10-13 15:10 ` Rich Felker 2 siblings, 0 replies; 18+ messages in thread From: Ruben Winistörfer @ 2015-10-13 14:16 UTC (permalink / raw) To: musl [-- Attachment #1: Type: text/plain, Size: 3375 bytes --] I just have modified the Alpine Linux kernel header patches to use them with the 4.2.3 version. (Not sure if that is a good idea, but why not try it...) So I am wondering right now, if they haven't solved the problem already you're discussing about. http://git.alpinelinux.org/cgit/aports/tree/main/linux-headers There is a patch for libc-compat.h... Maybe it helps, otherwise ignore my "interruption". ;-) Ruben Denys Vlasenko schrieb: > On Fri, Oct 9, 2015 at 9:46 PM, Rich Felker<dalias@aerifal.cx> wrote: >>> Looking at kernel's libc-compat.h, it looks like you can get away >>> with using __UAPI_DEF_foo's like this? >>> >>> >>> #if defined(__UAPI_DEF_SOCKADDR_IN)&& __UAPI_DEF_SOCKADDR_IN == 1 >>> /* kernel already defined the struct, do nothing */ >>> #else >>> struct sockaddr_in { >>> ... >>> }; >> This would address the case where the kernel header is included first, >> but it's not a case I or most of the musl community wants to support, >> because there's no guarantee that the kernel's definitions of these >> structures will actually be compatible with use elsewhere in the libc >> headers, etc. > > If kernel's definition does not match yours, there is a much > bigger problem than "includes do not compile": > kernel and userspace definitions of these structs *must* match > (modulo harmless things like different typedef names for field types). > > So in this case either kernel or libc would need to be fixed. > >> The other direction, suppressing kernel headers' definition of the >> structs, is what we want to work, but they've restricted their logic >> for that to only work when __GLIBC__ is defined. :( > > Yes, you will have to do by hand the thing which kernel > automagically does for glibc - namely, define to 0: > >>> #undef __UAPI_DEF_SOCKADDR_IN >>> /* tell kernel to not define the struct */ >>> #define __UAPI_DEF_SOCKADDR_IN 0 >>> #endif > >> We could do something like this but then we would need to keep up with >> the list of all the __UAPI defines we need to suppress unwanted kernel >> definitions. > > Looking at libc-compat.h, this list is at the moment only about > 13 defines long: > > #define __UAPI_DEF_IN_ADDR 0 > #define __UAPI_DEF_IN_IPPROTO 0 > #define __UAPI_DEF_IN_PKTINFO 0 > #define __UAPI_DEF_IP_MREQ 0 > #define __UAPI_DEF_SOCKADDR_IN 0 > #define __UAPI_DEF_IN_CLASS 0 > #define __UAPI_DEF_IN6_ADDR 0 > #define __UAPI_DEF_SOCKADDR_IN6 0 > #define __UAPI_DEF_IPV6_MREQ 0 > #define __UAPI_DEF_IPPROTO_V6 0 > #define __UAPI_DEF_IPV6_OPTIONS 0 > #define __UAPI_DEF_IN6_PKTINFO 0 > #define __UAPI_DEF_IP6_MTUINFO 0 > > >> What if we could get the kernel to change the #if defined(__GLIBC__) >> to #if defined(__GLIBC__) || defined(__UAPI_DONTNEED_DEFS) or similar, >> so that there would only be one macro we need to define, and the >> kernel would then use the same logic it uses with glibc to suppress >> all of these. > > Or ask kernel to remove "define to 0" glibc hack and ask glibc to > do its own job. Why one libc should have preferential treatment? > > Or ask kernel to stop using structures with userspace names. > This should not be that hard: > > struct __kernel_sockaddr_in {...} > #if __KERNEL__ > # define sockaddr_in __kernel_sockaddr_in > #endif [-- Attachment #2: Type: text/html, Size: 4291 bytes --] ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Re: musl and kernel headers [was Re: system-images 1.4.2: od is broken; bzip2 is missing] 2015-10-13 12:10 ` Denys Vlasenko 2015-10-13 14:16 ` Ruben Winistörfer @ 2015-10-13 14:53 ` Szabolcs Nagy 2015-10-13 15:05 ` Rich Felker 2015-10-13 15:10 ` Rich Felker 2 siblings, 1 reply; 18+ messages in thread From: Szabolcs Nagy @ 2015-10-13 14:53 UTC (permalink / raw) To: musl; +Cc: Rich Felker, Rob Landley, Aboriginal Linux * Denys Vlasenko <vda.linux@googlemail.com> [2015-10-13 14:10:24 +0200]: > On Fri, Oct 9, 2015 at 9:46 PM, Rich Felker <dalias@aerifal.cx> wrote: > >> Looking at kernel's libc-compat.h, it looks like you can get away > >> with using __UAPI_DEF_foo's like this? > >> > >> > >> #if defined(__UAPI_DEF_SOCKADDR_IN) && __UAPI_DEF_SOCKADDR_IN == 1 > >> /* kernel already defined the struct, do nothing */ > >> #else > >> struct sockaddr_in { > >> ... > >> }; > > > > This would address the case where the kernel header is included first, > > but it's not a case I or most of the musl community wants to support, > > because there's no guarantee that the kernel's definitions of these > > structures will actually be compatible with use elsewhere in the libc > > headers, etc. > > If kernel's definition does not match yours, there is a much > bigger problem than "includes do not compile": > kernel and userspace definitions of these structs *must* match > (modulo harmless things like different typedef names for field types). > > So in this case either kernel or libc would need to be fixed. > why? in practice most types are c abi compatible with the kernel because translating the types at the syscall boundary is painful/impossible. but even with compatible binary representation there is plenty space for disagreement between kernel and libc on the source level. (of course code that includes both libc and kernel headers might not care about posix namespace violations or undefined behaviour in kernel headers..) and libc-compat does not cover all conflicting cases (i assume they just add workarouds when somebody hits a conflict), e.g. sys/inotify.h and linux/inotify.h are in conflict (and linux/inotify.h is not even standard c). ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Re: musl and kernel headers [was Re: system-images 1.4.2: od is broken; bzip2 is missing] 2015-10-13 14:53 ` Szabolcs Nagy @ 2015-10-13 15:05 ` Rich Felker 2015-10-13 18:02 ` Denys Vlasenko 0 siblings, 1 reply; 18+ messages in thread From: Rich Felker @ 2015-10-13 15:05 UTC (permalink / raw) To: musl, Aboriginal Linux On Tue, Oct 13, 2015 at 04:53:35PM +0200, Szabolcs Nagy wrote: > * Denys Vlasenko <vda.linux@googlemail.com> [2015-10-13 14:10:24 +0200]: > > On Fri, Oct 9, 2015 at 9:46 PM, Rich Felker <dalias@aerifal.cx> wrote: > > >> Looking at kernel's libc-compat.h, it looks like you can get away > > >> with using __UAPI_DEF_foo's like this? > > >> > > >> > > >> #if defined(__UAPI_DEF_SOCKADDR_IN) && __UAPI_DEF_SOCKADDR_IN == 1 > > >> /* kernel already defined the struct, do nothing */ > > >> #else > > >> struct sockaddr_in { > > >> ... > > >> }; > > > > > > This would address the case where the kernel header is included first, > > > but it's not a case I or most of the musl community wants to support, > > > because there's no guarantee that the kernel's definitions of these > > > structures will actually be compatible with use elsewhere in the libc > > > headers, etc. > > > > If kernel's definition does not match yours, there is a much > > bigger problem than "includes do not compile": > > kernel and userspace definitions of these structs *must* match > > (modulo harmless things like different typedef names for field types). > > > > So in this case either kernel or libc would need to be fixed. > > why? > > in practice most types are c abi compatible with the kernel > because translating the types at the syscall boundary is > painful/impossible. > > but even with compatible binary representation there is > plenty space for disagreement between kernel and libc on > the source level. (of course code that includes both libc > and kernel headers might not care about posix namespace > violations or undefined behaviour in kernel headers..) > > and libc-compat does not cover all conflicting cases > (i assume they just add workarouds when somebody hits > a conflict), e.g. sys/inotify.h and linux/inotify.h are > in conflict (and linux/inotify.h is not even standard c). Indeed the problem here is source compatibility, not binary compatibility. Issues like names of types, choice of distinct types that have the same size and representation but which are not compatible types (which make problems if you take the address of the member, including possibly aliasing problems which are real-world bugs), etc. If static linking of libc is used along with LTO these could spill deep into internals and result in broken codegen. Rich ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Re: musl and kernel headers [was Re: system-images 1.4.2: od is broken; bzip2 is missing] 2015-10-13 15:05 ` Rich Felker @ 2015-10-13 18:02 ` Denys Vlasenko 2015-10-13 18:56 ` Rich Felker 0 siblings, 1 reply; 18+ messages in thread From: Denys Vlasenko @ 2015-10-13 18:02 UTC (permalink / raw) To: musl; +Cc: Aboriginal Linux On Tue, Oct 13, 2015 at 5:05 PM, Rich Felker <dalias@aerifal.cx> wrote: >> > > This would address the case where the kernel header is included first, >> > > but it's not a case I or most of the musl community wants to support, >> > > because there's no guarantee that the kernel's definitions of these >> > > structures will actually be compatible with use elsewhere in the libc >> > > headers, etc. >> > >> > If kernel's definition does not match yours, there is a much >> > bigger problem than "includes do not compile": >> > kernel and userspace definitions of these structs *must* match >> > (modulo harmless things like different typedef names for field types). >> > >> > So in this case either kernel or libc would need to be fixed. >> >> why? >> >> in practice most types are c abi compatible with the kernel >> because translating the types at the syscall boundary is >> painful/impossible. >> >> but even with compatible binary representation there is >> plenty space for disagreement between kernel and libc on >> the source level. (of course code that includes both libc >> and kernel headers might not care about posix namespace >> violations or undefined behaviour in kernel headers..) >> >> and libc-compat does not cover all conflicting cases >> (i assume they just add workarouds when somebody hits >> a conflict), e.g. sys/inotify.h and linux/inotify.h are >> in conflict (and linux/inotify.h is not even standard c). > > Indeed the problem here is source compatibility, not binary > compatibility. Issues like names of types, choice of distinct types > that have the same size and representation but which are not > compatible types (which make problems if you take the address of the > member, including possibly aliasing problems which are real-world > bugs), etc. It's not that bad in practice. In C, typedefs are not new types. uint16_t, u16 and __u16 are all just aliases to "unsigned short". You won't get a conflict because different typedefs were used to declare a struct member whose address you are taking and passing to some function. If signedness and width match, then it will work without casts or compiler errors. ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Re: musl and kernel headers [was Re: system-images 1.4.2: od is broken; bzip2 is missing] 2015-10-13 18:02 ` Denys Vlasenko @ 2015-10-13 18:56 ` Rich Felker 0 siblings, 0 replies; 18+ messages in thread From: Rich Felker @ 2015-10-13 18:56 UTC (permalink / raw) To: Denys Vlasenko; +Cc: musl, Aboriginal Linux On Tue, Oct 13, 2015 at 08:02:47PM +0200, Denys Vlasenko wrote: > On Tue, Oct 13, 2015 at 5:05 PM, Rich Felker <dalias@aerifal.cx> wrote: > >> > > This would address the case where the kernel header is included first, > >> > > but it's not a case I or most of the musl community wants to support, > >> > > because there's no guarantee that the kernel's definitions of these > >> > > structures will actually be compatible with use elsewhere in the libc > >> > > headers, etc. > >> > > >> > If kernel's definition does not match yours, there is a much > >> > bigger problem than "includes do not compile": > >> > kernel and userspace definitions of these structs *must* match > >> > (modulo harmless things like different typedef names for field types). > >> > > >> > So in this case either kernel or libc would need to be fixed. > >> > >> why? > >> > >> in practice most types are c abi compatible with the kernel > >> because translating the types at the syscall boundary is > >> painful/impossible. > >> > >> but even with compatible binary representation there is > >> plenty space for disagreement between kernel and libc on > >> the source level. (of course code that includes both libc > >> and kernel headers might not care about posix namespace > >> violations or undefined behaviour in kernel headers..) > >> > >> and libc-compat does not cover all conflicting cases > >> (i assume they just add workarouds when somebody hits > >> a conflict), e.g. sys/inotify.h and linux/inotify.h are > >> in conflict (and linux/inotify.h is not even standard c). > > > > Indeed the problem here is source compatibility, not binary > > compatibility. Issues like names of types, choice of distinct types > > that have the same size and representation but which are not > > compatible types (which make problems if you take the address of the > > member, including possibly aliasing problems which are real-world > > bugs), etc. > > It's not that bad in practice. > > In C, typedefs are not new types. uint16_t, u16 and __u16 > are all just aliases to "unsigned short". > > You won't get a conflict because different typedefs > were used to declare a struct member whose address you are > taking and passing to some function. > If signedness and width match, then it will work without casts > or compiler errors. The problem is that u32 might be unsigned int or unsigned long (either possible on ILP32), u64 might be unsigned long or unsigned long long (either possible on LP64), etc. Also structs with different tags are distinct types. These are not ABI issues for machine-code object files but they are source-level compatibility issues and linking (arguably ABI) issues for LTO object files. They're also C++ issues; for instance it's possible that one translation unit declares a function taking a pointer to some network function without including the kernel header, and another defines it while including the kernel header, and the struct names or types mismatch (even in a layout-compatible way), the name mangling will be different and the code won't link. Rich ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Re: musl and kernel headers [was Re: system-images 1.4.2: od is broken; bzip2 is missing] 2015-10-13 12:10 ` Denys Vlasenko 2015-10-13 14:16 ` Ruben Winistörfer 2015-10-13 14:53 ` Szabolcs Nagy @ 2015-10-13 15:10 ` Rich Felker 2015-10-13 21:55 ` Isaac Dunham 2 siblings, 1 reply; 18+ messages in thread From: Rich Felker @ 2015-10-13 15:10 UTC (permalink / raw) To: Denys Vlasenko; +Cc: Rob Landley, Aboriginal Linux, musl On Tue, Oct 13, 2015 at 02:10:24PM +0200, Denys Vlasenko wrote: > > The other direction, suppressing kernel headers' definition of the > > structs, is what we want to work, but they've restricted their logic > > for that to only work when __GLIBC__ is defined. :( > > Yes, you will have to do by hand the thing which kernel > automagically does for glibc - namely, define to 0: > > >> #undef __UAPI_DEF_SOCKADDR_IN > >> /* tell kernel to not define the struct */ > >> #define __UAPI_DEF_SOCKADDR_IN 0 > >> #endif > > > We could do something like this but then we would need to keep up with > > the list of all the __UAPI defines we need to suppress unwanted kernel > > definitions. > > Looking at libc-compat.h, this list is at the moment only about > 13 defines long: > > #define __UAPI_DEF_IN_ADDR 0 > #define __UAPI_DEF_IN_IPPROTO 0 > #define __UAPI_DEF_IN_PKTINFO 0 > #define __UAPI_DEF_IP_MREQ 0 > #define __UAPI_DEF_SOCKADDR_IN 0 > #define __UAPI_DEF_IN_CLASS 0 > #define __UAPI_DEF_IN6_ADDR 0 > #define __UAPI_DEF_SOCKADDR_IN6 0 > #define __UAPI_DEF_IPV6_MREQ 0 > #define __UAPI_DEF_IPPROTO_V6 0 > #define __UAPI_DEF_IPV6_OPTIONS 0 > #define __UAPI_DEF_IN6_PKTINFO 0 > #define __UAPI_DEF_IP6_MTUINFO 0 Thanks for making the list! For the time being I don't think it would be too objectionable to simply define these in musl. Does anyone else have an opinion on it? > > What if we could get the kernel to change the #if defined(__GLIBC__) > > to #if defined(__GLIBC__) || defined(__UAPI_DONTNEED_DEFS) or similar, > > so that there would only be one macro we need to define, and the > > kernel would then use the same logic it uses with glibc to suppress > > all of these. > > Or ask kernel to remove "define to 0" glibc hack and ask glibc to > do its own job. Why one libc should have preferential treatment? Well they don't want regressions; changing this would make old versions of glibc break. Presumably before all this UAPI "cleanup" (I'm not sure it can really be called "clean" though) there was some other mechanism that prevented things from breaking when they were included with glibc. In any case current versions of glibc would be broken by a change now. This type of interdependency is part of why I think the whole thing (trying to make kernel headers for stuf that's already defined in libc usable from userspace) was a bad idea to begin with. > Or ask kernel to stop using structures with userspace names. > This should not be that hard: > > struct __kernel_sockaddr_in {...} > #if __KERNEL__ > # define sockaddr_in __kernel_sockaddr_in > #endif Yes but then some patchup would be needed for uClibc and other libcs that actually want the kernel to define the structs. This would work better if the #define was in the other direction: # define __kernel_sockaddr_in sockaddr_in and done _before_ declaration of the structure, so that the actual struct tag would end up being the desired name in this case (both for kernel-internal and for uClibc, etc. use). Rich ^ permalink raw reply [flat|nested] 18+ messages in thread
* Re: Re: musl and kernel headers [was Re: system-images 1.4.2: od is broken; bzip2 is missing] 2015-10-13 15:10 ` Rich Felker @ 2015-10-13 21:55 ` Isaac Dunham 0 siblings, 0 replies; 18+ messages in thread From: Isaac Dunham @ 2015-10-13 21:55 UTC (permalink / raw) To: musl; +Cc: Denys Vlasenko, Aboriginal Linux On Tue, Oct 13, 2015 at 11:10:49AM -0400, Rich Felker wrote: > On Tue, Oct 13, 2015 at 02:10:24PM +0200, Denys Vlasenko wrote: > > > The other direction, suppressing kernel headers' definition of the > > > structs, is what we want to work, but they've restricted their logic > > > for that to only work when __GLIBC__ is defined. :( > > > > Yes, you will have to do by hand the thing which kernel > > automagically does for glibc - namely, define to 0: > > > > >> #undef __UAPI_DEF_SOCKADDR_IN > > >> /* tell kernel to not define the struct */ > > >> #define __UAPI_DEF_SOCKADDR_IN 0 > > >> #endif > > > > > We could do something like this but then we would need to keep up with > > > the list of all the __UAPI defines we need to suppress unwanted kernel > > > definitions. > > > > Looking at libc-compat.h, this list is at the moment only about > > 13 defines long: > > > > #define __UAPI_DEF_IN_ADDR 0 > > #define __UAPI_DEF_IN_IPPROTO 0 > > #define __UAPI_DEF_IN_PKTINFO 0 > > #define __UAPI_DEF_IP_MREQ 0 > > #define __UAPI_DEF_SOCKADDR_IN 0 > > #define __UAPI_DEF_IN_CLASS 0 > > #define __UAPI_DEF_IN6_ADDR 0 > > #define __UAPI_DEF_SOCKADDR_IN6 0 > > #define __UAPI_DEF_IPV6_MREQ 0 > > #define __UAPI_DEF_IPPROTO_V6 0 > > #define __UAPI_DEF_IPV6_OPTIONS 0 > > #define __UAPI_DEF_IN6_PKTINFO 0 > > #define __UAPI_DEF_IP6_MTUINFO 0 > > Thanks for making the list! For the time being I don't think it would > be too objectionable to simply define these in musl. Does anyone else > have an opinion on it? I'd like to see it for the time being, though long-term I still assume that the only real solution would be something like the proposed change in libc-compat.h: -#ifdef __GLIBC__ +#if defined(__GLIBC__) || defined(__UAPI_DONTNEED_DEFS) Thanks, Isaac Dunham ^ permalink raw reply [flat|nested] 18+ messages in thread
end of thread, other threads:[~2015-10-13 21:55 UTC | newest] Thread overview: 18+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <CAK1hOcNm-rwGdr1_THZoHA-TTxOjpzUU=Lnraqt6mj+JmBdvFw@mail.gmail.com> [not found] ` <5612925A.4070402@landley.net> 2015-10-06 1:44 ` musl and kernel headers [was Re: system-images 1.4.2: od is broken; bzip2 is missing] Rich Felker 2015-10-06 2:24 ` Rob Landley 2015-10-06 11:01 ` Szabolcs Nagy 2015-10-06 14:30 ` Rich Felker 2015-10-06 16:05 ` Denys Vlasenko 2015-10-06 16:09 ` Denys Vlasenko 2015-10-08 16:58 ` Rich Felker 2015-10-09 19:11 ` Denys Vlasenko 2015-10-09 19:46 ` Rich Felker 2015-10-10 4:56 ` Rob Landley 2015-10-13 12:10 ` Denys Vlasenko 2015-10-13 14:16 ` Ruben Winistörfer 2015-10-13 14:53 ` Szabolcs Nagy 2015-10-13 15:05 ` Rich Felker 2015-10-13 18:02 ` Denys Vlasenko 2015-10-13 18:56 ` Rich Felker 2015-10-13 15:10 ` Rich Felker 2015-10-13 21:55 ` Isaac Dunham
Code repositories for project(s) associated with this public inbox https://git.vuxu.org/mirror/musl/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).