[ISSUE] qemu-aarch64-static works in glibc but the musl version is missing certain network functions.

[ISSUE] qemu-aarch64-static works in glibc but the musl version is missing certain network functions.

[ejolson2005]

[-- Attachment #1: Type: text/plain, Size: 3195 bytes --]

New issue by ejolson2005 on void-packages repository

https://github.com/void-linux/void-packages/issues/23557

Description:
<!-- Don't request update of package. We have a script for that. https://alpha.de.repo.voidlinux.org/void-updates/void-updates.txt . However, a quality pull request may help. -->
### System
```
$ xuname
Void 5.4.50_1 x86_64-musl AuthenticAMD notuptodate rF
$ xbps-query -p pkgver qemu-user-static
qemu-user-static-5.0.0_2
```

### Expected behavior'
When used in a qemu chroot expect networking to work.

### Actual behavior

DNS lookups don't work nor does ip address find the network devices.  Note that specifying network addresses by IP number allows things like ssh to connect, but this is not a DNS problem because /etc/resolv.conf is correct and moreover, things begin working if the qemu-aarch64-static binary from the glibc version of Void Linux is copied in order the one compiled with musl and reregistered.

### Steps to reproduce the behavior

With the musl version of qemu-aarch64-static copied into an aarch64 chroot in which a correct resolv.conf has been placed and the proc, dev and sys file systems mounted I get
```
[root@silver images]# chroot iroot
root@silver:/# ping google.com
ping: google.com: Temporary failure in name resolution
root@silver:/# ip address
netlink receive error Message too long (90)
Dump terminated
```
Using exact same chroot but with the glibc version of qemu-aarch64-static copied in from a glibc version of Void  Linux and reregistered I get
```
[root@silver images]# chroot iroot
root@silver:/# ping google.com
PING google.com (172.217.5.206) 56(84) bytes of data.
64 bytes from lax28s10-in-f206.1e100.net (172.217.5.206): icmp_seq=1 ttl=115 time=39.5 ms
64 bytes from lax28s10-in-f206.1e100.net (172.217.5.206): icmp_seq=2 ttl=115 time=38.7 ms
64 bytes from lax28s10-in-f206.1e100.net (172.217.5.206): icmp_seq=3 ttl=115 time=37.4 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 6ms
rtt min/avg/max/mdev = 37.354/38.513/39.486/0.880 ms
root@silver:/# ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: esan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 68:1c:a2:12:af:95 brd ff:ff:ff:ff:ff:ff
    inet 192.168.47.254/25 brd 192.168.47.255 scope global noprefixroute esan
       valid_lft forever preferred_lft forever
    inet6 fe80::1ba9:d1d:c8aa:20f7/64 scope link 
       valid_lft forever preferred_lft forever
```
which is correct.  This error appears related to the musl library and possibly related to

https://github.com/docker/for-mac/issues/1621

which identifies a similar build problem for qemu-aarch64-static on Mac OS.  I've not taken any look at the code, but expect somewhere the build process for aarch64 checks for glibc and assumes it's not Linux otherwise.

Re: qemu-aarch64-static works in glibc but the musl version is missing certain network functions.

[ericonr]

[-- Attachment #1: Type: text/plain, Size: 348 bytes --]

New comment by ericonr on void-packages repository

https://github.com/void-linux/void-packages/issues/23557#issuecomment-657949062

Comment:
Can you show your `/etc/resolv.conf` file? musl has different behavior from glibc in regards to DNS. You can find it here https://wiki.musl-libc.org/functional-differences-from-glibc.html#Name-Resolver/DNS

Re: qemu-aarch64-static works in glibc but the musl version is missing certain network functions.

[ejolson2005]

[-- Attachment #1: Type: text/plain, Size: 642 bytes --]

New comment by ejolson2005 on void-packages repository

https://github.com/void-linux/void-packages/issues/23557#issuecomment-657950605

Comment:
The resolv.conf in the chroot looks like
```
domain wulf
search wulf math.unr.edu
nameserver 192.168.174.129
```
Note the root filesystem in the chroot did not come from an aarch64 version of Void Linux but is aarch64 Debian.  My understanding is that qemu-aarch64-static is statically linked (whether from glibc or musl versions of Void), so copying them around should just work, and it does at least copying the one from the glibc system to the musl system.  Everything else is the same.

Re: qemu-aarch64-static works in glibc but the musl version is missing certain network functions.

[ejolson2005]

[-- Attachment #1: Type: text/plain, Size: 775 bytes --]

New comment by ejolson2005 on void-packages repository

https://github.com/void-linux/void-packages/issues/23557#issuecomment-657950605

Comment:
The resolv.conf in the chroot looks like
```
domain wulf
search wulf math.unr.edu
nameserver 192.168.174.129
```
The name server is running on the inteternal LAN but accessing one of the Google public name servers doesn't change the behavior.

Note the root filesystem in the chroot did not come from an aarch64 version of Void Linux but is aarch64 Debian.  My understanding is that qemu-aarch64-static is statically linked (whether from glibc or musl versions of Void), so copying them around should just work, and it does at least copying the one from the glibc system to the musl system.  Everything else is the same.

Re: qemu-aarch64-static works in glibc but the musl version is missing certain network functions.

[ejolson2005]

[-- Attachment #1: Type: text/plain, Size: 773 bytes --]

New comment by ejolson2005 on void-packages repository

https://github.com/void-linux/void-packages/issues/23557#issuecomment-657950605

Comment:
The resolv.conf in the chroot looks like
```
domain wulf
search wulf math.unr.edu
nameserver 192.168.174.129
```
The name server is running on the internal LAN but accessing one of the Google public name servers doesn't change the behavior.

Note the root filesystem in the chroot did not come from an aarch64 version of Void Linux but is aarch64 Debian.  My understanding is that qemu-aarch64-static is statically linked (whether from glibc or musl versions of Void), so copying them around should just work, and it does at least copying the one from the glibc system to the musl system.  Everything else is the same.

Re: qemu-aarch64-static works in glibc but the musl version is missing certain network functions.

[ejolson2005]

[-- Attachment #1: Type: text/plain, Size: 850 bytes --]

New comment by ejolson2005 on void-packages repository

https://github.com/void-linux/void-packages/issues/23557#issuecomment-657950605

Comment:
The resolv.conf in the chroot looks like
```
domain wulf
search wulf math.unr.edu
nameserver 192.168.174.129
```
The name server is running on the internal LAN but accessing one of the Google public name servers doesn't change the behavior.  [Edit: if I delete the domain and search lines, again nothing changes].

Note the root filesystem in the chroot did not come from an aarch64 version of Void Linux but is aarch64 Debian.  My understanding is that qemu-aarch64-static is statically linked (whether from glibc or musl versions of Void), so copying them around should just work, and it does at least copying the one from the glibc system to the musl system.  Everything else is the same.

Re: qemu-aarch64-static works in glibc but the musl version is missing certain network functions.

[ejolson2005]

[-- Attachment #1: Type: text/plain, Size: 930 bytes --]

New comment by ejolson2005 on void-packages repository

https://github.com/void-linux/void-packages/issues/23557#issuecomment-657950605

Comment:
The resolv.conf in the chroot looks like
```
domain wulf
search wulf math.unr.edu
nameserver 192.168.174.129
```
The name server is running on the internal LAN but accessing one of the Google public name servers doesn't change the behavior.  [Edit: if I delete the domain and search lines, again nothing changes].

Note the root filesystem in the chroot did not come from an aarch64 version of Void Linux but is aarch64 Debian.  [Edit: I get similar results with an aarch64 version of Gentoo in the chroot].  My understanding is that qemu-aarch64-static is statically linked (whether from glibc or musl versions of Void), so copying them around should just work, and it does at least copying the one from the glibc system to the musl system.  Everything else is the same.

Re: qemu-aarch64-static works in glibc but the musl version is missing certain network functions.

[sgn]

[-- Attachment #1: Type: text/plain, Size: 153 bytes --]

New comment by sgn on void-packages repository

https://github.com/void-linux/void-packages/issues/23557#issuecomment-660429026

Comment:
Bug confirmed.

Re: qemu-aarch64-static works in glibc but the musl version is missing certain network functions.

[ericonr]

[-- Attachment #1: Type: text/plain, Size: 178 bytes --]

New comment by ericonr on void-packages repository

https://github.com/void-linux/void-packages/issues/23557#issuecomment-718319991

Comment:
Is this still happening with 5.1.0?

Re: qemu-aarch64-static works in glibc but the musl version is missing certain network functions.

[richfelker]

[-- Attachment #1: Type: text/plain, Size: 638 bytes --]

New comment by richfelker on void-packages repository

https://github.com/void-linux/void-packages/issues/23557#issuecomment-718324863

Comment:
Try minimizing this to a single guest binary execution (ideally a static linked musl one so it has minimal spurious syscalls) rather than having to be in the chroot, and executing it once with `-strace` passed to the qemu, and again with `strace` on the host, to see if we can get a picture of what's going on and where it's failing. This is probably qemu using some libc struct and thinking it matches a kernel uapi struct where it doesn't, but it might just be a bug in musl aarch64 port.

Re: qemu-aarch64-static works in glibc but the musl version is missing certain network functions.

[ericonr]

[-- Attachment #1: Type: text/plain, Size: 986 bytes --]

New comment by ericonr on void-packages repository

https://github.com/void-linux/void-packages/issues/23557#issuecomment-718364461

Comment:
I'm using https://github.com/ericonr/purr-c , because it's a reasonably simple project of my own and I already had the code written.

Building an aarch64-musl static binary and using x86_64-musl's qemu, it all seems to work fine. The issue, instead, seems to be when using x86_64-musl's qemu with a binary built for aarch64-glibc, where I get "Temporary failure in name resolution". Using x86_64-glibc's qemu, all works fine.

Command line used, because, as warned by the linker, statically linking `getaddrinfo` requires dynamic modules at runtime: `QEMU_LD_PREFIX=masterdir-glibc/usr/aarch64-linux-gnu qemu-aarch64-static masterdir-glibc/purr-c/purr -d r https://google.com/`.

I have a gist for the 4 combinations of `qemu -strace`, `strace qemu` and glibc, musl hosts: https://gist.github.com/ericonr/95a41049b96e6256ab08a40bab311994

Re: qemu-aarch64-static works in glibc but the musl version is missing certain network functions.

[richfelker]

[-- Attachment #1: Type: text/plain, Size: 936 bytes --]

New comment by richfelker on void-packages repository

https://github.com/void-linux/void-packages/issues/23557#issuecomment-718392360

Comment:
OK, this turned out to be from `sendmsg` failing with `EMSGSIZE`, which is part of a systemic issue of qemu `linux-user/syscall.c` mixing libc types and raw syscalls. This was introduced in 2016 as part of "safe syscalls" for fixing interaction of emulated syscalls with signals, and the fix basically precluded ever being able to run qemu linux-user guest on non-linux host, and made it hard to run even on a linux host if the libc and kernel uapi types don't entirely match.

Working around the `sendmsg` problem is fairly easy just by making `do_sendrecvmsg_locked` zero-initialize its `struct msghdr`, but there are a large number of othe `safe_*` calls in that file that are making invalid assumptions about libc and kernel types matching, especially in the way of time64 breakage.

Re: [ISSUE] [CLOSED] qemu-aarch64-static works in glibc but the musl version is missing certain network functions.

[ericonr]

[-- Attachment #1: Type: text/plain, Size: 3185 bytes --]

Closed issue by ejolson2005 on void-packages repository

https://github.com/void-linux/void-packages/issues/23557

Description:
<!-- Don't request update of package. We have a script for that. https://alpha.de.repo.voidlinux.org/void-updates/void-updates.txt . However, a quality pull request may help. -->
### System
```
$ xuname
Void 5.4.50_1 x86_64-musl AuthenticAMD notuptodate rF
$ xbps-query -p pkgver qemu-user-static
qemu-user-static-5.0.0_2
```

### Expected behavior'
When used in a qemu chroot expect networking to work.

### Actual behavior

DNS lookups don't work nor does "ip address" find the network devices.  Note that specifying network addresses by IP number allows things like ssh to connect, but this is not a DNS problem because /etc/resolv.conf is correct and moreover things begin working if the qemu-aarch64-static binary from the glibc version of Void Linux is copied in over the one included with musl and reregistered.

### Steps to reproduce the behavior

With the musl version of qemu-aarch64-static copied into an aarch64 chroot in which a correct resolv.conf has been placed and the proc, dev and sys file systems mounted I get
```
[root@silver images]# chroot iroot
root@silver:/# ping google.com
ping: google.com: Temporary failure in name resolution
root@silver:/# ip address
netlink receive error Message too long (90)
Dump terminated
```
Using exact same chroot but with the glibc version of qemu-aarch64-static copied in from a glibc version of Void  Linux and reregistered I get
```
[root@silver images]# chroot iroot
root@silver:/# ping google.com
PING google.com (172.217.5.206) 56(84) bytes of data.
64 bytes from lax28s10-in-f206.1e100.net (172.217.5.206): icmp_seq=1 ttl=115 time=39.5 ms
64 bytes from lax28s10-in-f206.1e100.net (172.217.5.206): icmp_seq=2 ttl=115 time=38.7 ms
64 bytes from lax28s10-in-f206.1e100.net (172.217.5.206): icmp_seq=3 ttl=115 time=37.4 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 6ms
rtt min/avg/max/mdev = 37.354/38.513/39.486/0.880 ms
root@silver:/# ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: esan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 68:1c:a2:12:af:95 brd ff:ff:ff:ff:ff:ff
    inet 192.168.47.254/25 brd 192.168.47.255 scope global noprefixroute esan
       valid_lft forever preferred_lft forever
    inet6 fe80::1ba9:d1d:c8aa:20f7/64 scope link 
       valid_lft forever preferred_lft forever
```
which is correct.  This error appears caused by the musl library and possibly related to

https://github.com/docker/for-mac/issues/1621

which identifies a similar build problem for qemu-aarch64-static on Mac OS.  I've not taken any look at the code, but expect somewhere the build process checks for glibc and assumes it's not Linux otherwise.