Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 333 bytes --]

New comment by pullmoll on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489323846
Comment:
It seems the underlying problem is fixed. I could now cross build `webkit2gtk` using the official repo from `x86_64` to `armv7l-musl`. If anyone can confirm this we can remove the `broken=`.

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 312 bytes --]

New comment by newbluemoon on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489329790
Comment:
I’m currently building it (x86_64-musl --> armv7l-musl). Will test it on my Raspberry Pi when done to make sure that there are no runtime issues and report back.

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 327 bytes --]

New comment by newbluemoon on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489358528
Comment:
Sorry to report, but the x86_64-musl --> armv7l-musl build doesn’t finish here (with everything up-to-date). Same issue as above: `qemu-arm-static` just seems to be stuck... :(

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 399 bytes --]

New comment by pullmoll on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489360839
Comment:
Hmm.. I'll verify again later or tomorrow. It worked here with a freshly zapped and `binary-boostrap`ped environment for `x86_64` using packages from `alpha.de.repo.voidlinux.org/current`. Maybe your `repocache` is different? Mine was empty before I tried.

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 340 bytes --]

New comment by newbluemoon on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489363902
Comment:
I zapped the masterdir first, too, and binary-bootstrapped from the same repo to have a clean environment. But I’m on a musl-system, not just a musl-chroot, if that’s of any relevance.

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 1103 bytes --]

New comment by newbluemoon on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489365964
Comment:
Hmm, maybe I’m not getting something, but in `/usr/bin/g-ir-scanner`

```
#!/usr/bin/env bash
# Check if we are running in an xbps-src environment and run the wrapper if that
# is the case.
if [ -n "$XBPS_CROSS_BASE" -a -n "$XBPS_TARGET_MACHINE" -a -n "$XBPS_VERSION" ]; then
	# This prevents g-ir-scanner from writing cache data to $HOME
	export GI_SCANNER_DISABLE_CACHE=1
	
	exec /usr/bin/g-ir-scanner.wrapped \
				 --use-binary-wrapper=/usr/bin/g-ir-scanner-qemuwrapper \
				 --use-ldd-wrapper=/usr/bin/g-ir-scanner-lddwrapper \
				 --add-include-path=${XBPS_CROSS_BASE}/usr/share/gir-1.0 \
				 --add-include-path=${XBPS_CROSS_BASE}/usr/lib/gir-1.0 \
				 "${@//-I\/usr\/include/-I${XBPS_CROSS_BASE}\/usr\/include}"
fi
	
exec /usr/bin/g-ir-scanner.wrapped "$@"
```

shouldn’t the last line belong to the ‘else’ branch of the if clause? Else there are two `exec`s in the cross case.
Same with `g-ir-compiler`.

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 499 bytes --]

New comment by pullmoll on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489380804
Comment:
Oh, our builders are `x86_64` with `glibc` AFAIK. And that was what I tried.
I haven't tried cross compiling from `x86_64-musl` to `armv7l-musl` yet.

For the `g-ir-scanner` wrapper I think you are basically right, but then `exec` won't return anyway and thus it's okay to have a default path not inside an `else`. AFAIK `exec` is implicitly `noreturn`.

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 587 bytes --]

New comment by newbluemoon on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489389490
Comment:
I’ll try again with x86_64 --> armv7l-musl today. But my hardware isn’t that new so it’ll take some time. ;)
But I assume you’re using a x86_64-musl chroot, right?

As for the `exec`, now that you mention `noreturn` something made “click”. :) Was trying to figure out why there are two qemu instances running. But it seems they’re invoked in the makefile. Only with `-j1` there’s only one instance.

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 344 bytes --]

New comment by pullmoll on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489425806
Comment:
No, I used an `x86_64` chroot and I am on a x86_64 glibc system. Just now trying to cross compile from a `x86_64-musl` chroot with official repo packages. It already succeeded with my local packages.

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 580 bytes --]

New comment by newbluemoon on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489426962
Comment:
Hmm, I tried x86_64 glibc host --> x86_64-musl chroot --> armv7l-musl target with a freshly git-cloned void-packages repo (everything from scratch) and `qemu-arm-static` just got stuck again.
Same with x86_64-musl host --> x86_64 chroot --> armv7l-musl target (though I think that combination doesn’t really make sense, but just for the sake of it ;)).
Still in progress is x86_64 host --> x86_64 chroot --> armv7l-musl target.

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 573 bytes --]

New comment by pullmoll on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489429219
Comment:
Here also the final test succeeded: x86_64 host, x86_64-musl chroot, cross to armv7l-musl, all with official repo packages. Whatever the difference between your and my environment is, it's probably not in the packages per se. It could be something like a wrong `-march=native` somewhere and your host's CPU being different from mine (AMD Ryzen 1950X).
I wonder if I should give the builders another try at compiling webkit2gtk...

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 1279 bytes --]

New comment by newbluemoon on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489439856
Comment:
I did some more digging:

There is a temporary directory `/builddir/webkitgtk-2.24.1/build/Source/WebKit/tmp-<something>` (it gets deleted when the build is aborted) with an executable `WebKit2-4.0` which is executed basically as

`./WebKit2-4.0 --introspect-dump=functions.txt,dump.xml`

which yields the introspection data.

Copied to my Raspberry Pi and executed everything is fine. However, when run with `qemu-arm-static` the output file `dump.xml` is truncated, or to be more precise, running `./WebKit2-4.0` just doesn’t continue and hangs.

So I chrooted into the masterdir and ran

`qemu-arm-static -d unimp -E LD_LIBRARY_PATH="/usr/armv7l-linux-musleabihf/usr/lib" -L "/usr/armv7l-linux-musleabihf" ./WebKit2-4.0 --introspect-dump=functions.txt,dump.xml`

manually and got

`Unsupported syscall: 389`

which is the `membarrier` syscall which if I recall correctly musl started using with 1.1.22.

So could this be the culprit? But then I have no idea why it’s working for @pullmoll (maybe it’s random?).
For me x86_64 host --> x86_64 chroot --> armv7l-musl target also doesn’t work.

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 331 bytes --]

New comment by pullmoll on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489440872
Comment:
It probably is the culprit, however I don't understand why you get to see this `Unsupported syscall: 389`  message. You have a kernel where this is enabled `grep MEMBARRIER /boot/config*`?

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 239 bytes --]

New comment by newbluemoon on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489441225
Comment:
I’m using the latest standard kernel (5.0.12), nothing fancy, and yes it’s enabled.

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 180 bytes --]

New comment by pullmoll on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489441494
Comment:
:man_shrugging: now I'm out of ideas.

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 286 bytes --]

New comment by newbluemoon on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489441834
Comment:
Maybe we should stop for today and do something entirely different. I found that solutions tend to present themselves when doing nothing. :)

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 242 bytes --]

New comment by jnbr on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489443959
Comment:
Our qemu is built without membarrier support. Maybe build qemu with `--enable-membarrier`and try again.

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 185 bytes --]

New comment by newbluemoon on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489444777
Comment:
Thanks @jnbr, compiling right now... :)

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 770 bytes --]

New comment by newbluemoon on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489451930
Comment:
No, sadly a membarrier-enabled qemu doesn’t work for me either.
When I run the qemu command from above with option `qemu-arm-static -strace ...` it endlessly prints lines like 

```mremap(-1277952,4096,8192,0,0,0) = -1 errno=12 (Out of memory)```

(though my machine is very far from memory exhaustion).

So maybe it’s not (solely) membarrier. I found this thread with similar errors on the musl mailing list:
https://www.openwall.com/lists/musl/2017/06/21/2
which suggests a bug in qemu. And maybe it is also subject to some features of the hardware used, because it’s working sometimes, apparently.

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 287 bytes --]

New comment by jnbr on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489453284
Comment:
Fwiw: with membarrier enabled in qemu-user-static, it completed here on a x86_64-glibc host (Ryzen 1700 CPU), x86_64-musl masterdir for armv7l-musl.

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 385 bytes --]

New comment by newbluemoon on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489454087
Comment:
@jnbr I think it would also work for you with the standard qemu, because it did for @pullmoll (also Ryzen). I tried on a ~10 year old Athlon II X2. So I guess it is indeed some hardware feature which the builders are probably also lacking.

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 640 bytes --]

New comment by pullmoll on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489455769
Comment:
I somehow suspect a connection to the recently change transparent hugepage handling in the kernels. I have `transparent_hugepage=madvise` in `/etc/default/grub`. I see no usage of `madvise(2)` in the qemu linux-user code, though.

If it is really an issue with qemu's `target_mremap()` function when `new_size > old_size` it could perhaps be the function `mmap_reserve()` in `linux-user/mmap.c:580` which is failing because it cannot find a consecutive range of pages. Just a wild guess, though.

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 641 bytes --]

New comment by pullmoll on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489455769
Comment:
I somehow suspect a connection to the recently changed transparent hugepage handling in the kernels. I have `transparent_hugepage=madvise` in `/etc/default/grub`. I see no usage of `madvise(2)` in the qemu linux-user code, though.

If it is really an issue with qemu's `target_mremap()` function when `new_size > old_size` it could perhaps be the function `mmap_reserve()` in `linux-user/mmap.c:580` which is failing because it cannot find a consecutive range of pages. Just a wild guess, though.

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 203 bytes --]

New comment by jnbr on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489456985
Comment:
Forget what I've said before, stupid me forgot the `-a` Flag ...

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 287 bytes --]

New comment by jnbr on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489453284
Comment:
Fwiw: with membarrier enabled in qemu-user-static, it completed here on a x86_64-glibc host (Ryzen 1700 CPU), x86_64-musl masterdir for armv7l-musl.

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 893 bytes --]

New comment by pullmoll on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489457846
Comment:
Hmm.. this code seems wrong for me `linux-user/mmap.c:726…`:
```
    726         int prot = 0;
    727         if (reserved_va && old_size < new_size) {
    728             abi_ulong addr;
    729             for (addr = old_addr + old_size;
    730                  addr < old_addr + new_size;
    731                  addr++) {
    732                 prot |= page_get_flags(addr);
    733             }
    734         }
```
The loop with `addr` is for every byte of the range and not for every page using `addr += TARGET_PAGE_SIZE` like it is done in other places in the same source. If you `mremap()` and increase size by several MiB that will create millions of calls to `page_get_flags(addr)`. Or is this really intended here?

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 1173 bytes --]

New comment by pullmoll on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489457846
Comment:
Hmm.. this code seems wrong for me `linux-user/mmap.c:726…`:
```
    726         int prot = 0;
    727         if (reserved_va && old_size < new_size) {
    728             abi_ulong addr;
    729             for (addr = old_addr + old_size;
    730                  addr < old_addr + new_size;
    731                  addr++) {
    732                 prot |= page_get_flags(addr);
    733             }
    734         }
```
The loop with `addr` is for every byte of the range and not for every page using `addr += TARGET_PAGE_SIZE` like it is done in other places in the same source. If you `mremap()` and increase size by several MiB that will create millions of calls to `page_get_flags(addr)`. Or is this really intended here?

Thinking about this and looking at the implementation of page_get_flags() it should suffice to skip addr in TARGET_PAGE_SIZE steps and in addition test the hight address `old_addr + new_size - 1`.

But then all this does not seem to be a possible cause for the ENOMEM error.

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 710 bytes --]

New comment by pullmoll on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-490076227
Comment:
To simplify debugging (cross) packages there's now `GIR_EXTRA_OPTIONS` which can be specified in a template like e.g. `GIR_EXTRA_OPTIONS="-strace"` to see what happens in the regular build log.

What is missing, also for `GIR_EXTRA_LIBS_PATH`, is a `unset GIR_EXTRA_…` in the right place, or otherwise dependencies of a package are built with the same settings for these environment variables. This unset should perhaps be done in `common/xbps-src/shutils/common.sh` function `setup_pkg()` somewhere around `# Start with a sane environment`. What do you think?

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 210 bytes --]

New comment by xtraeme on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-490077874
Comment:
You can do this currently in `common/environment/setup/sourcepkg.sh`

Re: [ISSUE] [CLOSED] Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 1663 bytes --]

Closed issue by pullmoll on void-packages repository

https://github.com/void-linux/void-packages/issues/11426
Description: As you may have noticed in the commit logs we have a problem with cross `arm*-musl` trying to run the cross `g-ir-scanner` with `g-ir-scanner-qemuwrapper`. It hangs for (literally) hours and does not finish. It's seemingly not a problem of the builders or age of (re-)built packages because it happens for me locally as well and I built all of the dependencies here.

What I know from looking at `top` when the build hangs is that it seems as if two processes in parallel are running `g-ir-scanner-qemuwrapper` to scan the introspection info. It does not happen with `arm*-gnu` cross builds so it has to be something specific to Musl libc or the cross environment for Musl. I wanted to try if it also happens when using a 32bit environment (i686) for cross compiling `webkit2gtk` for e.g. `armv7l-musl` to see if it's perhaps a 64 vs. 32 bit build environment issue.

For now I disabled `gir` for cross `webkit2gtk` to `arm*-musl` but of course this makes packages depending on the introspection files of `webkit2gtk` incomplete or even fail.

In case you find time to test this on your local box, you'd have to add `gir` to `build_options_default` in line 43 of `srcpkgs/webkit2gtk/template` and take a look at the processes running after the binaries were built (i.e. after cmake prints 100%).

I have no idea yet why it works for `arm*-gnu` but not for `arm*-musl` as for the most part we solved the cross `gobject-introspection` problems with `*-musl` and cross `webkit2gtk` to `aarch64-musl` or `ppc64-musl` work just fine.

Re: Cross arm*-musl hangs when runing g-ir-scanner-qemuwrapper

[voidlinux-github]

[-- Attachment #1: Type: text/plain, Size: 325 bytes --]

New comment by newbluemoon on void-packages repository

https://github.com/void-linux/void-packages/issues/11426#issuecomment-489358528
Comment:
Sorry to report, but the x86_64-musl --> armv7l-musl build doesn’t finish here (with everying up-to-date). Same issue as above: `qemu-arm-static` just seems to be stuck... :(