[ISSUE] Package for containerd on aarch64-musl produces segfault on service startup

[ISSUE] Package for containerd on aarch64-musl produces segfault on service startup

[moabeat-berlin]

[-- Attachment #1: Type: text/plain, Size: 2120 bytes --]

New issue by moabeat-berlin on void-packages repository

https://github.com/void-linux/void-packages/issues/53763

Description:
### Is this a new report?

Yes

### System Info

Void 6.6.42_1 aarch64-musl

### Package(s) Affected

containerd-2.0.1_1

### Does a report exist for this bug with the project's home (upstream) and/or another distro?

_No response_

### Expected behaviour

The service containerd should start up normally, when added to the `default` runlevel.

### Actual behaviour

The service containerd crashed with a segfault. Log from svlogtail are below.

```
2024-12-30T12:39:25.16653 daemon.notice: Dec 30 12:39:25 docker: time="2024-12-30T13:39:25.166262821+01:00" level=info msg="Starting up"
2024-12-30T12:39:25.16741 daemon.notice: Dec 30 12:39:25 docker: time="2024-12-30T13:39:25.167289121+01:00" level=info msg="containerd not running, starting managed containerd"
2024-12-30T12:39:25.16914 daemon.notice: Dec 30 12:39:25 docker: time="2024-12-30T13:39:25.168702786+01:00" level=info msg="started new containerd process" address=/var/run/docker/containerd/containerd.sock module=libcontainerd pid=1617
2024-12-30T12:39:25.16960 daemon.notice: Dec 30 12:39:25 docker: time="2024-12-30T13:39:25.169396684+01:00" level=error msg="containerd did not exit successfully" error="signal: segmentation fault" module=libcontainerd
2024-12-30T12:39:34.77693 daemon.notice: Dec 30 12:39:34 docker: time="2024-12-30T13:39:34.776333061+01:00" level=info msg="started new containerd process" address=/var/run/docker/containerd/containerd.sock module=libcontainerd pid=1618
2024-12-30T12:39:34.77865 daemon.notice: Dec 30 12:39:34 docker: time="2024-12-30T13:39:34.778053907+01:00" level=error msg="containerd did not exit successfully" error="signal: segmentation fault" module=libcontainerd
2024-12-30T12:39:40.16892 daemon.notice: Dec 30 12:39:40 docker: failed to start containerd: timeout waiting for containerd to start
``` 

Downgrade to version 1.7.17_2 fixed the issue.

### Steps to reproduce

1. Update containerd package to version 2.0.1
2. Reboot
3. containerd crashes as described

Re: Package for containerd on aarch64-musl produces segfault on service startup

[classabbyamp]

[-- Attachment #1: Type: text/plain, Size: 207 bytes --]

New comment by classabbyamp on void-packages repository

https://github.com/void-linux/void-packages/issues/53763#issuecomment-2565732762

Comment:
what happens when you run `containerd` as root on its own?

Re: Package for containerd on aarch64-musl produces segfault on service startup

[moabeat-berlin]

[-- Attachment #1: Type: text/plain, Size: 197 bytes --]

New comment by moabeat-berlin on void-packages repository

https://github.com/void-linux/void-packages/issues/53763#issuecomment-2565979412

Comment:
Same result. Also throws a segmentation fault.

Re: Package for containerd on aarch64-musl produces segfault on service startup

[classabbyamp]

[-- Attachment #1: Type: text/plain, Size: 228 bytes --]

New comment by classabbyamp on void-packages repository

https://github.com/void-linux/void-packages/issues/53763#issuecomment-2565995441

Comment:
can you [get a backtrace](https://placeviolette.net/tech-notes/001-backtrace/)?

Re: Package for containerd on aarch64-musl produces segfault on service startup

[moabeat-berlin]

[-- Attachment #1: Type: text/plain, Size: 1431 bytes --]

New comment by moabeat-berlin on void-packages repository

https://github.com/void-linux/void-packages/issues/53763#issuecomment-2566002186

Comment:
Sure, I was able to do that. Unfortunately, it seem to not give any useful results.

```
❯ gdb -q containerd
Reading symbols from containerd...
Reading symbols from /usr/lib/debug//usr/bin/containerd...
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /usr/lib/debug/usr/bin/containerd.
Use `info auto-load python-scripts [REGEXP]' to list them.
(gdb) run
Starting program: /usr/bin/containerd
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.

Program received signal SIGSEGV, Segmentation fault.
0x0000007ff7fa75a4 in ?? ()
(gdb) bt
#0  0x0000007ff7fa75a4 in ?? ()
#1  0x0000007ff7ffedd0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
``` 

As far as I can assess it, all relevant debug packed were installed beforehand.

```
containerd-dbg-2.0.1_1: configuring ...
containerd-dbg-2.0.1_1: installed successfully.
libseccomp-dbg-2.5.5_1: configuring ...
libseccomp-dbg-2.5.5_1: installed successfully.
musl-dbg-1.1.24_22: configuring ...
musl-dbg-1.1.24_22: installed successfully.
runc-dbg-1.1.13_2: configuring ...
runc-dbg-1.1.13_2: installed successfully.
``` 

Re: Package for containerd on aarch64-musl produces segfault on service startup

[moabeat-berlin]

[-- Attachment #1: Type: text/plain, Size: 1432 bytes --]

New comment by moabeat-berlin on void-packages repository

https://github.com/void-linux/void-packages/issues/53763#issuecomment-2566002186

Comment:
Sure, I was able to do that. Unfortunately, it seems to not give any useful results.

```
❯ gdb -q containerd
Reading symbols from containerd...
Reading symbols from /usr/lib/debug//usr/bin/containerd...
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /usr/lib/debug/usr/bin/containerd.
Use `info auto-load python-scripts [REGEXP]' to list them.
(gdb) run
Starting program: /usr/bin/containerd
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.

Program received signal SIGSEGV, Segmentation fault.
0x0000007ff7fa75a4 in ?? ()
(gdb) bt
#0  0x0000007ff7fa75a4 in ?? ()
#1  0x0000007ff7ffedd0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
``` 

As far as I can assess it, all relevant debug packed were installed beforehand.

```
containerd-dbg-2.0.1_1: configuring ...
containerd-dbg-2.0.1_1: installed successfully.
libseccomp-dbg-2.5.5_1: configuring ...
libseccomp-dbg-2.5.5_1: installed successfully.
musl-dbg-1.1.24_22: configuring ...
musl-dbg-1.1.24_22: installed successfully.
runc-dbg-1.1.13_2: configuring ...
runc-dbg-1.1.13_2: installed successfully.
``` 

Re: Package for containerd on aarch64-musl produces segfault on service startup

[moabeat-berlin]

[-- Attachment #1: Type: text/plain, Size: 1434 bytes --]

New comment by moabeat-berlin on void-packages repository

https://github.com/void-linux/void-packages/issues/53763#issuecomment-2566002186

Comment:
Sure, I was able to do that. Unfortunately, it seems to not give any useful results.

```
❯ gdb -q containerd
Reading symbols from containerd...
Reading symbols from /usr/lib/debug//usr/bin/containerd...
warning: Missing auto-load script at offset 0 in section .debug_gdb_scripts
of file /usr/lib/debug/usr/bin/containerd.
Use `info auto-load python-scripts [REGEXP]' to list them.
(gdb) run
Starting program: /usr/bin/containerd
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.

Program received signal SIGSEGV, Segmentation fault.
0x0000007ff7fa75a4 in ?? ()
(gdb) bt
#0  0x0000007ff7fa75a4 in ?? ()
#1  0x0000007ff7ffedd0 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)
``` 

As far as I can assess it, all relevant debug packages were installed beforehand.

```
containerd-dbg-2.0.1_1: configuring ...
containerd-dbg-2.0.1_1: installed successfully.
libseccomp-dbg-2.5.5_1: configuring ...
libseccomp-dbg-2.5.5_1: installed successfully.
musl-dbg-1.1.24_22: configuring ...
musl-dbg-1.1.24_22: installed successfully.
runc-dbg-1.1.13_2: configuring ...
runc-dbg-1.1.13_2: installed successfully.
``` 

Re: Package for containerd on aarch64-musl produces segfault on service startup

[moabeat-berlin]

[-- Attachment #1: Type: text/plain, Size: 353 bytes --]

New comment by moabeat-berlin on void-packages repository

https://github.com/void-linux/void-packages/issues/53763#issuecomment-2566015565

Comment:
In the meantime, I tried to rebuild the package natively on the RPi4 4GB I discovered the issue on. Suprisingly, this worked and containerd in version 2.0.1 is now running without issues on the machine.

Re: Package for containerd on aarch64-musl produces segfault on service startup

[classabbyamp]

[-- Attachment #1: Type: text/plain, Size: 263 bytes --]

New comment by classabbyamp on void-packages repository

https://github.com/void-linux/void-packages/issues/53763#issuecomment-2566053556

Comment:
I was able to reproduce this in qemu-user, then was able to fix it by adding `-buildmode=pie`. patch incoming soon

Re: [ISSUE] [CLOSED] Package for containerd on aarch64-musl produces segfault on service startup

[classabbyamp]

[-- Attachment #1: Type: text/plain, Size: 2123 bytes --]

Closed issue by moabeat-berlin on void-packages repository

https://github.com/void-linux/void-packages/issues/53763

Description:
### Is this a new report?

Yes

### System Info

Void 6.6.42_1 aarch64-musl

### Package(s) Affected

containerd-2.0.1_1

### Does a report exist for this bug with the project's home (upstream) and/or another distro?

_No response_

### Expected behaviour

The service containerd should start up normally, when added to the `default` runlevel.

### Actual behaviour

The service containerd crashed with a segfault. Log from svlogtail are below.

```
2024-12-30T12:39:25.16653 daemon.notice: Dec 30 12:39:25 docker: time="2024-12-30T13:39:25.166262821+01:00" level=info msg="Starting up"
2024-12-30T12:39:25.16741 daemon.notice: Dec 30 12:39:25 docker: time="2024-12-30T13:39:25.167289121+01:00" level=info msg="containerd not running, starting managed containerd"
2024-12-30T12:39:25.16914 daemon.notice: Dec 30 12:39:25 docker: time="2024-12-30T13:39:25.168702786+01:00" level=info msg="started new containerd process" address=/var/run/docker/containerd/containerd.sock module=libcontainerd pid=1617
2024-12-30T12:39:25.16960 daemon.notice: Dec 30 12:39:25 docker: time="2024-12-30T13:39:25.169396684+01:00" level=error msg="containerd did not exit successfully" error="signal: segmentation fault" module=libcontainerd
2024-12-30T12:39:34.77693 daemon.notice: Dec 30 12:39:34 docker: time="2024-12-30T13:39:34.776333061+01:00" level=info msg="started new containerd process" address=/var/run/docker/containerd/containerd.sock module=libcontainerd pid=1618
2024-12-30T12:39:34.77865 daemon.notice: Dec 30 12:39:34 docker: time="2024-12-30T13:39:34.778053907+01:00" level=error msg="containerd did not exit successfully" error="signal: segmentation fault" module=libcontainerd
2024-12-30T12:39:40.16892 daemon.notice: Dec 30 12:39:40 docker: failed to start containerd: timeout waiting for containerd to start
``` 

Downgrade to version 1.7.17_2 fixed the issue.

### Steps to reproduce

1. Update containerd package to version 2.0.1
2. Reboot
3. containerd crashes as described

Re: Package for containerd on aarch64-musl produces segfault on service startup

[moabeat-berlin]

[-- Attachment #1: Type: text/plain, Size: 279 bytes --]

New comment by moabeat-berlin on void-packages repository

https://github.com/void-linux/void-packages/issues/53763#issuecomment-2566292312

Comment:
I can confirm, the patched packages works flawlessly now. Thank you very much for fixing this so quickly! Very much appreciated!