[ISSUE] incus images based on Ubuntu fail to start

[ISSUE] incus images based on Ubuntu fail to start

[sbromberger]

[-- Attachment #1: Type: text/plain, Size: 1396 bytes --]

New issue by sbromberger on void-packages repository

https://github.com/void-linux/void-packages/issues/48356

Description:
### Is this a new report?

Yes

### System Info

Void 6.6.11_1 x86_64 AuthenticAMD uptodate rFF

### Package(s) Affected

incus-0.4.0_1

### Does a report exist for this bug with the project's home (upstream) and/or another distro?

_No response_

### Expected behaviour

Migrated from a home-built version of incus to the new package (yay!) - but while void images start up with no problem, ubuntu ones (including new images) fail to start. I expected the ubuntu images to start up normally.

### Actual behaviour

Ubuntu images (based on `lunar`; possibly others) fail to start. There is no diagnostic information printed either with `-v`.

`--console` produces the following error:

```
[root@elemental beryllium]# incus start testub --console
To detach from the console, press: <ctrl>+a q
Error: Failed running forkconsole: "container is not running: \"testub\""
                                                                         Error: write /dev/pts/ptmx: file already closed
```

`-debug` produces the following output: https://gist.github.com/sbromberger/e6adcff70ab7e6fd0410055cfde09e69



### Steps to reproduce

1. `incus create images:ubuntu/lunar testub`
2. `incus start testub`
3. `incus ls testub`
4. Observe that status is `STOPPED`

Re: incus images based on Ubuntu fail to start

[sbromberger]

[-- Attachment #1: Type: text/plain, Size: 1712 bytes --]

New comment by sbromberger on void-packages repository

https://github.com/void-linux/void-packages/issues/48356#issuecomment-1908655220

Comment:
Ah, I found some logs in /var/log/incus/testub:

Excerpt from `lxc.log`:
```
lxc testub 20240124172933.916 INFO     conf - ../src/lxc/conf.c:setup_personality:1917 - Set personality to "0lx0"
lxc testub 20240124172933.917 NOTICE   conf - ../src/lxc/conf.c:lxc_setup:4469 - The container "testub" is set up
lxc testub 20240124172933.918 NOTICE   start - ../src/lxc/start.c:start:2194 - Exec'ing "/sbin/init"
lxc testub 20240124172933.921 NOTICE   start - ../src/lxc/start.c:post_start:2205 - Started "/sbin/init" with pid "15889"
lxc testub 20240124172933.921 NOTICE   start - ../src/lxc/start.c:signal_handler:446 - Received 17 from pid 15890 instead of container init 15889
lxc testub 20240124172933.978 INFO     error - ../src/lxc/error.c:lxc_error_set_and_log:31 - Child <15889> ended on error (255)
lxc testub 20240124172933.995 INFO     conf - ../src/lxc/conf.c:run_script_argv:340 - Executing script "/usr/libexec/incus/incusd callhook /var/lib/incus "default" "testub" stopns" for container "testub"
lxc testub 20240124172934.129 INFO     conf - ../src/lxc/conf.c:lxc_map_ids:3603 - Caller maps host root. Writing mapping directly
lxc testub 20240124172934.130 NOTICE   utils - ../src/lxc/utils.c:lxc_drop_groups:1368 - Dropped supplimentary groups
lxc testub 20240124172934.147 INFO     conf - ../src/lxc/conf.c:run_script_argv:340 - Executing script "/usr/libexec/incus/incusd callhook /var/lib/incus "default" "testub" stop" for container "testub"
```

Full log here: https://gist.github.com/sbromberger/b05b018f3df1192a0a6bada49d8be1e8

Re: incus images based on Ubuntu fail to start

[sbromberger]

[-- Attachment #1: Type: text/plain, Size: 357 bytes --]

New comment by sbromberger on void-packages repository

https://github.com/void-linux/void-packages/issues/48356#issuecomment-1908674180

Comment:
I found these logs in `console.log` on another image as well:

```
Failed to mount cgroup at /sys/fs/cgroup/systemd: Operation not permitted
[!!!!!!] Failed to mount API filesystems.
Exiting PID 1...
```

Re: incus images based on Ubuntu fail to start

[CameronNemo]

[-- Attachment #1: Type: text/plain, Size: 864 bytes --]

New comment by CameronNemo on void-packages repository

https://github.com/void-linux/void-packages/issues/48356#issuecomment-1908800084

Comment:
As mentioned in the PR, does this happen with the unified cgroup mode i.e. a cgroup2-only setup?

The latest systemd release actually deprecated cgroup v1 support, and it will be removed in a future systemd release:

>    * We intend to remove cgroup v1 support from a systemd release after  
>      the end of 2023. If you run services that make explicit use of  
>      cgroup v1 features (i.e. the "legacy hierarchy" with separate  
>      hierarchies for each controller), please implement compatibility with  
>      cgroup v2 (i.e. the "unified hierarchy") sooner rather than later.  
>      Most of Linux userspace has been ported over already.

https://github.com/systemd/systemd/releases/tag/v255

Re: incus images based on Ubuntu fail to start

[BikyAlex]

[-- Attachment #1: Type: text/plain, Size: 768 bytes --]

New comment by BikyAlex on void-packages repository

https://github.com/void-linux/void-packages/issues/48356#issuecomment-1967590039

Comment:
Confirm this happens to other distros. I tested Debian and NixOS, both fail to start with the same operation not permitted / failed to mount API FS error.

I tested Incus on a NixOS host and there systemd distros launch fine. Not sure why this is an issue on Void alone.

Anyway, edited rc.conf and set "CGROUP_MODE=unified" and rebooted. Debian and NixOS start now. Not sure what's the point of keeping cgroup in hybrid by default in Void, if any other things depend on that.

Well, I'm happy I can use Incus with this workaround. I wanted to be able to run stuff on other distros in case there's no package in Void.

Re: incus images based on Ubuntu fail to start

[CameronNemo]

[-- Attachment #1: Type: text/plain, Size: 396 bytes --]

New comment by CameronNemo on void-packages repository

https://github.com/void-linux/void-packages/issues/48356#issuecomment-1967885164

Comment:
@BikyAlex I am not entirely sure how to escalate the "let's move CGROUP_MODE to unified by default" pitch, but I definitely don't think anything really depends on the hybrid cgroups anymore. Except for perhaps older device kernels like rpi kernels?

Re: incus images based on Ubuntu fail to start

[classabbyamp]

[-- Attachment #1: Type: text/plain, Size: 222 bytes --]

New comment by classabbyamp on void-packages repository

https://github.com/void-linux/void-packages/issues/48356#issuecomment-1967892009

Comment:
rpi kernels are on 6.1, not sure what the kconfig sets wrt cgroups though

Re: incus images based on Ubuntu fail to start

[CameronNemo]

[-- Attachment #1: Type: text/plain, Size: 220 bytes --]

New comment by CameronNemo on void-packages repository

https://github.com/void-linux/void-packages/issues/48356#issuecomment-1968052395

Comment:
@classabbyamp 

void-linux/void-runit#103 would resolve this issue IMO