[ISSUE] libvirt-7.5.0_1: [Regression] Unable to start any virtual machines created prior to the update

[ISSUE] libvirt-7.5.0_1: [Regression] Unable to start any virtual machines created prior to the update

[noarchwastaken]

[-- Attachment #1: Type: text/plain, Size: 971 bytes --]

New issue by noarchwastaken on void-packages repository

https://github.com/void-linux/void-packages/issues/31904

Description:
<!-- Don't request update of package. We have a script for that. https://alpha.de.repo.voidlinux.org/void-updates/void-updates.txt . However, a quality pull request may help. -->
### System

* xuname:  
  Void 5.12.14_1 x86_64 GenuineIntel uptodate rFFFFFFFFFF
* package:  
  libvirt-7.5.0_1
  Was working fine in libvirt-7.4.0_1

### Expected behavior

`virsh start <domain>` should start the corresponding virtual machine.

### Actual behavior

```
$ doas virsh start voidlinux
error: Failed to start domain 'voidlinux'
error: internal error: cannot load AppArmor profile 'libvirt-aa58ea57-f6d7-48b1-b727-de97f4c508af'
```

### Steps to reproduce the behavior

1. Enable AppArmor
2. Create a VM using libvirt<=7.4.0_1
3. Update libvirt to 7.5.0_1, reboot to reload apparmor profiles
4. Try to launch the VM again

Re: libvirt-7.5.0_1: [Regression] Unable to create or start VMs

[noarchwastaken]

[-- Attachment #1: Type: text/plain, Size: 176 bytes --]

New comment by noarchwastaken on void-packages repository

https://github.com/void-linux/void-packages/issues/31904#issuecomment-884671346

Comment:
Some more info on the bug:

Re: libvirt-7.5.0_1: [Regression] Unable to create or start VMs

[noarchwastaken]

[-- Attachment #1: Type: text/plain, Size: 176 bytes --]

New comment by noarchwastaken on void-packages repository

https://github.com/void-linux/void-packages/issues/31904#issuecomment-884671346

Comment:
Some more info on the bug:

Re: libvirt-7.5.0_1: [Regression] Unable to create or start VMs

[noarchwastaken]

[-- Attachment #1: Type: text/plain, Size: 786 bytes --]

New comment by noarchwastaken on void-packages repository

https://github.com/void-linux/void-packages/issues/31904#issuecomment-884672975

Comment:
Some more info on the bug:

Everything starts to work again when I delete `/etc/apparmor.d/libvirt`. This directory doesn't exist in `libvirt-7.4.0_1`.

With the directory there, every time I try to create a VM, apparmor denies this:

```
2021-07-22T05:50:57.18772 kern.notice: [  697.561573] audit: type=1400 audit(1626933057.186:725): apparmor="DENIED" operation="exec" profile="virt-aa-helper" name="/usr/b
in/apparmor_parser" pid=4069 comm="virt-aa-helper" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
```

And that's why I'm able to create and run new VMs with `usr.lib.libvirt.virt-aa-helper` set to complain mode.

Re: libvirt-7.5.0_1: [Regression] Unable to create or start VMs

[noarchwastaken]

[-- Attachment #1: Type: text/plain, Size: 766 bytes --]

New comment by noarchwastaken on void-packages repository

https://github.com/void-linux/void-packages/issues/31904#issuecomment-884672975

Comment:
Some more info on the bug:

A workaround is to delete `/etc/apparmor.d/libvirt`. This directory doesn't exist in `libvirt-7.4.0_1`.

With the directory there, every time I try to create a VM, apparmor denies this:

```
2021-07-22T05:50:57.18772 kern.notice: [  697.561573] audit: type=1400 audit(1626933057.186:725): apparmor="DENIED" operation="exec" profile="virt-aa-helper" name="/usr/b
in/apparmor_parser" pid=4069 comm="virt-aa-helper" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
```

And that's why I'm able to create and run new VMs with `usr.lib.libvirt.virt-aa-helper` set to complain mode.

Re: libvirt-7.5.0_1: [Regression] Unable to create or start VMs

[noarchwastaken]

[-- Attachment #1: Type: text/plain, Size: 801 bytes --]

New comment by noarchwastaken on void-packages repository

https://github.com/void-linux/void-packages/issues/31904#issuecomment-884672975

Comment:
Some more info on the bug:

A workaround is to delete `/etc/apparmor.d/libvirt`, which makes everything work again. This directory doesn't exist in `libvirt-7.4.0_1`.

With the directory there, every time I try to create a VM, apparmor denies this:

```
2021-07-22T05:50:57.18772 kern.notice: [  697.561573] audit: type=1400 audit(1626933057.186:725): apparmor="DENIED" operation="exec" profile="virt-aa-helper" name="/usr/b
in/apparmor_parser" pid=4069 comm="virt-aa-helper" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
```

And that's why I'm able to create and run new VMs with `usr.lib.libvirt.virt-aa-helper` set to complain mode.

Re: libvirt-7.5.0_1: [Regression] Unable to create or start VMs

[paper42]

[-- Attachment #1: Type: text/plain, Size: 527 bytes --]

New comment by paper42 on void-packages repository

https://github.com/void-linux/void-packages/issues/31904#issuecomment-884789341

Comment:
@noarchwastaken Can you try adjusting this line in usr.lib.libvirt.virt-aa-helper?
```
/{usr/,}sbin/apparmor_parser Ux
```
to
```
/{usr/,}{s,}bin/apparmor_parser Ux
```

apparmor profiles from upstream projects are often broken because each distribution has different paths, but if fixes your issue, you could upstream it and submit a PR with a patch for the libvirt package.

Re: libvirt-7.5.0_1: [Regression] Unable to create or start VMs

[noarchwastaken]

[-- Attachment #1: Type: text/plain, Size: 884 bytes --]

New comment by noarchwastaken on void-packages repository

https://github.com/void-linux/void-packages/issues/31904#issuecomment-886050757

Comment:

> @noarchwastaken Can you try adjusting this line in usr.lib.libvirt.virt-aa-helper?
> 
> ```
> /{usr/,}sbin/apparmor_parser Ux
> ```
> 
> to
> 
> ```
> /{usr/,}{s,}bin/apparmor_parser Ux
> ```
> 
> apparmor profiles from upstream projects are often broken because each distribution has different paths, but if fixes your issue, you could upstream it and submit a PR with a patch for the libvirt package.

With the change, the `apparmor_parser` warning went away, but I'm still unable to create new UEFI-based VMs... The error is the same.

This time I can't see any kernel log popping up.

Can anyone replicate this issue? I can't strictly control the variables because I'm testing it on my daily driver laptop.

Re: libvirt-7.5.0_1: [Regression] Unable to create or start VMs

[noarchwastaken]

[-- Attachment #1: Type: text/plain, Size: 216 bytes --]

New comment by noarchwastaken on void-packages repository

https://github.com/void-linux/void-packages/issues/31904#issuecomment-886051070

Comment:
Note, the last change allows me to create new BIOS-based VM again.

Re: libvirt-7.5.0_1: [Regression] Unable to create or start VMs

[FollieHiyuki]

[-- Attachment #1: Type: text/plain, Size: 679 bytes --]

New comment by FollieHiyuki on void-packages repository

https://github.com/void-linux/void-packages/issues/31904#issuecomment-886092446

Comment:
> @noarchwastaken Can you try adjusting this line in usr.lib.libvirt.virt-aa-helper?
> 
> ```
> /{usr/,}sbin/apparmor_parser Ux
> ```
> 
> to
> 
> ```
> /{usr/,}{s,}bin/apparmor_parser Ux
> ```
> 
> apparmor profiles from upstream projects are often broken because each distribution has different paths, but if fixes your issue, you could upstream it and submit a PR with a patch for the libvirt package.

Can confirm. After applying this change, I can create BIOS VMs. But UEFI VMs are still stuck at the same error.

Re: libvirt-7.5.0_1: [Regression] Unable to create or start VMs

[sernkut]

[-- Attachment #1: Type: text/plain, Size: 832 bytes --]

New comment by sernkut on void-packages repository

https://github.com/void-linux/void-packages/issues/31904#issuecomment-901483813

Comment:
This is caused by `virt-aa-manager` as it's trying to automatically generate the AppArmor profile on VM statup.  
`virt-aa-manager` disallows some paths in generated AppArmor profiles wich can be found here [`src/security/virt-aa-helper.c:454-490:valid_path()`](https://github.com/libvirt/libvirt/blob/master/src/security/virt-aa-helper.c#L454-L490)  
  
I fixed this by looking at `/usr/share/qemu/firmware/60-edk2-x86_64.json` and then copied the executable and nvram-template files to `/usr/share/ovmf` and finally created a copy of the json file with a new name eg. `60-edk2-x86_64-custom.json` and updated the paths in this new file.
  
Also i'm sorry for creating a extra issue.

Re: libvirt-7.5.0_1: [Regression] Unable to create or start VMs

[sernkut]

[-- Attachment #1: Type: text/plain, Size: 977 bytes --]

New comment by sernkut on void-packages repository

https://github.com/void-linux/void-packages/issues/31904#issuecomment-901483813

Comment:
This is caused by `virt-aa-manager` as it's trying to automatically generate the AppArmor profile on VM startup.  
`virt-aa-manager` disallows some paths in generated AppArmor profiles which can be found here [`src/security/virt-aa-helper.c:454-490:valid_path()`](https://github.com/libvirt/libvirt/blob/master/src/security/virt-aa-helper.c#L454-L490)  

I fixed this for the `edk2-x86_64-code.fd` firmware image  by copying the executable and nvram-template files referenced in `/usr/share/qemu/firmware/60-edk2-x86_64.json` to `/usr/share/ovmf`.  
I then made a copy of `60-edk2-x86_64.json` with some other name like: `60-edk2-x86_64-custom.json` and updated the file paths in this copied file to reference the ones in `/usr/share/ovmf`.  
  
Also i am sorry for creating a extra issue.  
  
Edit: Make soulution more clear

Re: libvirt-7.5.0_1: [Regression] Unable to create or start VMs

[noarchwastaken]

[-- Attachment #1: Type: text/plain, Size: 839 bytes --]

New comment by noarchwastaken on void-packages repository

https://github.com/void-linux/void-packages/issues/31904#issuecomment-1091754469

Comment:
In #32562 (the now) ghost mentioned we could also patch `virt-aa-helper`, and I prefer it since:

First, looking at `restricted-rw[]` it's all publicly accessible files with no confidential information. /usr/share/qemu/ also doesn't contain confidential files, so even in the case of a breakout that will be stopped by apparmor, the risk is managable.

Second, the moving firmware to /usr/share/ovmf method breaks existing VM configurations (except maybe we link the firmwares back? But existing VMs still suffer from the same problem, and it requires manual intervention.)

Last, there are several closed PRs attempting to add `ovmf` so far, and I don't think we need them anymore?

Re: libvirt-7.5.0_1: [Regression] Unable to create or start VMs

[github-actions]

[-- Attachment #1: Type: text/plain, Size: 293 bytes --]

New comment by github-actions[bot] on void-packages repository

https://github.com/void-linux/void-packages/issues/31904#issuecomment-1176964062

Comment:
Issues become stale 90 days after last activity and are closed 14 days after that.  If this issue is still relevant bump it or assign it.

Re: libvirt-7.5.0_1: [Regression] Unable to create or start VMs

[noarchwastaken]

[-- Attachment #1: Type: text/plain, Size: 657 bytes --]

New comment by noarchwastaken on void-packages repository

https://github.com/void-linux/void-packages/issues/31904#issuecomment-1178348136

Comment:
Bump. I still have to manually workaround it with every libvirt update.

On July 7, 2022 2:14:04 AM UTC, "github-actions[bot]" ***@***.***> wrote:
>Issues become stale 90 days after last activity and are closed 14 days after that.  If this issue is still relevant bump it or assign it.
>
>-- 
>Reply to this email directly or view it on GitHub:
>https://github.com/void-linux/void-packages/issues/31904#issuecomment-1176964062
>You are receiving this because you were mentioned.
>
>Message ID: ***@***.***>