[ISSUE] linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[ISSUE] linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[rdslw]

[-- Attachment #1: Type: text/plain, Size: 2112 bytes --]

New issue by rdslw on void-packages repository

https://github.com/void-linux/void-packages/issues/47011

Description:
### Is this a new report?

Yes

### System Info

Void x86_musl

### Package(s) Affected

linux5.15-5.15.*

### Does a report exist for this bug with the project's home (upstream) and/or another distro?

no, this is void specific bug.

### Expected behaviour

package linux5.15 is able to install even if /boot is a vfat partition, and chown/chmod commands may return errors, which are ignored by install scripts of linux5 package

### Actual behaviour

If the /boot (where /vmlinuz-XXX and config-XXX and initramfs-XXX are installed) is vfat filesystem (not the ext4/btrfs/others), installation of linux5.15 package fails, due to the inability to set ownership and permissions during xbps install.

What's funny, files ARE copied to /boot, but the install process fails with error after copy. So it is enough to run xpbs install three times in a row (without any fixes) and the third run, completes fine.

Important: this bug is old, it was not introduced recently, but probably been with us for a long time.

Error messages:
```
linux5.15-5.15.131_1: updating to 5.15.137_1 ...
linux5.15-5.15.137_1: unpacking ...
ERROR: linux5.15-5.15.137_1: [unpack] failed to extract file `./boot/vmlinuz-5.15.137_1': Operation not permitted
ERROR: linux5.15-5.15.137_1: [unpack] failed to extract files: Operation not permitted
ERROR: linux5.15-5.15.137_1: [unpack] failed to unpack files from archive: Operation not permitted

linux5.15-5.15.131_1: updating to 5.15.137_1 ...
linux5.15-5.15.137_1: unpacking ...
ERROR: linux5.15-5.15.137_1: [unpack] failed to extract file `./boot/config-5.15.137_1': Operation not permitted
ERROR: linux5.15-5.15.137_1: [unpack] failed to extract files: Operation not permitted
ERROR: linux5.15-5.15.137_1: [unpack] failed to unpack files from archive: Operation not permitted
```

### Steps to reproduce

1. use /boot on vfat partition (for examply to have shared EFI and /boot space).
2. try to upgrade (important) linux5.15 package (newer kernel)

Re: linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[Duncaen]

[-- Attachment #1: Type: text/plain, Size: 253 bytes --]

New comment by Duncaen on void-packages repository

https://github.com/void-linux/void-packages/issues/47011#issuecomment-1789436659

Comment:
I don't think that is the issue, /boot of vfat for efi is a pretty standard setup. Can you maybe strace this?

Re: linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[Duncaen]

[-- Attachment #1: Type: text/plain, Size: 253 bytes --]

New comment by Duncaen on void-packages repository

https://github.com/void-linux/void-packages/issues/47011#issuecomment-1789436659

Comment:
I don't think that is the issue, /boot on vfat for efi is a pretty standard setup. Can you maybe strace this?

Re: linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[rdslw]

[-- Attachment #1: Type: text/plain, Size: 952 bytes --]

New comment by rdslw on void-packages repository

https://github.com/void-linux/void-packages/issues/47011#issuecomment-1789462848

Comment:
@Duncaen sure, I can. Here I was using 'xbps-install -Su' command, would you like to just strace it, and try to find calls around error/post log?

Currently I'm only able to downgrade linux kernel package, as I have the newest one installed today. Can you guide me how to do it, as simple ```xbps-install --force --debug /var/cache/xbps/linux5.15-5.15.131_1.x86_64-musl.xbps``` results  in error: _Package '/var/cache/xbps/linux5.15-5.15.131_1.x86_64-musl.xbps' not found in repository pool_

P.S. My guess about chmod/chown is because after each command/error, the offending file from error message *is* unpacked to /boot, but the whole command aborts while the next (exactly the same xbps-install -Su) invocation moves forward and fails on another file, then third call succeeds. I might be wrong though.

Re: linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[classabbyamp]

[-- Attachment #1: Type: text/plain, Size: 195 bytes --]

New comment by classabbyamp on void-packages repository

https://github.com/void-linux/void-packages/issues/47011#issuecomment-1789469505

Comment:
don't use the full path, just the package name

Re: linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[rdslw]

[-- Attachment #1: Type: text/plain, Size: 570 bytes --]

New comment by rdslw on void-packages repository

https://github.com/void-linux/void-packages/issues/47011#issuecomment-1789491251

Comment:
@classabbyamp appreciate you want to help. Unfortunately, it's not the path.
For reference the proper procedure requires two separate commands:
```
xbps-rindex -a /var/cache/xbps/pkg-1.0_1.xbps
xbps-install -R /var/cache/xbps/ -f pkg-1.0_1
```
as explained here: https://docs.voidlinux.org/xbps/advanced-usage.html?highlight=xdowng#via-xdowngrade

 **strace results for this issue follows shortly in my next comment.**

Re: linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[rdslw]

[-- Attachment #1: Type: text/plain, Size: 2999 bytes --]

New comment by rdslw on void-packages repository

https://github.com/void-linux/void-packages/issues/47011#issuecomment-1789495385

Comment:
@Duncaen as I expected, this is due to the chown, strace is:
```
[...]
open("boot/vmlinuz-5.15.131_1", O_WRONLY|O_CREAT|O_EXCL|O_CLOEXEC, 0644) = 7
fcntl(7, F_SETFD, FD_CLOEXEC)           = 0
fcntl(7, F_GETFD)                       = 0x1 (flags FD_CLOEXEC)
write(7, "MZ\352\7\0\300\7\214\310\216\330\216\300\216\3201\344\373\374\276@\0\254 \300t\t\264\16\273\7\0"..., 13312) = 13312
read(6, "ze\240e}M\205\363\203\362\304\214`f\261\306M=\354\340J\273\272Y\247E\22\343+\323+\212"..., 4096) = 4096

[... lot of writing to FD no 7]
write(7, "\20\2315\fQ\23\f\306\321\226!Z\331*D\333\333\206h}\273\20\235\302\304\366!\272\211\316\35B"..., 102176) = 102176
fchown(7, 0, 0)                         = -1 EPERM (Operation not permitted)
lchown("boot/vmlinuz-5.15.131_1", 0, 0) = -1 EPERM (Operation not permitted)
utimensat(7, NULL, [{tv_sec=1698864954, tv_nsec=0} /* 2023-11-01T19:55:54+0100 */, {tv_sec=1694612722, tv_nsec=0} /* 2023-09-13T15:45:22+0200 */], 0) = 0
close(7)                                = 0
writev(2, [{iov_base="", iov_len=0}, {iov_base="ERROR: ", iov_len=7}], 2ERROR: ) = 7
writev(2, [{iov_base="", iov_len=0}, {iov_base="linux5.15-5.15.131_1: [unpack] f"..., iov_len=106}], 2linux5.15-5.15.131_1: [unpack] failed to extract file `./boot/vmlinuz-5.15.131_1': Operation not permitted) = 106
writev(2, [{iov_base="\n", iov_len=1}, {iov_base=NULL, iov_len=0}], 2
) = 1
socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 7
connect(7, {sa_family=AF_UNIX, sun_path="/dev/log"}, 12) = 0
sendto(7, "<11>Nov  1 18:56:00 xbps-install"..., 141, 0, NULL, 0) = 141
writev(2, [{iov_base="", iov_len=0}, {iov_base="ERROR: ", iov_len=7}], 2ERROR: ) = 7
writev(2, [{iov_base="linux5.15-5.15.131_1: [unpack] f"..., iov_len=80}, {iov_base=NULL, iov_len=0}], 2linux5.15-5.15.131_1: [unpack] failed to extract files: Operation not permitted
) = 80
sendto(7, "<11>Nov  1 18:56:00 xbps-install"..., 114, 0, NULL, 0) = 114
writev(2, [{iov_base="", iov_len=0}, {iov_base="ERROR: ", iov_len=7}], 2ERROR: ) = 7
writev(2, [{iov_base="", iov_len=0}, {iov_base="linux5.15-5.15.131_1: [unpack] f"..., iov_len=91}], 2linux5.15-5.15.131_1: [unpack] failed to unpack files from archive: Operation not permitted) = 91
writev(2, [{iov_base="\n", iov_len=1}, {iov_base=NULL, iov_len=0}], 2
) = 1
sendto(7, "<11>Nov  1 18:56:00 xbps-install"..., 126, 0, NULL, 0) = 126
close(6)                                = 0
munmap(0x79b456975000, 4591616)         = 0
umask(022)                              = 022
writev(2, [{iov_base="", iov_len=0}, {iov_base="Transaction failed! see above fo"..., iov_len=42}], 2Transaction failed! see above for errors.
) = 42
fcntl(3, F_SETLK, {l_type=F_UNLCK, l_whence=SEEK_CUR, l_start=0, l_len=0}) = 0
close(3)                                = 0
exit_group(1)                           = ?
+++ exited with 1 +++
```

Re: linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[Duncaen]

[-- Attachment #1: Type: text/plain, Size: 422 bytes --]

New comment by Duncaen on void-packages repository

https://github.com/void-linux/void-packages/issues/47011#issuecomment-1789504692

Comment:
I wonder if this is either because of a specific kernel version or different mount flags than everyone else uses. This works fine for the raspberry pi images which have vfat /boot by default. And for other x86_64, like the one I'm currently writing from where it works fine too.

Re: linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[Duncaen]

[-- Attachment #1: Type: text/plain, Size: 430 bytes --]

New comment by Duncaen on void-packages repository

https://github.com/void-linux/void-packages/issues/47011#issuecomment-1789504692

Comment:
I wonder if this is either because of a specific kernel version or different mount flags than everyone else uses. This works fine for the raspberry pi images which have vfat /boot by default. And for other x86_64 systems, like the one I'm currently writing from where it works fine too.

Re: linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[Duncaen]

[-- Attachment #1: Type: text/plain, Size: 424 bytes --]

New comment by Duncaen on void-packages repository

https://github.com/void-linux/void-packages/issues/47011#issuecomment-1789526946

Comment:
Correctly handling this in xbps would require quiet a bit of rework since currently libarchive extracts and handles the mode, and file times. xbps would have to manually chown the files and then I guess ignore chown errors on vfat, so that EPERM from open/write don't get ignored.

Re: linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[Duncaen]

[-- Attachment #1: Type: text/plain, Size: 424 bytes --]

New comment by Duncaen on void-packages repository

https://github.com/void-linux/void-packages/issues/47011#issuecomment-1789526946

Comment:
Correctly handling this in xbps would require quite a bit of rework since currently libarchive extracts and handles the mode, and file times. xbps would have to manually chown the files and then I guess ignore chown errors on vfat, so that EPERM from open/write don't get ignored.

Re: linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[classabbyamp]

[-- Attachment #1: Type: text/plain, Size: 199 bytes --]

New comment by classabbyamp on void-packages repository

https://github.com/void-linux/void-packages/issues/47011#issuecomment-1789527680

Comment:
what's the output of `cat /etc/fstab` and `mount`?

Re: linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[Duncaen]

[-- Attachment #1: Type: text/plain, Size: 421 bytes --]

New comment by Duncaen on void-packages repository

https://github.com/void-linux/void-packages/issues/47011#issuecomment-1789526946

Comment:
Correctly handling this in xbps would require quite a bit of rework since currently libarchive extracts, handles the mode, and file times. xbps would have to manually chown the files and then I guess ignore chown errors on vfat, so that EPERM from open/write don't get ignored.

Re: linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[rdslw]

[-- Attachment #1: Type: text/plain, Size: 888 bytes --]

New comment by rdslw on void-packages repository

https://github.com/void-linux/void-packages/issues/47011#issuecomment-1791057531

Comment:
@Duncaen @classabbyamp I do use different /boot mount flags, so this might be the problem. Let me know any tests you want to do it.
```
$> grep boot /etc/fstab                                                                                                               ~
LABEL=ESP           /boot         vfat          rw,relatime,fmask=0117,dmask=0007,gid=10,iocharset=iso8859-1,codepage=437,shortname=mixed,errors=remount-ro    0 2
$> mount |grep boot                                                                                                                   ~
/dev/nvme0n1p1 on /boot type vfat (rw,relatime,gid=10,fmask=0117,dmask=0007,allow_utime=0020,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro)
```

Re: linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[Duncaen]

[-- Attachment #1: Type: text/plain, Size: 448 bytes --]

New comment by Duncaen on void-packages repository

https://github.com/void-linux/void-packages/issues/47011#issuecomment-1791694333

Comment:
Weird mine are effectively basically the same, except `gid`. The `quiet` option should be what would make attribute changes return 0 instead of errors, but I don't even have that set here so not sure why its returning 0 for me anyways. Maybe I'm missing some other place or reason why errors are ignored.

Re: linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[Duncaen]

[-- Attachment #1: Type: text/plain, Size: 438 bytes --]

New comment by Duncaen on void-packages repository

https://github.com/void-linux/void-packages/issues/47011#issuecomment-1791694333

Comment:
Weird mine are effectively the same, except `gid`. The `quiet` option should be what would make attribute changes return 0 instead of errors, but I don't even have that set here so not sure why its returning 0 for me anyways. Maybe I'm missing some other place or reason why errors are ignored.

Re: linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[Duncaen]

[-- Attachment #1: Type: text/plain, Size: 522 bytes --]

New comment by Duncaen on void-packages repository

https://github.com/void-linux/void-packages/issues/47011#issuecomment-1791694333

Comment:
Weird mine are effectively the same, except `gid`. The `quiet` option should be what would make attribute changes return 0 instead of errors, but I don't even have that set here so not sure why its returning 0 for me anyways. Maybe I'm missing some other place or reason why errors are ignored.

Maybe its the `gid`, but `xbps-install` is running as root so it shouldn't care.

Re: linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[Duncaen]

[-- Attachment #1: Type: text/plain, Size: 657 bytes --]

New comment by Duncaen on void-packages repository

https://github.com/void-linux/void-packages/issues/47011#issuecomment-1791694333

Comment:
Weird mine are effectively the same, except `gid`. The `quiet` option should be what would make attribute changes return 0 instead of errors, but I don't even have that set here so not sure why its returning 0 for me anyways. Maybe I'm missing some other place or reason why errors are ignored.

Maybe its the `gid`, but `xbps-install` is running as root so it shouldn't care. Oh wait its chown that failing and not chmod, so it does make sense that this fails because you set the gid to 10 instead of root (0).

Re: linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[rdslw]

[-- Attachment #1: Type: text/plain, Size: 419 bytes --]

New comment by rdslw on void-packages repository

https://github.com/void-linux/void-packages/issues/47011#issuecomment-1792017730

Comment:
Doing tests, appreciate your help @Duncaen 
started with `quiet` while leaving `gid=10` in place. Manual upgrade of package did NOT result in error. Will wait to be sure, for next official update of linux5.15 and normal `xbps-install -Su` and report here if `quiet` solved it.

Re: [ISSUE] [CLOSED] linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 2115 bytes --]

Closed issue by rdslw on void-packages repository

https://github.com/void-linux/void-packages/issues/47011

Description:
### Is this a new report?

Yes

### System Info

Void x86_musl

### Package(s) Affected

linux5.15-5.15.*

### Does a report exist for this bug with the project's home (upstream) and/or another distro?

no, this is void specific bug.

### Expected behaviour

package linux5.15 is able to install even if /boot is a vfat partition, and chown/chmod commands may return errors, which are ignored by install scripts of linux5 package

### Actual behaviour

If the /boot (where /vmlinuz-XXX and config-XXX and initramfs-XXX are installed) is vfat filesystem (not the ext4/btrfs/others), installation of linux5.15 package fails, due to the inability to set ownership and permissions during xbps install.

What's funny, files ARE copied to /boot, but the install process fails with error after copy. So it is enough to run xpbs install three times in a row (without any fixes) and the third run, completes fine.

Important: this bug is old, it was not introduced recently, but probably been with us for a long time.

Error messages:
```
linux5.15-5.15.131_1: updating to 5.15.137_1 ...
linux5.15-5.15.137_1: unpacking ...
ERROR: linux5.15-5.15.137_1: [unpack] failed to extract file `./boot/vmlinuz-5.15.137_1': Operation not permitted
ERROR: linux5.15-5.15.137_1: [unpack] failed to extract files: Operation not permitted
ERROR: linux5.15-5.15.137_1: [unpack] failed to unpack files from archive: Operation not permitted

linux5.15-5.15.131_1: updating to 5.15.137_1 ...
linux5.15-5.15.137_1: unpacking ...
ERROR: linux5.15-5.15.137_1: [unpack] failed to extract file `./boot/config-5.15.137_1': Operation not permitted
ERROR: linux5.15-5.15.137_1: [unpack] failed to extract files: Operation not permitted
ERROR: linux5.15-5.15.137_1: [unpack] failed to unpack files from archive: Operation not permitted
```

### Steps to reproduce

1. use /boot on vfat partition (for examply to have shared EFI and /boot space).
2. try to upgrade (important) linux5.15 package (newer kernel)

Re: linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 339 bytes --]

New comment by ahesford on void-packages repository

https://github.com/void-linux/void-packages/issues/47011#issuecomment-1837297363

Comment:
Seems like this is caused by specific configuration options and the errors are not unexpected. If you have convincing evidence that this issue is not caused by mount options, please let us know.

Re: linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[rdslw]

[-- Attachment #1: Type: text/plain, Size: 471 bytes --]

New comment by rdslw on void-packages repository

https://github.com/void-linux/void-packages/issues/47011#issuecomment-1890411799

Comment:
I can confirm that adding quiet to the mount options of /boot resolved problem.
It is still bug in xbps, as it should correctly be able to write linux5.15 files on the vfat partition (this is widely used scenario) mounted with gid option, nevertheless, adding quiet is easier (especially if user previously added gid himself :) 

Re: linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[Duncaen]

[-- Attachment #1: Type: text/plain, Size: 420 bytes --]

New comment by Duncaen on void-packages repository

https://github.com/void-linux/void-packages/issues/47011#issuecomment-1890419836

Comment:
The package contains files that are owned by root, xbps tries to set the owner/group. I don't think xbps should ignore failures of setting the expected owner, even making this filesystem specific seems wrong. What is the reason to set `gid=` on `/boot` and what user is that? 

Re: linux5.15 package install scripts fails if /boot is mounted on vfat fstype

[rdslw]

[-- Attachment #1: Type: text/plain, Size: 956 bytes --]

New comment by rdslw on void-packages repository

https://github.com/void-linux/void-packages/issues/47011#issuecomment-1890431965

Comment:
Thanks for answering it! 

The actual reason is to have ability to writeable (some) files on /boot vfat partition, while not allowing all users to write them. Here gid points to some wheels/adm gid which only some users belong. Due to the limitations of vfat fs, it's widely adopted to use some form of gid/uid/fmask/dmask on such fs, to have simple access control.

(Just to make it clear: xbps is being run under root) 

But moving back to the error:
I think "xbps should ignore failures" is not true, as there is no ownership on vfat filesystem type (used on /boot) therefore this is not a failure, but xbps bug (not understanding that sometimes this operation (permissions/ownership) is not needed)

I consider FS settings machine, not void specific. One can ask, why using vfat on this filesystem? ;)