[musl] unlink on NFS volume fails silently

[musl] unlink on NFS volume fails silently

[Stephen Von Takach]

[-- Attachment #1: Type: text/plain, Size: 903 bytes --]

Hi,

We recently had to move a service from being built on alpine linux to
debian linux as we were getting silent failures when deleting a directory
with many files on an NFS volume. Basically this call to unlink was not
raising an error if the file failed to delete
https://github.com/crystal-lang/crystal/blob/master/src/crystal/system/unix/file.cr#L129

We replicated the issue in an alpine container with rm -rf
/nfs_mount/git_repo_to_delete and it also failed to successfully delete all
the files, it did raise an error though (I assume it checked the file was
removed before continuing) not entirely sure.

Both these operations succeed with glibc when using debian.
Looks a bit like this issue:
https://gitlab.alpinelinux.org/alpine/aports/-/issues/10960


Stephen von Takach Dukai

Engineering Lead

PlaceOS

Australia, Hong Kong, London, New York

p: +61 408 419 954

e: steve@place.technology

[-- Attachment #2: Type: text/html, Size: 4184 bytes --]

Re: [musl] unlink on NFS volume fails silently

[Stephen Von Takach]

[-- Attachment #1: Type: text/plain, Size: 2014 bytes --]

Yes we traced this.
The libc unlink function on musl returned 0 for a filename on an NFS mount
that wasn't deleted.
https://www.gnu.org/software/libc/manual/html_node/Deleting-Files.html

The same call to unlink on glibc returned 0 and actually removed the file.

The issue occurs when there is a high volume of files being removed


Stephen von Takach Dukai

Engineering Lead

PlaceOS

Australia, Hong Kong, London, New York

p: +61 408 419 954

e: steve@place.technology


On Tue, 8 Jul 2025 at 11:08, Rich Felker <dalias@libc.org> wrote:

> On Sun, Jul 06, 2025 at 04:25:31PM +1000, Stephen Von Takach wrote:
> > Hi,
> >
> > We recently had to move a service from being built on alpine linux to
> > debian linux as we were getting silent failures when deleting a directory
> > with many files on an NFS volume. Basically this call to unlink was not
> > raising an error if the file failed to delete
> >
> https://github.com/crystal-lang/crystal/blob/master/src/crystal/system/unix/file.cr#L129
> >
> > We replicated the issue in an alpine container with rm -rf
> > /nfs_mount/git_repo_to_delete and it also failed to successfully delete
> all
> > the files, it did raise an error though (I assume it checked the file was
> > removed before continuing) not entirely sure.
> >
> > Both these operations succeed with glibc when using debian.
> > Looks a bit like this issue:
> > https://gitlab.alpinelinux.org/alpine/aports/-/issues/10960
>
> Assuming you actually traced and saw the unlink syscall succeed, the
> root cause here is the filesystem/kernel lying about that. Standard
> "NFS Considered Harmful" stuff.
>
> But the fact that you're seeing different behavior on Alpine is almost
> surely a matter of busybox rm vs GNU coreutils differences in how they
> behave under faulty kernel behavior. The easy solution is probably
> installing the coreutils package. Otherwise, investigate what busybox
> is doing differently and if there's a way it could be made more
> reliable in this situation.
>
> Rich
>
>

[-- Attachment #2: Type: text/html, Size: 5806 bytes --]

Re: [musl] unlink on NFS volume fails silently

[Stephen Von Takach]

[-- Attachment #1: Type: text/plain, Size: 1217 bytes --]

We're using docker containers running on the same kernel using the same
mount setup.
Works on debian, does not work on alpine.

The difference is at the libc interface.


Stephen von Takach Dukai

Engineering Lead

PlaceOS

Australia, Hong Kong, London, New York

p: +61 408 419 954

e: steve@place.technology


On Thu, 10 Jul 2025 at 04:41, Rich Felker <dalias@libc.org> wrote:

> On Wed, Jul 09, 2025 at 01:59:18PM +1000, Stephen Von Takach wrote:
> > Yes we traced this.
> > The libc unlink function on musl returned 0 for a filename on an NFS
> mount
> > that wasn't deleted.
> > https://www.gnu.org/software/libc/manual/html_node/Deleting-Files.html
> >
> > The same call to unlink on glibc returned 0 and actually removed the
> file.
>
> They are not doing anything different. If you're getting a different
> result, it's something else different about the systems, possibly as
> random as timing differences between the program using musl and the
> one using glibc. But I would also check for different kernel versions
> or configurations, mount setups, etc.
>
> > The issue occurs when there is a high volume of files being removed
>
> Sounds like a timing dependent bug in the NFS implementation.
>
> Rich
>

[-- Attachment #2: Type: text/html, Size: 4695 bytes --]