realpath after chroot

realpath after chroot

[Reiner Herrmann]

[-- Attachment #1: Type: text/plain, Size: 717 bytes --]

Hi,

I noticed that realpath is no longer working after chroot is called.
With the example from below I get the following output with musl 1.1.23:

# ./a.out
/
No such file or directory

With glibc it is working as I would expect:

# ./a.out
/
/


#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <limits.h>
#include <unistd.h>
#include <errno.h>

int main() {
	char *p = realpath("/", NULL);
	if (!p) {
		printf("%s\n", strerror(errno));
	} else {
		printf("%s\n", p);
		free(p);
	}

	if (chroot("/tmp") == -1)
		printf("chroot failed\n");

	p = realpath("/", NULL);
	if (!p) {
		printf("%s\n", strerror(errno));
	} else {
		printf("%s\n", p);
		free(p);
	}

	return 0;
}


Kind regards,
  Reiner

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: realpath after chroot

[Rich Felker]

On Tue, Oct 08, 2019 at 07:24:02PM +0200, Reiner Herrmann wrote:
> Hi,
> 
> I noticed that realpath is no longer working after chroot is called.
> With the example from below I get the following output with musl 1.1.23:
> 
> # ./a.out
> /
> No such file or directory
> 
> With glibc it is working as I would expect:
> 
> # ./a.out
> /
> /

This is a documented requirement:

    musl aims to avoid imposing filesystem policy; however, the
    following minimal set of filesystems dependencies must be met in
    order for programs using musl to function correctly:

    ...

    - /proc - must be a mount point for Linux procfs or a symlink to
      such. Several functions such as realpath, fexecve, and a number
      of the "at" functions added in POSIX 2008 need access to /proc
      to function correctly.

Source: https://www.musl-libc.org/doc/1.0.0/manual.html

It's been discussed in more depth in other places. Basically, Linux
makes it impossible to do some things needed for a fully working
C/POSIX implementation without /proc, so we have to treat it as a
"requirement". Some subset of functionality works without it, but no
formal specification of exactly what works is made by musl.

For realpath, indeed it can be implemented in userspace without /proc,
and it may be desirable to do so as a fallback. It might make sense to
do an analysis of "how essential" /proc still is on reasonably recent
kernels; if the need for it is isolated to dynamic linker stuff (rpath
origin, etc.) then it might make a lot of sense to formalize that
/proc is only mandatory for certain things.

Rich

Re: realpath after chroot

[Reiner Herrmann]

[-- Attachment #1: Type: text/plain, Size: 1188 bytes --]

On Tue, Oct 08, 2019 at 01:38:50PM -0400, Rich Felker wrote:
> > I noticed that realpath is no longer working after chroot is called.
> 
> This is a documented requirement:
> 
[...]
> 
> Source: https://www.musl-libc.org/doc/1.0.0/manual.html
> 
> It's been discussed in more depth in other places. Basically, Linux
> makes it impossible to do some things needed for a fully working
> C/POSIX implementation without /proc, so we have to treat it as a
> "requirement". Some subset of functionality works without it, but no
> formal specification of exactly what works is made by musl.
> 
> For realpath, indeed it can be implemented in userspace without /proc,
> and it may be desirable to do so as a fallback. It might make sense to
> do an analysis of "how essential" /proc still is on reasonably recent
> kernels; if the need for it is isolated to dynamic linker stuff (rpath
> origin, etc.) then it might make a lot of sense to formalize that
> /proc is only mandatory for certain things.

Interesting, thank you for the explanation.
For a daemon that chroots somewhere I then need to either mount proc
into the chroot or avoid realpath.

Regards,
  Reiner

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: realpath after chroot

[Markus Wichmann]

On Tue, Oct 08, 2019 at 01:38:50PM -0400, Rich Felker wrote:
> It's been discussed in more depth in other places. Basically, Linux
> makes it impossible to do some things needed for a fully working
> C/POSIX implementation without /proc, so we have to treat it as a
> "requirement". Some subset of functionality works without it, but no
> formal specification of exactly what works is made by musl.
>

Well, what does depend on /proc at the moment? Of course, there is
everything calling __procfdname(), so that would be

- realpath() (main path)
- fexecve() (fallback path)
- fchmod() (fallback path)
- fchmodat() (main path for AT_SYMLINK_NOFOLLOW)
- fstatat() (fallback path)
- fchdir() (fallback path)
- fchown() (fallback path)
- ttyname_r() (main path), and ttyname() by extension

Besides that, the dynlinker tries to find $ORIGIN by means of
/proc/self/exe, the static version of dl_iterate_phdr() uses the same
file name as executable file name (does that count as a "use"?), and of
course pthread_setname_np() tries to write to
/proc/self/task/<tid>/comm.

So, on a recent enough kernel, only realpath(), ttyname(),
pthread_setname_np() and some fchmodat() calls remain.

> For realpath, indeed it can be implemented in userspace without /proc,
> and it may be desirable to do so as a fallback.

The only algorithm I can think of for this is to implement
path_resolution(7). Which I did once (to see if I could), and it was a
lot of code. But it boils down to calling lstat() on every single path
component, and then readlink(), as necessary.

Personally, I dislike that, because you have to implement something that
already happens in the kernel, but now outside the kernel. With the same
algorithm present in two places, it is likely that they will diverge.
One point of disagreement might be the number of symlinks required to
return ELOOP. Another might be name or path size limits (maybe a future
kernel no longer enforces them).

> It might make sense to
> do an analysis of "how essential" /proc still is on reasonably recent
> kernels; if the need for it is isolated to dynamic linker stuff (rpath
> origin, etc.) then it might make a lot of sense to formalize that
> /proc is only mandatory for certain things.

See above, though I don't know if you want to formalize that.

Ciao,
Markus

Re: realpath after chroot

[Rich Felker]

On Tue, Oct 08, 2019 at 09:56:23PM +0200, Markus Wichmann wrote:
> On Tue, Oct 08, 2019 at 01:38:50PM -0400, Rich Felker wrote:
> > It's been discussed in more depth in other places. Basically, Linux
> > makes it impossible to do some things needed for a fully working
> > C/POSIX implementation without /proc, so we have to treat it as a
> > "requirement". Some subset of functionality works without it, but no
> > formal specification of exactly what works is made by musl.
> >
> 
> Well, what does depend on /proc at the moment? Of course, there is
> everything calling __procfdname(), so that would be
> 
> - realpath() (main path)
> - fexecve() (fallback path)
> - fchmod() (fallback path)
> - fchmodat() (main path for AT_SYMLINK_NOFOLLOW)
> - fstatat() (fallback path)
> - fchdir() (fallback path)
> - fchown() (fallback path)
> - ttyname_r() (main path), and ttyname() by extension

Thanks for working these out.

For the ones marked "fallback path", it's not entirely clear whether
the fallback path is only needed for old kernels, or possibly needed
even on recent/current ones. Historically Linux was very sloppy about
supporting some of these operations on O_PATH (used for
O_EXEC/O_SEARCH) fds.

> Besides that, the dynlinker tries to find $ORIGIN by means of
> /proc/self/exe, the static version of dl_iterate_phdr() uses the same
> file name as executable file name (does that count as a "use"?), and of
> course pthread_setname_np() tries to write to
> /proc/self/task/<tid>/comm.
> 
> So, on a recent enough kernel, only realpath(), ttyname(),
> pthread_setname_np() and some fchmodat() calls remain.

And possibly others (see above).

> > For realpath, indeed it can be implemented in userspace without /proc,
> > and it may be desirable to do so as a fallback.
> 
> The only algorithm I can think of for this is to implement
> path_resolution(7). Which I did once (to see if I could), and it was a
> lot of code. But it boils down to calling lstat() on every single path
> component, and then readlink(), as necessary.
> 
> Personally, I dislike that, because you have to implement something that
> already happens in the kernel, but now outside the kernel. With the same
> algorithm present in two places, it is likely that they will diverge.
> One point of disagreement might be the number of symlinks required to
> return ELOOP.

Perhaps; I'd have to think more about whether this is actually a
problem.

> Another might be name or path size limits (maybe a future
> kernel no longer enforces them).

Regardless of whether kernel enforces PATH_MAX, we do. It's a feature.
So I don't think this is a problem.

> > It might make sense to
> > do an analysis of "how essential" /proc still is on reasonably recent
> > kernels; if the need for it is isolated to dynamic linker stuff (rpath
> > origin, etc.) then it might make a lot of sense to formalize that
> > /proc is only mandatory for certain things.
> 
> See above, though I don't know if you want to formalize that.

Indeed, there are at least a few items of "standard functionality"
that depend on /proc, regardless of the status of the "fallback" ones:
at least ttyname and fchmodat. Note that ttyname can be done without
/proc by searching /dev for matching dev_t, either using known
patterns for tty names or a global search, but this is ugly too.

Rich

Re: realpath after chroot

[Markus Wichmann]

On Tue, Oct 08, 2019 at 05:10:10PM -0400, Rich Felker wrote:
> On Tue, Oct 08, 2019 at 09:56:23PM +0200, Markus Wichmann wrote:
> > Well, what does depend on /proc at the moment? Of course, there is
> > everything calling __procfdname(), so that would be
> >
> > - realpath() (main path)
> > - fexecve() (fallback path)
> > - fchmod() (fallback path)
> > - fchmodat() (main path for AT_SYMLINK_NOFOLLOW)
> > - fstatat() (fallback path)
> > - fchdir() (fallback path)
> > - fchown() (fallback path)
> > - ttyname_r() (main path), and ttyname() by extension
>
> Thanks for working these out.
>
> For the ones marked "fallback path", it's not entirely clear whether
> the fallback path is only needed for old kernels, or possibly needed
> even on recent/current ones. Historically Linux was very sloppy about
> supporting some of these operations on O_PATH (used for
> O_EXEC/O_SEARCH) fds.
>

Yes, I just had a glance at the code. For fchmod(), the fallback path is
triggered if SYS_fchmod returned EBADF, and yet the file descriptor
flags could be retrieved, indicating the FD is open. I'm guessing
SYS_fchmod is no longer an optional system call, but is bad about
O_PATH?

fstatat() is really special. There is the SYS_statx codepath which is
ignored on most architectures. At least until you push the time64_t
stuff. And __procfdname() is only called in the AT_EMPTY_PATH case, if
SYS_fstat displayed the same behavior as above (returning EBADF on an
open FD), and SYS_fstatat failed with EINVAL.

fchdir() and fchown() have the same code (entering the fallback path on
EBADF with open FD).

So it appears that fexecve() is the exception here, entering the
fallback path on ENOSYS.

> Indeed, there are at least a few items of "standard functionality"
> that depend on /proc, regardless of the status of the "fallback" ones:
> at least ttyname and fchmodat. Note that ttyname can be done without
> /proc by searching /dev for matching dev_t, either using known
> patterns for tty names or a global search, but this is ugly too.
>

I was about to protest that this would add /dev to the list of
dependencies. Then I noticed that ttyname() tries to stat() its result,
so if /dev isn't in the chroot jail, it does not work, either.

Ciao,
Markus

Re: realpath after chroot

[Rich Felker]

On Wed, Oct 09, 2019 at 05:47:49AM +0200, Markus Wichmann wrote:
> On Tue, Oct 08, 2019 at 05:10:10PM -0400, Rich Felker wrote:
> > On Tue, Oct 08, 2019 at 09:56:23PM +0200, Markus Wichmann wrote:
> > > Well, what does depend on /proc at the moment? Of course, there is
> > > everything calling __procfdname(), so that would be
> > >
> > > - realpath() (main path)
> > > - fexecve() (fallback path)
> > > - fchmod() (fallback path)
> > > - fchmodat() (main path for AT_SYMLINK_NOFOLLOW)
> > > - fstatat() (fallback path)
> > > - fchdir() (fallback path)
> > > - fchown() (fallback path)
> > > - ttyname_r() (main path), and ttyname() by extension
> >
> > Thanks for working these out.
> >
> > For the ones marked "fallback path", it's not entirely clear whether
> > the fallback path is only needed for old kernels, or possibly needed
> > even on recent/current ones. Historically Linux was very sloppy about
> > supporting some of these operations on O_PATH (used for
> > O_EXEC/O_SEARCH) fds.
> >
> 
> Yes, I just had a glance at the code. For fchmod(), the fallback path is
> triggered if SYS_fchmod returned EBADF, and yet the file descriptor
> flags could be retrieved, indicating the FD is open. I'm guessing
> SYS_fchmod is no longer an optional system call, but is bad about
> O_PATH?

It was never optional. The fallback is for kernels that return EBADF
on O_PATH fds.

> fstatat() is really special. There is the SYS_statx codepath which is
> ignored on most architectures. At least until you push the time64_t
> stuff. And __procfdname() is only called in the AT_EMPTY_PATH case, if
> SYS_fstat displayed the same behavior as above (returning EBADF on an
> open FD), and SYS_fstatat failed with EINVAL.

Even after time64 there will be plenty of kernels that don't have
statx. I'm not sure when fstat stopped failing with EBADF for O_PATH.
That's the important case I'm aware of.

> fchdir() and fchown() have the same code (entering the fallback path on
> EBADF with open FD).

Right.

> So it appears that fexecve() is the exception here, entering the
> fallback path on ENOSYS.

I don't see how it's more exceptional than the others.

> > Indeed, there are at least a few items of "standard functionality"
> > that depend on /proc, regardless of the status of the "fallback" ones:
> > at least ttyname and fchmodat. Note that ttyname can be done without
> > /proc by searching /dev for matching dev_t, either using known
> > patterns for tty names or a global search, but this is ugly too.
> >
> 
> I was about to protest that this would add /dev to the list of
> dependencies. Then I noticed that ttyname() tries to stat() its result,
> so if /dev isn't in the chroot jail, it does not work, either.

/dev is a hard POSIX dependency for /dev/null and /dev/tty, maybe
other things too.

Rich

Re: realpath after chroot

[Markus Wichmann]

On Wed, Oct 09, 2019 at 07:45:10AM -0400, Rich Felker wrote:
> On Wed, Oct 09, 2019 at 05:47:49AM +0200, Markus Wichmann wrote:
> > So it appears that fexecve() is the exception here, entering the
> > fallback path on ENOSYS.
>
> I don't see how it's more exceptional than the others.
>

What I meant is, having a fallback for ENOSYS/"kernel too old" is kind
of normal. Having a fallback for "kernel is drunk", not so much.

> /dev is a hard POSIX dependency for /dev/null and /dev/tty, maybe
> other things too.

Right, I forgot about those.

Ciao,
Markus