* readdir(3): behavior on descriptors with O_SEARCH [not found] <CAMqzjetOkwQ7wi4p3MY_HT46v6pVRh_eHSi6SbwC96qoz+ivFg@mail.gmail.com> @ 2016-08-27 18:23 ` Dmitry Selyutin 2016-08-28 7:10 ` Markus Wichmann 0 siblings, 1 reply; 8+ messages in thread From: Dmitry Selyutin @ 2016-08-27 18:23 UTC (permalink / raw) To: musl [-- Attachment #1: Type: text/plain, Size: 2165 bytes --] Hello everyone! First of all, thank you for musl; it is an amazing tool and I really glad I'm using it. I'm able to compile and run everything I write (of course if POSIX standard applies). Recently I've found some edge case where both musl-gcc and musl-clang behave differently compared to usual gcc, clang and tcc, and I think the reason lies in the musl library. The example code: int const flags = (O_DIRECTORY | O_SEARCH); int descriptor = open(path, flags); DIR *handle = fdopendir(descriptor); struct dirent *entry = readdir(handle); To cut the long story short, any attempt to call readdir(3) on directory handle obtained via fdopendir(3) returns NULL and sets the errno variable to EBADF. This behavior arises only on descriptors opened with (O_DIRECTORY | O_SEARCH) flags enabled; it goes away if O_SEARCH flag is removed. So it seems that O_SEARCH is the reason; I thought that this flag tells exactly "well, I'm going to use it for search only", which implies "well, I'm going to use only readdir(3) to get information about files inside". Is my interpretation correct? I'm not really sure if it is a bug, since I suspect POSIX may allow open(3) with (O_DIRECTORY | O_SEARCH) flags to behave in an implementation-defined matter; it can be possible that file descriptors obtained via open(3) with O_DIRECTORY flag set are guaranteed to work only with fchdir(3) and *at(3) operations. However, if such behavior is intentional, it would be a good idea (in my opinion) make fdopendir(3) return NULL (though it won't match behavior e.g. for glibc). I don't know if this behavior still matters since I doubt I use the latest musl version; I couldn't find any relevant information about this behavior though. The musl version I'm using is 1.1.15; the code is compiled for x86_64. I'm using the version provided by Arch repositories. I don't have a stable Internet connection now, so I couldn't manage to take a look at the corresponding code in musl; I'll do it as soon as possible and create a patch if my assumptions are correct (and if the above behavior still applies to musl). Thank you for musl and patience (well, it was quite a long letter)! [-- Attachment #2: Type: text/html, Size: 2444 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: readdir(3): behavior on descriptors with O_SEARCH 2016-08-27 18:23 ` readdir(3): behavior on descriptors with O_SEARCH Dmitry Selyutin @ 2016-08-28 7:10 ` Markus Wichmann 2016-08-28 8:12 ` Dmitry Selyutin 0 siblings, 1 reply; 8+ messages in thread From: Markus Wichmann @ 2016-08-28 7:10 UTC (permalink / raw) To: musl On Sat, Aug 27, 2016 at 09:23:50PM +0300, Dmitry Selyutin wrote: > int const flags = (O_DIRECTORY | O_SEARCH); > int descriptor = open(path, flags); > DIR *handle = fdopendir(descriptor); > struct dirent *entry = readdir(handle); > > To cut the long story short, any attempt to call readdir(3) on directory > handle obtained via fdopendir(3) returns NULL and sets the errno variable > to EBADF. This behavior arises only on descriptors opened with (O_DIRECTORY > | O_SEARCH) flags enabled; it goes away if O_SEARCH flag is removed. > Try strace(1). That should tell you what glibc and musl are doing differently. For instace, whether glibc removes O_SEARCH from the fd. musl defines O_SEARCH to be equal to O_PATH. The manpage says that O_PATH means the file isn't opened for reading. I guess if you do that then getdents(2) will fail, which is what musl uses to implement readdir(3). I tried to do the same trace in glibc 2.19 (which is what Debian stable is using right now), but to no avail: O_SEARCH isn't even mentioned anywhere in that code. But its implementation of fdopendir(3) rejects fds open only for writing. The readdir(3) implementation is, of course, overcomplicated, but also seems to just call getdents(2). And then it tries to pack the kernel structures into its own structures, probably for ABI reasons. And people wonder why I dislike dynamic linking... > So it seems that O_SEARCH is the reason; I thought that this flag tells > exactly "well, I'm going to use it for search only", which implies "well, > I'm going to use only readdir(3) to get information about files inside". Is > my interpretation correct? > My manpage doesn't know O_SEARCH, but it knows O_PATH, and then you're wrong. It means "I'll only use this fd in *at() and fchdir() and similar; this fd isn't open for reading." > I'm not really sure if it is a bug, since I suspect POSIX may allow open(3) > with (O_DIRECTORY | O_SEARCH) flags to behave in an implementation-defined > matter; it can be possible that file descriptors obtained via open(3) with > O_DIRECTORY flag set are guaranteed to work only with fchdir(3) and *at(3) > operations. However, if such behavior is intentional, it would be a good > idea (in my opinion) make fdopendir(3) return NULL (though it won't match > behavior e.g. for glibc). > POSIX doesn't know O_SEARCH or O_PATH, and thus mandates nothing about their meaning. Ciao, Markus ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: readdir(3): behavior on descriptors with O_SEARCH 2016-08-28 7:10 ` Markus Wichmann @ 2016-08-28 8:12 ` Dmitry Selyutin 2016-08-28 9:02 ` Dmitry Selyutin 0 siblings, 1 reply; 8+ messages in thread From: Dmitry Selyutin @ 2016-08-28 8:12 UTC (permalink / raw) To: musl [-- Attachment #1: Type: text/plain, Size: 3418 bytes --] Hi Markus, thank you for your reply! > POSIX doesn't know O_SEARCH or O_PATH, and thus mandates nothing about > their meaning. POSIX 2008 with 2013 corrigenda mentions both O_SEARCH and O_EXEC. http://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html > Try strace(1). That should tell you what glibc and musl are doing > differently. For instace, whether glibc removes O_SEARCH from the fd. Thank you! I'll try it and report the results here. FWIW it does not seem to be correct that O_SEARCH can be equal to O_PATH; from what I gathered from manuals and various mailing lists discussions, O_SEARCH may be equal to O_EXEC since each of these flags is valid either for directory or file respectively. 28 авг. 2016 г. 10:11 пользователь "Markus Wichmann" <nullplan@gmx.net> написал: > On Sat, Aug 27, 2016 at 09:23:50PM +0300, Dmitry Selyutin wrote: > > int const flags = (O_DIRECTORY | O_SEARCH); > > int descriptor = open(path, flags); > > DIR *handle = fdopendir(descriptor); > > struct dirent *entry = readdir(handle); > > > > To cut the long story short, any attempt to call readdir(3) on directory > > handle obtained via fdopendir(3) returns NULL and sets the errno variable > > to EBADF. This behavior arises only on descriptors opened with > (O_DIRECTORY > > | O_SEARCH) flags enabled; it goes away if O_SEARCH flag is removed. > > > > Try strace(1). That should tell you what glibc and musl are doing > differently. For instace, whether glibc removes O_SEARCH from the fd. > > musl defines O_SEARCH to be equal to O_PATH. The manpage says that > O_PATH means the file isn't opened for reading. I guess if you do that > then getdents(2) will fail, which is what musl uses to implement > readdir(3). > > I tried to do the same trace in glibc 2.19 (which is what Debian stable > is using right now), but to no avail: O_SEARCH isn't even mentioned > anywhere in that code. But its implementation of fdopendir(3) rejects > fds open only for writing. The readdir(3) implementation is, of course, > overcomplicated, but also seems to just call getdents(2). And then it > tries to pack the kernel structures into its own structures, probably > for ABI reasons. And people wonder why I dislike dynamic linking... > > > So it seems that O_SEARCH is the reason; I thought that this flag tells > > exactly "well, I'm going to use it for search only", which implies "well, > > I'm going to use only readdir(3) to get information about files inside". > Is > > my interpretation correct? > > > > My manpage doesn't know O_SEARCH, but it knows O_PATH, and then you're > wrong. It means "I'll only use this fd in *at() and fchdir() and > similar; this fd isn't open for reading." > > > I'm not really sure if it is a bug, since I suspect POSIX may allow > open(3) > > with (O_DIRECTORY | O_SEARCH) flags to behave in an > implementation-defined > > matter; it can be possible that file descriptors obtained via open(3) > with > > O_DIRECTORY flag set are guaranteed to work only with fchdir(3) and > *at(3) > > operations. However, if such behavior is intentional, it would be a good > > idea (in my opinion) make fdopendir(3) return NULL (though it won't match > > behavior e.g. for glibc). > > > > POSIX doesn't know O_SEARCH or O_PATH, and thus mandates nothing about > their meaning. > > Ciao, > Markus > [-- Attachment #2: Type: text/html, Size: 4068 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: readdir(3): behavior on descriptors with O_SEARCH 2016-08-28 8:12 ` Dmitry Selyutin @ 2016-08-28 9:02 ` Dmitry Selyutin 2016-08-28 15:06 ` Rich Felker 0 siblings, 1 reply; 8+ messages in thread From: Dmitry Selyutin @ 2016-08-28 9:02 UTC (permalink / raw) To: Dmitry Selyutin; +Cc: musl [-- Attachment #1: Type: text/plain, Size: 1205 bytes --] I think the mistery is partially solved: glibc simply doesn't have O_SEARCH, so the following code #ifdef O_SEARCH flags |= O_SEARCH #endif does not execute (so only O_DIRECTORY is being set). Dumb, had to check twice, sorry for the noise. However, the question if it is correct to define O_SEARCH to be equal to O_PATH. From what I see, both O_SEARCH and O_EXEC have the same value as O_PATH, but I'm not sure if this solution is technically correct. However, I suspect that support for O_EXEC and O_SEARCH must be provided by the kernel first, so until kernel implements such functionality, all talks seem to be meaningless (unless someone wants to emulate such functionality in the userspace). Anyway, I tend to think that any attempt to use O_SEARCH until then should either return -1 on open(3), or NULL on fdopendir(3); I really don't think that O_SEARCH shall mean the same as O_PATH. But I'm by no means claim myself to be a POSIX expert; what do you think? The information on O_SEARCH seems to be incomplete and even inconsistent, so it may turn that my claims are incorrect. Any suggestions are welcome; I'll take a more deep look at it in the evening. And again, thank you for your help! [-- Attachment #2: Type: text/html, Size: 1355 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: readdir(3): behavior on descriptors with O_SEARCH 2016-08-28 9:02 ` Dmitry Selyutin @ 2016-08-28 15:06 ` Rich Felker 2016-08-28 16:02 ` Dmitry Selyutin 0 siblings, 1 reply; 8+ messages in thread From: Rich Felker @ 2016-08-28 15:06 UTC (permalink / raw) To: musl On Sun, Aug 28, 2016 at 12:02:02PM +0300, Dmitry Selyutin wrote: > I think the mistery is partially solved: glibc simply doesn't have > O_SEARCH, so the following code > > #ifdef O_SEARCH > flags |= O_SEARCH > #endif If a program is doing this then trying to fdopendir/readdir, it's a bug in the program. A directory opened with O_SEARCH is only usable for search (attempting to access a file/directory in that directory by name, using one of the *at functions) not for reading. Unix has always distinguished search (+x) and read (+r) permission for directories and O_SEARCH vs O_RDONLY is similar. > However, the question if it is correct to define O_SEARCH to be equal to > O_PATH. From what I see, both O_SEARCH and O_EXEC have the same value as > O_PATH, but I'm not sure if this solution is technically correct. It's not quite, but there are only a few minor technical differences, and most of them can be papered over from userspace. We're not doing all that yet. The only one I'm aware of that needs kernel help is allowing search even if the user has lost permission (chmod -x) between the time the directory was opened successfully for O_SEARCH and the time of the *at function. > However, I suspect that support for O_EXEC and O_SEARCH must be provided by > the kernel first, so until kernel implements such functionality, all talks > seem to be meaningless (unless someone wants to emulate such functionality > in the userspace). Anyway, I tend to think that any attempt to use O_SEARCH > until then should either return -1 on open(3), or NULL on fdopendir(3); I > really don't think that O_SEARCH shall mean the same as O_PATH. It's been an ongoing fight even trying to get the kernel to reserve a bit number for them. It looks like the correct course of action (the one that's compatible with their non-action) is going to be using O_PATH|3 for both of them. This allows userspace open to process O_SEARCH and O_EXEC slightly differently from O_PATH (O_NOFOLLOW has different semantics) and the kernel will ignore the extra access mode bits with O_PATH anyway. > But I'm by no means claim myself to be a POSIX expert; what do you think? > The information on O_SEARCH seems to be incomplete and even inconsistent, > so it may turn that my claims are incorrect. Any suggestions are welcome; > I'll take a more deep look at it in the evening. I don't see what's incomplete or inconsistent about it. Rich ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: readdir(3): behavior on descriptors with O_SEARCH 2016-08-28 15:06 ` Rich Felker @ 2016-08-28 16:02 ` Dmitry Selyutin 2016-08-28 16:20 ` Rich Felker 0 siblings, 1 reply; 8+ messages in thread From: Dmitry Selyutin @ 2016-08-28 16:02 UTC (permalink / raw) To: musl [-- Attachment #1: Type: text/plain, Size: 2168 bytes --] Hi Rich, thank you for the detailed answer! > If a program is doing this then trying to fdopendir/readdir, it's a > bug in the program. A directory opened with O_SEARCH is only usable > for search (attempting to access a file/directory in that directory > by name, using one of the *at functions) not for reading. Unix has > always distinguished search (+x) and read (+r) permission for > directories and O_SEARCH vs O_RDONLY is similar. Well, that's probably the worst kind of bugs, since it was caused by misinterpretation of the documentation. That was written intentionally due to misunderstanding, so I really want to clarify the situation here. So does it mean that descriptors opened with O_SEARCH are usable only for *at functions? I've been under the impression that it's part of the O_PATH functionality, not O_SEARCH. > It's been an ongoing fight even trying to get the kernel to reserve a > bit number for them. It looks like the correct course of action (the > one that's compatible with their non-action) is going to be using > O_PATH|3 for both of them. This allows userspace open to process > O_SEARCH and O_EXEC slightly differently from O_PATH (O_NOFOLLOW has > different semantics) and the kernel will ignore the extra access mode > bits with O_PATH anyway. At the same time, if I understand you correctly, you mean that O_PATH is a different beast than O_SEARCH. I've found a mail in the DragonFly mailing list, which also slightly touches this topic[0]. They also propose to use value 3 as currently unused value; I don't know if it is implemented yet. FWIW, neither OpenBSD nor FreeBSD provide O_SEARCH flag; the latter provides O_EXEC though. I finally have a good access to the Internet so I managed to find a detailed discussion on this topic[1]. Sorry guys (and especially you, Rich) that I didn't find it before; it would have saved some questions earlier. Rich, could you please elaborate on the exact semantics of O_SEARCH flag? Again, thank you very much for your answer and for your patience! [0] https://www.dragonflybsd.org/mailarchive/kernel/2009-08/msg00000.html [1] https://sourceware.org/ml/libc-alpha/2013-08/msg00016.html [-- Attachment #2: Type: text/html, Size: 2733 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: readdir(3): behavior on descriptors with O_SEARCH 2016-08-28 16:02 ` Dmitry Selyutin @ 2016-08-28 16:20 ` Rich Felker 2016-08-28 17:14 ` Dmitry Selyutin 0 siblings, 1 reply; 8+ messages in thread From: Rich Felker @ 2016-08-28 16:20 UTC (permalink / raw) To: musl On Sun, Aug 28, 2016 at 07:02:18PM +0300, Dmitry Selyutin wrote: > Hi Rich, > > thank you for the detailed answer! > > > If a program is doing this then trying to fdopendir/readdir, it's a > > bug in the program. A directory opened with O_SEARCH is only usable > > for search (attempting to access a file/directory in that directory > > by name, using one of the *at functions) not for reading. Unix has > > always distinguished search (+x) and read (+r) permission for > > directories and O_SEARCH vs O_RDONLY is similar. > Well, that's probably the worst kind of bugs, since it was caused by > misinterpretation of the documentation. > That was written intentionally due to misunderstanding, so I really want to > clarify the situation here. > So does it mean that descriptors opened with O_SEARCH are usable only for > *at functions? > I've been under the impression that it's part of the O_PATH functionality, > not O_SEARCH. http://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html O_RDONLY Open for reading only. ... O_SEARCH Open directory for search only. Note the word "only". Moreover it's clear from the permissions model that you can't read from a directory opened for O_SEARCH, since O_SEARCH does not require read permission; permission enforcement is specific in the errors (and possibly elsewhere): [EACCESS] Search permission is denied on a component of the path prefix, or the file exists and the permissions specified by oflag are denied, or the file does not exist and write permission is denied for the parent directory of the file to be created, or O_TRUNC is specified and write permission is denied. The relevant part of this text is "or the file exists and the permissions specified by oflag are denied". If you could read from a fd opened for O_SEARCH, you could bypass the check for "r" permission at the fs level and read any directory to which you had "x" permission. (In a sense you can do this anyway by brute-force search, but that does not work when filenames in a +x-r directory are sufficiently long and unpredictable, which is the case for secure uses of +x-r. > > It's been an ongoing fight even trying to get the kernel to reserve a > > bit number for them. It looks like the correct course of action (the > > one that's compatible with their non-action) is going to be using > > O_PATH|3 for both of them. This allows userspace open to process > > O_SEARCH and O_EXEC slightly differently from O_PATH (O_NOFOLLOW has > > different semantics) and the kernel will ignore the extra access mode > > bits with O_PATH anyway. > At the same time, if I understand you correctly, you mean that O_PATH is a > different beast than O_SEARCH. Linux implemented O_PATH independently of POSIX O_SEARCH and O_EXEC, not specifically with the intent to model them but rather to allow more general things. > I've found a mail in the DragonFly mailing list, which also slightly > touches this topic[0]. > They also propose to use value 3 as currently unused value; I don't know if > it is implemented yet. > FWIW, neither OpenBSD nor FreeBSD provide O_SEARCH flag; the latter > provides O_EXEC though. Linux actually uses the value 3 for a wacky purpose: ioctl-only opening of certain device nodes. Opening with mode 3 requires +rw permissions. This was a huge waste of the value but I think they feel obligated to keep it for compatibility. > I finally have a good access to the Internet so I managed to find a > detailed discussion on this topic[1]. > Sorry guys (and especially you, Rich) that I didn't find it before; it > would have saved some questions earlier. > > Rich, could you please elaborate on the exact semantics of O_SEARCH flag? > Again, thank you very much for your answer and for your patience! My understanding is that you can use an O_SEARCH fd for the *at functions and in any place where an operation is just being performed on the fd (like fstat, fchown, etc.) where the open modes are not relavant (e.g. fchown does not use the mode the file was opened with but checks permissions at the time of the fchown). You definitely cannot use it for read or write operations. Rich ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: readdir(3): behavior on descriptors with O_SEARCH 2016-08-28 16:20 ` Rich Felker @ 2016-08-28 17:14 ` Dmitry Selyutin 0 siblings, 0 replies; 8+ messages in thread From: Dmitry Selyutin @ 2016-08-28 17:14 UTC (permalink / raw) To: musl [-- Attachment #1: Type: text/plain, Size: 133 bytes --] I think I got the idea; now I understand that the reported behavior is not a bug. Thank you very much for explanations and patience! [-- Attachment #2: Type: text/html, Size: 150 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2016-08-28 17:14 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- [not found] <CAMqzjetOkwQ7wi4p3MY_HT46v6pVRh_eHSi6SbwC96qoz+ivFg@mail.gmail.com> 2016-08-27 18:23 ` readdir(3): behavior on descriptors with O_SEARCH Dmitry Selyutin 2016-08-28 7:10 ` Markus Wichmann 2016-08-28 8:12 ` Dmitry Selyutin 2016-08-28 9:02 ` Dmitry Selyutin 2016-08-28 15:06 ` Rich Felker 2016-08-28 16:02 ` Dmitry Selyutin 2016-08-28 16:20 ` Rich Felker 2016-08-28 17:14 ` Dmitry Selyutin
Code repositories for project(s) associated with this public inbox https://git.vuxu.org/mirror/musl/ This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).