fdopen/fflush problem

fdopen/fflush problem

[黄建忠]

Hi, there,
I encounter a problem about fdopen and fflush with musl, here is the code:

#include <stdio.h>
#include <stdlib.h>
#include <fcntl.h>

int main()
{
char filename[17] = "/tmp/abcdeXXXXXX";
int fd = mkostemp(filename, O_WRONLY|O_CLOEXEC);
printf("filename: %s\n", filename);
if(fd < 0)
printf("fd error\n");

FILE *f = fdopen(fd, "we");
fputs("test string\n", f);
fflush(f);

if(ferror(f))
printf("file error\n");

fclose(f);
}

As expected, the final file should contains one line and no ferror reported.

Anybody can help to check it?

By the way, I did not subscribe to this mail list with this mail box,
please CC to jianzhong.huang AT i-soft.com.cn

-- 
Huang JianZhong

Re: fdopen/fflush problem

[Rich Felker]

On Fri, Sep 26, 2014 at 02:05:57PM +0800, 黄建忠 wrote:
> Hi, there,
> I encounter a problem about fdopen and fflush with musl, here is the code:
> 
> #include <stdio.h>
> #include <stdlib.h>
> #include <fcntl.h>
> 
> int main()
> {
> char filename[17] = "/tmp/abcdeXXXXXX";
> int fd = mkostemp(filename, O_WRONLY|O_CLOEXEC);
> printf("filename: %s\n", filename);
> if(fd < 0)
> printf("fd error\n");
> 
> FILE *f = fdopen(fd, "we");
> fputs("test string\n", f);
> fflush(f);
> 
> if(ferror(f))
> printf("file error\n");
> 
> fclose(f);
> }
> 
> As expected, the final file should contains one line and no ferror reported.
> 
> Anybody can help to check it?

See the following text from the man page:

    The mkostemp() function is like mkstemp(), with the difference
    that the following bits—with the same meaning as for open(2)—may
    be specified in flags: O_APPEND, O_CLOEXEC, and O_SYNC. Note that
    when creating the file, mkostemp() includes the values O_RDWR,
    O_CREAT, and O_EXCL in the flags argument given to open(2);
    including these values in the flags argument given to mkostemp()
    is unnecessary, and produces errors on some systems.

mkostemp implicitly uses the access mode O_RDWR. The flags argument
should only contain additional flags you want, not the access mode.
When you pass O_WRONLY, it gets OR'd onto O_RDWR, producing a result
of "3" which is "no access".

I could add code to make mkostemp strip the access mode bits before
OR'ing on O_RDWR, but since it's documented in the man page that this
can "produce errors", I think it's best for you just not to pass
O_WRONLY. Changing this line:

int fd = mkostemp(filename, O_WRONLY|O_CLOEXEC);

to this:

int fd = mkostemp(filename, O_CLOEXEC);

should fix the problem.

Here's a link to the man page:

http://man7.org/linux/man-pages/man3/mkstemp.3.html

Rich

Re: fdopen/fflush problem

[Szabolcs Nagy]

* Rich Felker <dalias@libc.org> [2014-09-26 02:18:10 -0400]:
> See the following text from the man page:
> 
>     The mkostemp() function is like mkstemp(), with the difference
>     that the following bits???with the same meaning as for open(2)???may
>     be specified in flags: O_APPEND, O_CLOEXEC, and O_SYNC. Note that
>     when creating the file, mkostemp() includes the values O_RDWR,
>     O_CREAT, and O_EXCL in the flags argument given to open(2);
>     including these values in the flags argument given to mkostemp()
>     is unnecessary, and produces errors on some systems.

ah i didnt see this, so it's not a musl bug

Re: fdopen/fflush problem

[Rich Felker]

On Fri, Sep 26, 2014 at 08:39:39AM +0200, Szabolcs Nagy wrote:
> * Rich Felker <dalias@libc.org> [2014-09-26 02:18:10 -0400]:
> > See the following text from the man page:
> > 
> >     The mkostemp() function is like mkstemp(), with the difference
> >     that the following bits???with the same meaning as for open(2)???may
> >     be specified in flags: O_APPEND, O_CLOEXEC, and O_SYNC. Note that
> >     when creating the file, mkostemp() includes the values O_RDWR,
> >     O_CREAT, and O_EXCL in the flags argument given to open(2);
> >     including these values in the flags argument given to mkostemp()
> >     is unnecessary, and produces errors on some systems.
> 
> ah i didnt see this, so it's not a musl bug

I forgot that this function was accepted for inclusion in POSIX, so we
need to check that too. The accepted text is in
http://austingroupbugs.net/view.php?id=411

    "The mkostemp( ) function shall be equivalent to the mkstemp( )
    function, except that the flag argument may contain additional
    flags (from <fcntl.h>) to be used as if by open( ). Behavior is
    unspecified if the flag argument contains more than the following
    flags:"

The list that follows includes just O_APPEND, O_CLOEXEC, and O_*SYNC,
so including O_WRONLY has unspecified behavior. However, since this is
just unspecified, not undefined, it seems bad for something "horribly
wrong" to happen, like a file with no access like we're getting now.
Indeed, POSIX will define an optional error:

    "The mkostemp( ) function may fail if:

    [EINVAL] The value of the flag argument is invalid."

So I think we should either make this error be detected, or silently
mask off the bad access mode. My leaning would be towards reporting it
as an error. Opinions?

Rich

Re: fdopen/fflush problem

[u-wsnj]

On Fri, Sep 26, 2014 at 03:05:48AM -0400, Rich Felker wrote:
>     unspecified if the flag argument contains more than the following
>     flags:"
> 
> The list that follows includes just O_APPEND, O_CLOEXEC, and O_*SYNC,
> so including O_WRONLY has unspecified behavior. However, since this is
> just unspecified, not undefined, it seems bad for something "horribly
> wrong" to happen, like a file with no access like we're getting now.
> Indeed, POSIX will define an optional error:
> 
>     "The mkostemp( ) function may fail if:
> 
>     [EINVAL] The value of the flag argument is invalid."
> 
> So I think we should either make this error be detected, or silently
> mask off the bad access mode. My leaning would be towards reporting it
> as an error. Opinions?

+1 (reporting an error)

Rune

Re: fdopen/fflush problem

[Anthony G. Basile]

On 09/26/14 03:23, u-wsnj@aetey.se wrote:
> On Fri, Sep 26, 2014 at 03:05:48AM -0400, Rich Felker wrote:
>>      unspecified if the flag argument contains more than the following
>>      flags:"
>>
>> The list that follows includes just O_APPEND, O_CLOEXEC, and O_*SYNC,
>> so including O_WRONLY has unspecified behavior. However, since this is
>> just unspecified, not undefined, it seems bad for something "horribly
>> wrong" to happen, like a file with no access like we're getting now.
>> Indeed, POSIX will define an optional error:
>>
>>      "The mkostemp( ) function may fail if:
>>
>>      [EINVAL] The value of the flag argument is invalid."
>>
>> So I think we should either make this error be detected, or silently
>> mask off the bad access mode. My leaning would be towards reporting it
>> as an error. Opinions?
>
> +1 (reporting an error)
>
> Rune
>

Both uclibc [1] and glibc are okay with the flag combination 
O_WRONLY|O_CLOEXEC.  With unspecified behaviour you really can't say 
which way to go here for portability.  The correct thing to do would be 
to get this behaviour specified (in SUSv4 or something??) since mkostemp 
is currently a GNU-ism.


Ref.

[1] In uclibc libc/stdlib/mkostemp.c calls __gen_tempname in 
libc/misc/internals/tempname.c with kind = __GT_FILE and opens the file with

     fd = open (tmpl, O_RDWR | O_CREAT | O_EXCL, mode);

which is different from how musl does it in __mkostemps in 
src/temp/mkostemps.c

    fd = open(template, flags | O_RDWR | O_CREAT | O_EXCL, 0600);

Looks like uclibc completely ignores O_APPEND, O_CLOEXEC, and O_*SYNC.

See

http://git.uclibc.org/uClibc/tree/libc/stdlib/mkostemp.c
http://git.uclibc.org/uClibc/tree/libc/misc/internals/tempname.c

http://git.musl-libc.org/cgit/musl/tree/src/temp/mkostemp.c
http://git.musl-libc.org/cgit/musl/tree/src/temp/mkostemps.c


-- 
Anthony G. Basile, Ph. D.
Chair of Information Technology
D'Youville College
Buffalo, NY 14201
(716) 829-8197

Re: fdopen/fflush problem

[Rich Felker]

On Sat, Sep 27, 2014 at 09:29:20AM -0400, Anthony G. Basile wrote:
> On 09/26/14 03:23, u-wsnj@aetey.se wrote:
> >On Fri, Sep 26, 2014 at 03:05:48AM -0400, Rich Felker wrote:
> >>     unspecified if the flag argument contains more than the following
> >>     flags:"
> >>
> >>The list that follows includes just O_APPEND, O_CLOEXEC, and O_*SYNC,
> >>so including O_WRONLY has unspecified behavior. However, since this is
> >>just unspecified, not undefined, it seems bad for something "horribly
> >>wrong" to happen, like a file with no access like we're getting now.
> >>Indeed, POSIX will define an optional error:
> >>
> >>     "The mkostemp( ) function may fail if:
> >>
> >>     [EINVAL] The value of the flag argument is invalid."
> >>
> >>So I think we should either make this error be detected, or silently
> >>mask off the bad access mode. My leaning would be towards reporting it
> >>as an error. Opinions?
> >
> >+1 (reporting an error)
> >
> >Rune
> >
> 
> Both uclibc [1] and glibc are okay with the flag combination
> O_WRONLY|O_CLOEXEC.  With unspecified behaviour you really can't say
> which way to go here for portability.  The correct thing to do would
> be to get this behaviour specified (in SUSv4 or something??) since
> mkostemp is currently a GNU-ism.

Except that it's not; it's accepted for POSIX and you can read the
text for it here:

http://austingroupbugs.net/view.php?id=411

It seems the unspecified behavior is intentional. But I don't see
anything you could meaningfully do with the access mode since it's not
mandatory; an explicit mode of O_RDONLY would be indistinguishable
from a request for the default (O_RDWR), which is not a big problem
since O_RDONLY doesn't make sense for a new file you're creating, but
this only works because O_RDONLY happens to be the one with value 0.
If O_WRONLY had the value zero, you couldn't honor it. So really, the
only viable implementation choices seem to be silently ignoring the
mode, or throwing an error if any mode bits are specified. And since
we can't detect all modes (e.g. O_RDONLY can't be detected since it's
0) I think it somewhat makes sense, from a consistency standpoint, to
just ignore the access mode bits...

> [1] In uclibc libc/stdlib/mkostemp.c calls __gen_tempname in
> libc/misc/internals/tempname.c with kind = __GT_FILE and opens the
> file with
> 
>     fd = open (tmpl, O_RDWR | O_CREAT | O_EXCL, mode);
> 
> which is different from how musl does it in __mkostemps in
> src/temp/mkostemps.c
> 
>    fd = open(template, flags | O_RDWR | O_CREAT | O_EXCL, 0600);
> 
> Looks like uclibc completely ignores O_APPEND, O_CLOEXEC, and O_*SYNC.

Wow. So they just made a fake version of this function which ignores
the whole purpose it was created for -- atomic close-on-exec? Care to
report this?

Rich

Re: fdopen/fflush problem

[Anthony G. Basile]

On 09/27/14 09:38, Rich Felker wrote:
> On Sat, Sep 27, 2014 at 09:29:20AM -0400, Anthony G. Basile wrote:
>> On 09/26/14 03:23, u-wsnj@aetey.se wrote:
>>> On Fri, Sep 26, 2014 at 03:05:48AM -0400, Rich Felker wrote:
>>>>      unspecified if the flag argument contains more than the following
>>>>      flags:"
>>>>
>>>> The list that follows includes just O_APPEND, O_CLOEXEC, and O_*SYNC,
>>>> so including O_WRONLY has unspecified behavior. However, since this is
>>>> just unspecified, not undefined, it seems bad for something "horribly
>>>> wrong" to happen, like a file with no access like we're getting now.
>>>> Indeed, POSIX will define an optional error:
>>>>
>>>>      "The mkostemp( ) function may fail if:
>>>>
>>>>      [EINVAL] The value of the flag argument is invalid."
>>>>
>>>> So I think we should either make this error be detected, or silently
>>>> mask off the bad access mode. My leaning would be towards reporting it
>>>> as an error. Opinions?
>>>
>>> +1 (reporting an error)
>>>
>>> Rune
>>>
>>
>> Both uclibc [1] and glibc are okay with the flag combination
>> O_WRONLY|O_CLOEXEC.  With unspecified behaviour you really can't say
>> which way to go here for portability.  The correct thing to do would
>> be to get this behaviour specified (in SUSv4 or something??) since
>> mkostemp is currently a GNU-ism.
>
> Except that it's not; it's accepted for POSIX and you can read the
> text for it here:
>
> http://austingroupbugs.net/view.php?id=411

Thanks.  I didn't know that.

>
> It seems the unspecified behavior is intentional. But I don't see
> anything you could meaningfully do with the access mode since it's not
> mandatory; an explicit mode of O_RDONLY would be indistinguishable
> from a request for the default (O_RDWR), which is not a big problem
> since O_RDONLY doesn't make sense for a new file you're creating, but
> this only works because O_RDONLY happens to be the one with value 0.
> If O_WRONLY had the value zero, you couldn't honor it. So really, the
> only viable implementation choices seem to be silently ignoring the
> mode, or throwing an error if any mode bits are specified. And since
> we can't detect all modes (e.g. O_RDONLY can't be detected since it's
> 0) I think it somewhat makes sense, from a consistency standpoint, to
> just ignore the access mode bits...

Yeah, makes sense.

>
>> [1] In uclibc libc/stdlib/mkostemp.c calls __gen_tempname in
>> libc/misc/internals/tempname.c with kind = __GT_FILE and opens the
>> file with
>>
>>      fd = open (tmpl, O_RDWR | O_CREAT | O_EXCL, mode);
>>
>> which is different from how musl does it in __mkostemps in
>> src/temp/mkostemps.c
>>
>>     fd = open(template, flags | O_RDWR | O_CREAT | O_EXCL, 0600);
>>
>> Looks like uclibc completely ignores O_APPEND, O_CLOEXEC, and O_*SYNC.
>
> Wow. So they just made a fake version of this function which ignores
> the whole purpose it was created for -- atomic close-on-exec? Care to
> report this?
>
> Rich
>

Yes, I am definitely going to report this.  Defeating the atomic 
O_CLOEXEC is painful.  This is the second time atomicity was overlooked 
in uclibc (that I know of), the other being pread/pwrite implemented as 
open/lseek/{read,write}.  I'm not sure if this is legacy cruft going 
back to time when linux kernels didn't provide the support or what.


-- 
Anthony G. Basile, Ph. D.
Chair of Information Technology
D'Youville College
Buffalo, NY 14201
(716) 829-8197

Re: fdopen/fflush problem

[Szabolcs Nagy]

* ?????? <jianzhong.huang@i-soft.com.cn> [2014-09-26 14:05:57 +0800]:
> Hi, there,
> I encounter a problem about fdopen and fflush with musl, here is the code:

the strace is

open("/tmp/abcdeLidlpm", O_ACCMODE|O_CREAT|O_EXCL|O_LARGEFILE|O_CLOEXEC, 0600) = 3
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
writev(1, [{"filename: /tmp/abcdeLidlpm", 26}, {"\n", 1}], 2filename: /tmp/abcdeLidlpm
) = 27
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfbdb314) = -1 ENOTTY (Inappropriate ioctl for device)
writev(3, [{"test string\n", 12}, {NULL, 0}], 2) = -1 EBADF (Bad file descriptor)

> int main()
> {
> char filename[17] = "/tmp/abcdeXXXXXX";
> int fd = mkostemp(filename, O_WRONLY|O_CLOEXEC);

it seems O_WRONLY gets lost

this is what musl does:

	if ((fd = open(template, flags | O_RDWR | O_CREAT | O_EXCL, 0600))>=0)

and O_WRONLY is 1, O_RDWR is 2 and O_ACCMODE is 3

so it is a musl bug: due to historical accidents
O_RDONLY and O_WRONLY are not proper bitflags
and thus O_WRONLY|O_RDWR has special meaning

(for details see "File access mode" section in
http://man7.org/linux/man-pages/man2/open.2.html
)

> printf("filename: %s\n", filename);
> if(fd < 0)
> printf("fd error\n");
> 
> FILE *f = fdopen(fd, "we");
> fputs("test string\n", f);
> fflush(f);
> 
> if(ferror(f))
> printf("file error\n");
> 
> fclose(f);
> }
> 
> As expected, the final file should contains one line and no ferror reported.
> 
> Anybody can help to check it?
> 
> By the way, I did not subscribe to this mail list with this mail box,
> please CC to jianzhong.huang AT i-soft.com.cn
> 
> -- 
> Huang JianZhong
> 
>