[PATCH] scanf: handle the L modifier for integers

[PATCH] scanf: handle the L modifier for integers

[Andrei Vagin]

Look at this code:
   char str[] = "sigmask: 0x200";
   long long mask = 0;
   int ret;

   ret = sscanf(str, "sigmask: %Lx", &mask));
   printf("%d %llx\n", ret, mask);

Without this patch, ret will be 1 and mask will be 0. It is obviously
incorrect. According to the man page, L should work like ll:

 L Indicates that the conversion will be either e, f, or g and the
   next pointer is a pointer to long double or the conversion will
   be d, i, o, u, or x and the next pointer is a pointer to long
   long.

Signed-off-by: Andrei Vagin <avagin@virtuozzo.com>
---
 src/stdio/vfscanf.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/src/stdio/vfscanf.c b/src/stdio/vfscanf.c
index 9e030fc4..4d0d771e 100644
--- a/src/stdio/vfscanf.c
+++ b/src/stdio/vfscanf.c
@@ -36,6 +36,7 @@ static void store_int(void *dest, int size, unsigned long long i)
 		*(long *)dest = i;
 		break;
 	case SIZE_ll:
+	case SIZE_L:
 		*(long long *)dest = i;
 		break;
 	}
-- 
2.14.3

Re: [PATCH] scanf: handle the L modifier for integers

[Laurent Bercot]

>Without this patch, ret will be 1 and mask will be 0. It is obviously
>incorrect. According to the man page, L should work like ll:
>
>L Indicates that the conversion will be either e, f, or g and the
>   next pointer is a pointer to long double or the conversion will
>   be d, i, o, u, or x and the next pointer is a pointer to long
>   long.

  This is a GNU extension. POSIX states that L is only valid before
a floating-point conversion specifier:

L
     Specifies that a following a, A, e, E, f, F, g, or G conversion 
specifier
     applies to an argument with type pointer to long double.

  from 
http://pubs.opengroup.org/onlinepubs/9699919799/functions/scanf.html

  So, it is valid for musl not to accept %Lx.
  Now, the argument that it's a good idea to align musl's behaviour to
glibc's whenever possible is a sensible one. But it's a decision for
the musl authors to make, and the pros and cons need to be carefully
balanced; musl's current behaviour is not _incorrect_.

--
  Laurent

Re: [PATCH] scanf: handle the L modifier for integers

[Andrei Vagin]

>>Without this patch, ret will be 1 and mask will be 0. It is obviously
>>incorrect. According to the man page, L should work like ll:
>>
>>L Indicates that the conversion will be either e, f, or g and the
>>   next pointer is a pointer to long double or the conversion will
>>   be d, i, o, u, or x and the next pointer is a pointer to long
>>   long.
>
>  This is a GNU extension. POSIX states that L is only valid before
>a floating-point conversion specifier:
>
>L
>     Specifies that a following a, A, e, E, f, F, g, or G conversion 
>specifier
>     applies to an argument with type pointer to long double.
>
>  from 
>http://pubs.opengroup.org/onlinepubs/9699919799/functions/scanf.html
>
>  So, it is valid for musl not to accept %Lx.
>  Now, the argument that it's a good idea to align musl's behaviour to
>glibc's whenever possible is a sensible one. But it's a decision for
>the musl authors to make, and the pros and cons need to be carefully
>balanced; musl's current behaviour is not _incorrect_.

It is incorrect, because scanf() has to return 0, or it has to handle the
L modifier. Currently it doesn't handle L and return 1, so the
application can't detect this issue.

I would prefer a case when musl works like glibc, if there are not any
reason to not to do that. For example,  now Alpine Linux is very popular
and there are a lot of packages. In many cases, a maintainer, who adds a
new package, fixes compile-time errors and doesn't run any tests.
A target application can work differently with musl comparing with glibc
due to this sort of issues.

Re: Re: [PATCH] scanf: handle the L modifier for integers

[Natanael Copa]

On Thu, 31 May 2018 12:00:22 -0700
Andrei Vagin <avagin@virtuozzo.com> wrote:

> >>Without this patch, ret will be 1 and mask will be 0. It is obviously
> >>incorrect. According to the man page, L should work like ll:
> >>
> >>L Indicates that the conversion will be either e, f, or g and the
> >>   next pointer is a pointer to long double or the conversion will
> >>   be d, i, o, u, or x and the next pointer is a pointer to long
> >>   long.  
> >
> >  This is a GNU extension. POSIX states that L is only valid before
> >a floating-point conversion specifier:
> >
> >L
> >     Specifies that a following a, A, e, E, f, F, g, or G conversion 
> >specifier
> >     applies to an argument with type pointer to long double.
> >
> >  from 
> >http://pubs.opengroup.org/onlinepubs/9699919799/functions/scanf.html
> >
> >  So, it is valid for musl not to accept %Lx.
> >  Now, the argument that it's a good idea to align musl's behaviour to
> >glibc's whenever possible is a sensible one. But it's a decision for
> >the musl authors to make, and the pros and cons need to be carefully
> >balanced; musl's current behaviour is not _incorrect_.  
> 
> It is incorrect, because scanf() has to return 0, or it has to handle the
> L modifier. Currently it doesn't handle L and return 1, so the
> application can't detect this issue.

That sounds like a bug in musl libc.
 
> I would prefer a case when musl works like glibc, if there are not any
> reason to not to do that. For example,  now Alpine Linux is very popular
> and there are a lot of packages. In many cases, a maintainer, who adds a
> new package, fixes compile-time errors and doesn't run any tests.
> A target application can work differently with musl comparing with glibc
> due to this sort of issues.

FreeBSD man page says:

     L	      Indicates	that the conversion will be one	of a, e, f, or g and
	      the next pointer is a pointer to long double.

NetBSD man page says:

     L       Indicates that the conversion will be efg and the next pointer is
             a pointer to long double.

OpenBSD man page says:
     
L
    Indicates that the conversion will be one of efg and the next pointer is a pointer to long double.

So the application will break on most (every) system that is not GNU
libc. It would be better to fix the application in this case:


   char str[] = "sigmask: 0x200";
   long long mask = 0;
   int ret;

#if defined(__GLIBC__)
   ret = sscanf(str, "sigmask: %Lx", &mask));
#else
   ret = sscanf(str, "sigmask: %llx", &mask));
#endif
   printf("%d %llx\n", ret, mask);



Or just use %llx which is POSIX and should work everywhere.

That said, those things are tricky to detect at compile time as you
mentioned and they are tricky to detect with configure scripts that
works with cross compilation. Also many developers seems to think that
Linux == glibc so they only read the GNU manuals, so yeah, implement
glibc behavior here seems like a good idea, unless someone else has a
brilliant idea how to catch this at compile time.

In any case, I think the application should be fixed too.

-nc

Re: Re: [PATCH] scanf: handle the L modifier for integers

[Andrei Vagin]

On Thu, May 31, 2018 at 10:44:42PM +0200, Natanael Copa wrote:
> On Thu, 31 May 2018 12:00:22 -0700
> Andrei Vagin <avagin@virtuozzo.com> wrote:
> 
> > >>Without this patch, ret will be 1 and mask will be 0. It is obviously
> > >>incorrect. According to the man page, L should work like ll:
> > >>
> > >>L Indicates that the conversion will be either e, f, or g and the
> > >>   next pointer is a pointer to long double or the conversion will
> > >>   be d, i, o, u, or x and the next pointer is a pointer to long
> > >>   long.  
> > >
> > >  This is a GNU extension. POSIX states that L is only valid before
> > >a floating-point conversion specifier:
> > >
> > >L
> > >     Specifies that a following a, A, e, E, f, F, g, or G conversion 
> > >specifier
> > >     applies to an argument with type pointer to long double.
> > >
> > >  from 
> > >http://pubs.opengroup.org/onlinepubs/9699919799/functions/scanf.html
> > >
> > >  So, it is valid for musl not to accept %Lx.
> > >  Now, the argument that it's a good idea to align musl's behaviour to
> > >glibc's whenever possible is a sensible one. But it's a decision for
> > >the musl authors to make, and the pros and cons need to be carefully
> > >balanced; musl's current behaviour is not _incorrect_.  
> > 
> > It is incorrect, because scanf() has to return 0, or it has to handle the
> > L modifier. Currently it doesn't handle L and return 1, so the
> > application can't detect this issue.
> 
> That sounds like a bug in musl libc.
>  
> > I would prefer a case when musl works like glibc, if there are not any
> > reason to not to do that. For example,  now Alpine Linux is very popular
> > and there are a lot of packages. In many cases, a maintainer, who adds a
> > new package, fixes compile-time errors and doesn't run any tests.
> > A target application can work differently with musl comparing with glibc
> > due to this sort of issues.
> 
> FreeBSD man page says:
> 
>      L	      Indicates	that the conversion will be one	of a, e, f, or g and
> 	      the next pointer is a pointer to long double.
> 
> NetBSD man page says:
> 
>      L       Indicates that the conversion will be efg and the next pointer is
>              a pointer to long double.
> 
> OpenBSD man page says:
>      
> L
>     Indicates that the conversion will be one of efg and the next pointer is a pointer to long double.

I have shown the quote from the scanf man page of Alpine Linux,
which is based on musl-libc.

> 
> So the application will break on most (every) system that is not GNU
> libc. It would be better to fix the application in this case:

In our days, a lot of applications are tested only for Linux. In many
cases, they probably can work on any unix system too, but they are not
tested and in many cases this means that they don't work there.

> 
> 
>    char str[] = "sigmask: 0x200";
>    long long mask = 0;
>    int ret;
> 
> #if defined(__GLIBC__)
>    ret = sscanf(str, "sigmask: %Lx", &mask));
> #else
>    ret = sscanf(str, "sigmask: %llx", &mask));
> #endif
>    printf("%d %llx\n", ret, mask);
> 
> 
> 
> Or just use %llx which is POSIX and should work everywhere.

Yes, yes, yes. But I spent about an hour to understand why my application
works incorrectly.

There is another thing, that fprintf(stderr, "smth smth %Lx\n")
doesn't work too, so I saw nothing wrong in a log file.

> 
> That said, those things are tricky to detect at compile time as you
> mentioned and they are tricky to detect with configure scripts that
> works with cross compilation. Also many developers seems to think that
> Linux == glibc so they only read the GNU manuals, so yeah, implement
> glibc behavior here seems like a good idea, unless someone else has a
> brilliant idea how to catch this at compile time.

+1

> 
> In any case, I think the application should be fixed too.

Sure, I already sent a patch. I just want to prevent my situation for
other users.

I like musl-libc, and I think the user experience will be better if it
will have fewer cases when applications work with glibc and don't work
with musl. I'm a developer of the CRIU project and our experience shows
that there is a dozen of issues which has to be fixed to support musl. I
don't think that we are so unprofessional and others don't have similar
problems;).

Thanks,
Andrei

> 
> -nc

Re: Re: [PATCH] scanf: handle the L modifier for integers

[Rich Felker]

On Thu, May 31, 2018 at 10:44:42PM +0200, Natanael Copa wrote:
> On Thu, 31 May 2018 12:00:22 -0700
> Andrei Vagin <avagin@virtuozzo.com> wrote:
> 
> > >>Without this patch, ret will be 1 and mask will be 0. It is obviously
> > >>incorrect. According to the man page, L should work like ll:
> > >>
> > >>L Indicates that the conversion will be either e, f, or g and the
> > >>   next pointer is a pointer to long double or the conversion will
> > >>   be d, i, o, u, or x and the next pointer is a pointer to long
> > >>   long.  
> > >
> > >  This is a GNU extension. POSIX states that L is only valid before
> > >a floating-point conversion specifier:
> > >
> > >L
> > >     Specifies that a following a, A, e, E, f, F, g, or G conversion 
> > >specifier
> > >     applies to an argument with type pointer to long double.
> > >
> > >  from 
> > >http://pubs.opengroup.org/onlinepubs/9699919799/functions/scanf.html
> > >
> > >  So, it is valid for musl not to accept %Lx.
> > >  Now, the argument that it's a good idea to align musl's behaviour to
> > >glibc's whenever possible is a sensible one. But it's a decision for
> > >the musl authors to make, and the pros and cons need to be carefully
> > >balanced; musl's current behaviour is not _incorrect_.  
> > 
> > It is incorrect, because scanf() has to return 0, or it has to handle the
> > L modifier. Currently it doesn't handle L and return 1, so the
> > application can't detect this issue.
> 
> That sounds like a bug in musl libc.
>  
> > I would prefer a case when musl works like glibc, if there are not any
> > reason to not to do that. For example,  now Alpine Linux is very popular
> > and there are a lot of packages. In many cases, a maintainer, who adds a
> > new package, fixes compile-time errors and doesn't run any tests.
> > A target application can work differently with musl comparing with glibc
> > due to this sort of issues.
> 
> FreeBSD man page says:
> 
>      L	      Indicates	that the conversion will be one	of a, e, f, or g and
> 	      the next pointer is a pointer to long double.
> 
> NetBSD man page says:
> 
>      L       Indicates that the conversion will be efg and the next pointer is
>              a pointer to long double.
> 
> OpenBSD man page says:
>      
> L
>     Indicates that the conversion will be one of efg and the next pointer is a pointer to long double.
> 
> So the application will break on most (every) system that is not GNU
> libc. It would be better to fix the application in this case:
> 
> 
>    char str[] = "sigmask: 0x200";
>    long long mask = 0;
>    int ret;
> 
> #if defined(__GLIBC__)
>    ret = sscanf(str, "sigmask: %Lx", &mask));
> #else
>    ret = sscanf(str, "sigmask: %llx", &mask));
> #endif
>    printf("%d %llx\n", ret, mask);
> 
> 
> 
> Or just use %llx which is POSIX and should work everywhere.

Indeed, there is no reason to use %Lx anywhere. It's simply wrong.

> That said, those things are tricky to detect at compile time as you
> mentioned and they are tricky to detect with configure scripts that
> works with cross compilation.

If gcc does not catch this with -Wformat, it's a gcc bug that we
should report and try to get fixed. It's possible that they're making
an exception for the invalidity of L with integer formats since some
libcs support that, but I don't see any good reason for this; gcc
should still be warning about the incorrect and nonportable usage. I
can't imagine they'd be opposed to a patch to fix it.

> Also many developers seems to think that
> Linux == glibc so they only read the GNU manuals, so yeah, implement
> glibc behavior here seems like a good idea, unless someone else has a
> brilliant idea how to catch this at compile time.

Aside from fixing gcc at compile time, this has come up before (with
regard to printf, not scanf), and my leaning then and now was to
detect the UB at runtime by crashing rather than reporting an error as
we do now, since (1) it's UB, so an application can't reasonably
expect an error, and (2) applications seem to be ignoring errors
anyway.

We should also get the man page fixed. The printf man page is clear
that L with integer specifiers is a nonstandard extension and should
not be used (they're not documented under L, only as a note at the
end) but it seems whoever fixed this overlooked changing scanf at the
same time.

Rich

Re: Re: [PATCH] scanf: handle the L modifier for integers

[Szabolcs Nagy]

* Rich Felker <dalias@libc.org> [2018-05-31 19:44:36 -0400]:
> On Thu, May 31, 2018 at 10:44:42PM +0200, Natanael Copa wrote:
> > Also many developers seems to think that
> > Linux == glibc so they only read the GNU manuals, so yeah, implement
> > glibc behavior here seems like a good idea, unless someone else has a
> > brilliant idea how to catch this at compile time.
> 
> Aside from fixing gcc at compile time, this has come up before (with
> regard to printf, not scanf), and my leaning then and now was to
> detect the UB at runtime by crashing rather than reporting an error as
> we do now, since (1) it's UB, so an application can't reasonably
> expect an error, and (2) applications seem to be ignoring errors
> anyway.
> 
> We should also get the man page fixed. The printf man page is clear
> that L with integer specifiers is a nonstandard extension and should
> not be used (they're not documented under L, only as a note at the
> end) but it seems whoever fixed this overlooked changing scanf at the
> same time.
> 

also note that adding extensions to printf this way can break
forward compatibility, because the standard can introduce %Ld
with a different meaning, this happend before: in glibc scanf
%a was used for 'allocation modifier' then later iso c introduced
it for hex floats, now scanf behaves differently based on CFLAGS
(standard conform mode uses different scanf), this involves hacks
in glibc which nobody wants to repeat so nowadays new extensions
are only added once they are expected to be standardized.

(if musl aimed for full glibc compatibility then it would have
to copy the messy %a behaviour too, fortunately that's not in
widespread use just like the %Ld extension..)

Re: Re: [PATCH] scanf: handle the L modifier for integers

[Rich Felker]

On Fri, Jun 01, 2018 at 02:30:50AM +0200, Szabolcs Nagy wrote:
> * Rich Felker <dalias@libc.org> [2018-05-31 19:44:36 -0400]:
> > On Thu, May 31, 2018 at 10:44:42PM +0200, Natanael Copa wrote:
> > > Also many developers seems to think that
> > > Linux == glibc so they only read the GNU manuals, so yeah, implement
> > > glibc behavior here seems like a good idea, unless someone else has a
> > > brilliant idea how to catch this at compile time.
> > 
> > Aside from fixing gcc at compile time, this has come up before (with
> > regard to printf, not scanf), and my leaning then and now was to
> > detect the UB at runtime by crashing rather than reporting an error as
> > we do now, since (1) it's UB, so an application can't reasonably
> > expect an error, and (2) applications seem to be ignoring errors
> > anyway.
> > 
> > We should also get the man page fixed. The printf man page is clear
> > that L with integer specifiers is a nonstandard extension and should
> > not be used (they're not documented under L, only as a note at the
> > end) but it seems whoever fixed this overlooked changing scanf at the
> > same time.
> > 
> 
> also note that adding extensions to printf this way can break
> forward compatibility, because the standard can introduce %Ld
> with a different meaning, this happend before: in glibc scanf
> %a was used for 'allocation modifier' then later iso c introduced
> it for hex floats, now scanf behaves differently based on CFLAGS
> (standard conform mode uses different scanf), this involves hacks
> in glibc which nobody wants to repeat so nowadays new extensions
> are only added once they are expected to be standardized.

Yes. Not implementing nonstandard printf extensions was an intentional
choice, the only exception being %m which POSIX already specifies for
syslog(). The %a mess with scanf is a strong motivation for this
choice.

> (if musl aimed for full glibc compatibility then it would have
> to copy the messy %a behaviour too, fortunately that's not in
> widespread use just like the %Ld extension..)

musl also has general policy regarding inclusion or exclusion of
nonstandard functionality, and printf/scanf extensions fall pretty
strongly under exclude. They're not widely supported on other
implementations, already have portable alternatives, and have no way
to detect whether they're supported and work or not (since there are
no corresponding macros or configure-time symbol tests you could do to
check for them, and even runtime checks would invoke undefined
behavior.

Rich

Re: Re: [PATCH] scanf: handle the L modifier for integers

[Andrei Vagin]

On Thu, May 31, 2018 at 07:44:36PM -0400, Rich Felker wrote:
> On Thu, May 31, 2018 at 10:44:42PM +0200, Natanael Copa wrote:
> > On Thu, 31 May 2018 12:00:22 -0700
> > Andrei Vagin <avagin@virtuozzo.com> wrote:
> > 
> > > >>Without this patch, ret will be 1 and mask will be 0. It is obviously
> > > >>incorrect. According to the man page, L should work like ll:
> > > >>
> > > >>L Indicates that the conversion will be either e, f, or g and the
> > > >>   next pointer is a pointer to long double or the conversion will
> > > >>   be d, i, o, u, or x and the next pointer is a pointer to long
> > > >>   long.  
> > > >
> > > >  This is a GNU extension. POSIX states that L is only valid before
> > > >a floating-point conversion specifier:
> > > >
> > > >L
> > > >     Specifies that a following a, A, e, E, f, F, g, or G conversion 
> > > >specifier
> > > >     applies to an argument with type pointer to long double.
> > > >
> > > >  from 
> > > >http://pubs.opengroup.org/onlinepubs/9699919799/functions/scanf.html
> > > >
> > > >  So, it is valid for musl not to accept %Lx.
> > > >  Now, the argument that it's a good idea to align musl's behaviour to
> > > >glibc's whenever possible is a sensible one. But it's a decision for
> > > >the musl authors to make, and the pros and cons need to be carefully
> > > >balanced; musl's current behaviour is not _incorrect_.  
> > > 
> > > It is incorrect, because scanf() has to return 0, or it has to handle the
> > > L modifier. Currently it doesn't handle L and return 1, so the
> > > application can't detect this issue.
> > 
> > That sounds like a bug in musl libc.
> >  
> > > I would prefer a case when musl works like glibc, if there are not any
> > > reason to not to do that. For example,  now Alpine Linux is very popular
> > > and there are a lot of packages. In many cases, a maintainer, who adds a
> > > new package, fixes compile-time errors and doesn't run any tests.
> > > A target application can work differently with musl comparing with glibc
> > > due to this sort of issues.
> > 
> > FreeBSD man page says:
> > 
> >      L	      Indicates	that the conversion will be one	of a, e, f, or g and
> > 	      the next pointer is a pointer to long double.
> > 
> > NetBSD man page says:
> > 
> >      L       Indicates that the conversion will be efg and the next pointer is
> >              a pointer to long double.
> > 
> > OpenBSD man page says:
> >      
> > L
> >     Indicates that the conversion will be one of efg and the next pointer is a pointer to long double.
> > 
> > So the application will break on most (every) system that is not GNU
> > libc. It would be better to fix the application in this case:
> > 
> > 
> >    char str[] = "sigmask: 0x200";
> >    long long mask = 0;
> >    int ret;
> > 
> > #if defined(__GLIBC__)
> >    ret = sscanf(str, "sigmask: %Lx", &mask));
> > #else
> >    ret = sscanf(str, "sigmask: %llx", &mask));
> > #endif
> >    printf("%d %llx\n", ret, mask);
> > 
> > 
> > 
> > Or just use %llx which is POSIX and should work everywhere.
> 
> Indeed, there is no reason to use %Lx anywhere. It's simply wrong.
> 
> > That said, those things are tricky to detect at compile time as you
> > mentioned and they are tricky to detect with configure scripts that
> > works with cross compilation.
> 
> If gcc does not catch this with -Wformat, it's a gcc bug that we
> should report and try to get fixed. It's possible that they're making
> an exception for the invalidity of L with integer formats since some
> libcs support that, but I don't see any good reason for this; gcc
> should still be warning about the incorrect and nonportable usage. I
> can't imagine they'd be opposed to a patch to fix it.

I found that gcc catches this with -pedantic -Wformat:

/musl # gcc -Wall -pedantic test.c
test.c: In function 'main':
test.c:9:28: warning: ISO C does not support the '%Lx' gnu_scanf format [-Wformat=]
  ret = sscanf(str, "%llx %Lx", &a, &b);

> 
> > Also many developers seems to think that
> > Linux == glibc so they only read the GNU manuals, so yeah, implement
> > glibc behavior here seems like a good idea, unless someone else has a
> > brilliant idea how to catch this at compile time.
> 
> Aside from fixing gcc at compile time, this has come up before (with
> regard to printf, not scanf), and my leaning then and now was to
> detect the UB at runtime by crashing rather than reporting an error as
> we do now, since (1) it's UB, so an application can't reasonably
> expect an error, and (2) applications seem to be ignoring errors
> anyway.
> 
> We should also get the man page fixed. The printf man page is clear
> that L with integer specifiers is a nonstandard extension and should
> not be used (they're not documented under L, only as a note at the
> end) but it seems whoever fixed this overlooked changing scanf at the
> same time.
> 
> Rich