Bug in timezone handling (new zonename format like '<+04>-4' )

Bug in timezone handling (new zonename format like '<+04>-4' )

[Hannu Nyman]

The IANA timezone zoneinfo maintainers have decided to introduce new 
timezones in a simplified format with 2016b zoneinfo.
No more invented name strings, but just the time offset inside <...>
This seems to break musl.

http://mm.icann.org/pipermail/tz-announce/2016-March/000036.html
      New zones Europe/Astrakhan and Europe/Ulyanovsk ... Asia/Barnaul ...
      As a trial of a new system that needs less information to be made up, 
the new zones use numeric time zone abbreviations like "+04" instead of 
invented abbreviations like "ASTT".

I have tested with Openwrt trunk using musl (with Linux kernel 4.1) and older 
uClibc-based Openwrt CC15.05.1.
The uClibc-based system uses the new zones ok, but the musl-based build 
behaves badly, as it defaults to GMT and shows the zonename as '+04>-4'. 
(This is when full zoneinfo files are not installed and just the TZ variable 
is used.)

Examples at https://github.com/openwrt/luci/issues/675

(note: Openwrt patches musl to store TZ in /etc/TZ instead of an env 
variable, but that should have no impact in the displayed timezone.)

Example with musl:

old zone is ok:
Europe/Helsinki
root@OpenWrt:~# cat /etc/TZ
EET-2EEST,M3.5.0/3,M10.5.0/4
root@OpenWrt:~# uptime
  11:00:52 up 18:17,  load average: 0.43, 0.13, 0.11
root@OpenWrt:~# date
Wed Mar 30 11:00:55 EEST 2016

"new zone" is handle wrongly:
Europe/Astrakhan
root@OpenWrt:~# cat /etc/TZ
<+04>-4
root@OpenWrt:~# uptime
  08:02:52 up 18:19,  load average: 0.17, 0.18, 0.13
root@OpenWrt:~# date
Wed Mar 30 08:02:59 +04>-4 2016
(That should be Helsinki+1  (= +2 -DST) = 12:02:59)

Re: Bug in timezone handling (new zonename format like '<+04>-4' )

[Rich Felker]

On Thu, Mar 31, 2016 at 06:50:14PM +0300, Hannu Nyman wrote:
> The IANA timezone zoneinfo maintainers have decided to introduce new
> timezones in a simplified format with 2016b zoneinfo.
> No more invented name strings, but just the time offset inside <...>
> This seems to break musl.
> 
> http://mm.icann.org/pipermail/tz-announce/2016-March/000036.html
>      New zones Europe/Astrakhan and Europe/Ulyanovsk ... Asia/Barnaul ...
>      As a trial of a new system that needs less information to be
> made up, the new zones use numeric time zone abbreviations like
> "+04" instead of invented abbreviations like "ASTT".
> 
> I have tested with Openwrt trunk using musl (with Linux kernel 4.1)
> and older uClibc-based Openwrt CC15.05.1.
> The uClibc-based system uses the new zones ok, but the musl-based
> build behaves badly, as it defaults to GMT and shows the zonename as
> '+04>-4'. (This is when full zoneinfo files are not installed and
> just the TZ variable is used.)
> 
> Examples at https://github.com/openwrt/luci/issues/675
> 
> (note: Openwrt patches musl to store TZ in /etc/TZ instead of an env
> variable, but that should have no impact in the displayed timezone.)
> 
> Example with musl:
> 
> old zone is ok:
> Europe/Helsinki
> root@OpenWrt:~# cat /etc/TZ
> EET-2EEST,M3.5.0/3,M10.5.0/4
> root@OpenWrt:~# uptime
>  11:00:52 up 18:17,  load average: 0.43, 0.13, 0.11
> root@OpenWrt:~# date
> Wed Mar 30 11:00:55 EEST 2016
> 
> "new zone" is handle wrongly:
> Europe/Astrakhan
> root@OpenWrt:~# cat /etc/TZ
> <+04>-4
> root@OpenWrt:~# uptime
>  08:02:52 up 18:19,  load average: 0.17, 0.18, 0.13
> root@OpenWrt:~# date
> Wed Mar 30 08:02:59 +04>-4 2016
> (That should be Helsinki+1  (= +2 -DST) = 12:02:59)

I thought this was some nonstandard extension conflicting with
standard handling of the TZ variable, but in fact it's documented in
current POSIX as a quoted form:

http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html#tag_08_03

Does the specification there agree with your expectations of how it
should behave? Do you have any other references I could look at on how
this is supposed to be used?

Rich

Re: Bug in timezone handling (new zonename format like '<+04>-4' )

[Hannu Nyman]

On 31.3.2016 19:06, Rich Felker wrote:
> On Thu, Mar 31, 2016 at 06:50:14PM +0300, Hannu Nyman wrote:
>> The IANA timezone zoneinfo maintainers have decided to introduce new
>> timezones in a simplified format with 2016b zoneinfo.
>> No more invented name strings, but just the time offset inside <...>
>> This seems to break musl.
>>
>> http://mm.icann.org/pipermail/tz-announce/2016-March/000036.html
>>       New zones Europe/Astrakhan and Europe/Ulyanovsk ... Asia/Barnaul ...
>>       As a trial of a new system that needs less information to be
>> made up, the new zones use numeric time zone abbreviations like
>> "+04" instead of invented abbreviations like "ASTT".
>>
>> I have tested with Openwrt trunk using musl (with Linux kernel 4.1)
>> and older uClibc-based Openwrt CC15.05.1.
>> The uClibc-based system uses the new zones ok, but the musl-based
>> build behaves badly, as it defaults to GMT and shows the zonename as
>> '+04>-4'. (This is when full zoneinfo files are not installed and
>> just the TZ variable is used.)
>>
>> Examples at https://github.com/openwrt/luci/issues/675
>>
>> (note: Openwrt patches musl to store TZ in /etc/TZ instead of an env
>> variable, but that should have no impact in the displayed timezone.)
>>
>> ...
>>
>> "new zone" is handle wrongly:
>> Europe/Astrakhan
>> root@OpenWrt:~# cat /etc/TZ
>> <+04>-4
>> root@OpenWrt:~# uptime
>>   08:02:52 up 18:19,  load average: 0.17, 0.18, 0.13
>> root@OpenWrt:~# date
>> Wed Mar 30 08:02:59 +04>-4 2016
>> (That should be Helsinki+1  (= +2 -DST) = 12:02:59)
> I thought this was some nonstandard extension conflicting with
> standard handling of the TZ variable, but in fact it's documented in
> current POSIX as a quoted form:
>
> http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html#tag_08_03
>
> Does the specification there agree with your expectations of how it
> should behave? Do you have any other references I could look at on how
> this is supposed to be used?
>
> Rich

As far as I can see, the new timezones are according to the POSIX specs:
just the "quoted std + offset"

And musl has some handling for the "quoted form", e.g. in the getname function:
http://git.musl-libc.org/cgit/musl/tree/src/time/__tz.c#n82

But apparently not all functions utilize that, or something else goes wrong.

Sadly I can't offer you more examples. I just thought to highlight the issue 
after noticing it and determined it to be likely a musl issue.

My use case is with Openwrt+musl, which in practice means busybox utils 
combined with musl. As the older Openwrt CC15.05.1 works ok (with 
busybox+uClibc), I think that this is musl-specific. I haven't checked from 
busybox sources how e.g. "uptime" gets exactly handled and which musl 
functions it calls. But almost the same busybox version behaves ok with 
uClibc but fails with musl.

So far this issue only concerns three small timezones in Russia, but is all 
new zones get introduced this way, the issue may grow in importance.

Re: Bug in timezone handling (new zonename format like '<+04>-4' )

[Hannu Nyman]

Reference to earlier discussion in March 2016:
http://www.openwall.com/lists/musl/2016/03/31/10

I think that I have finally found the fix for the bug in handling <quoted> 
timezone names.

musl fails to parse timezone string if the zone name is defined in the quoted 
form with <>.
Example:
root@...nWrt:~# cat /etc/TZ
<+04>-4
root@...nWrt:~# date
Wed Mar 30 08:02:59 +04>-4 2016

(note that Openwrt/LEDE patches musl to use /etc/TZ instead of an environment 
variable, but that has no effect on the evaluation itself)

All new timezones & changes to old ones since zoneinfo 2016b seem to adopt 
the new way, and old zone name abbreviations (like EET) are being gradually 
removed. Currently already 47 timezones have numeric names inside < >. 
Examples of new timezone strings (after 2016g):
'Antarctica/Troll', '<+00>0<+02>-2,M3.5.0/1,M10.5.0/3'
'Asia/Baku', '<+04>-4'
'Europe/Istanbul', '<+03>-3'
'Europe/Minsk', '<+03>-3'

I think that I have now found the bug in musl and fixed it.

The core reason is a fault in the logic for quoted timezone names in function 
"getname" in __tz.c. The name string evaluation loop forgets to push the 
pointer forward to get the next character. The pointer "*p" is explicitly 
moved one char forward to skip the "<" (and afterwards for ">"), but during 
the actual name evaluation loop the same "*p" is used until "i" reaches 
TZNAME_MAX and breaks the loop. After the loop, at the end of the function 
"*p" is increased by the (wrongly detected) length. _POSIX_TZNAME_MAX=6, so 
the name in the previous example is thought to be "+04>-4" as the whole loop 
evaluates against the first char "+" and does not notice the name end marker ">".

http://git.musl-libc.org/cgit/musl/tree/src/time/__tz.c#n87

```
     if (**p == '<') {
         ++*p;
         for (i=0; **p!='>' && i<TZNAME_MAX; i++) <---- culprit: **p remains 
constant
             d[i] = (*p)[i];
         ++*p;
     } else {
         for (i=0; ((*p)[i]|32)-'a'<26U && i<TZNAME_MAX; i++)
             d[i] = (*p)[i];
     }
     *p += i;
     d[i] = 0;
```

Function "do_tzset" uses "getname" to both find the string and to move the 
pointer "s" forward. As the end of the timezone name is evaluated wrongly, 
the offset calculation & DST detection will also fail, as they start from the 
wrong place.

http://git.musl-libc.org/cgit/musl/tree/src/time/__tz.c#n219

```
     getname(std_name, &s);
     __tzname[0] = std_name;
     __timezone = getoff(&s);
     getname(dst_name, &s);
     __tzname[1] = dst_name;
```

I patched "getname" by replacing **p with (*p)[i] to make it to evaluate the 
correct character:

```
--- a/src/time/__tz.c
+++ b/src/time/__tz.c
@@ -87,7 +87,7 @@
      int i;
      if (**p == '<') {
          ++*p;
-        for (i=0; **p!='>' && i<TZNAME_MAX; i++)
+        for (i=0; (*p)[i]!='>' && i<TZNAME_MAX; i++)
              d[i] = (*p)[i];
          ++*p;
      } else {

```

I am not sure if the fix is optimal, but it seems to work.

After that change I am finally able to select timezones with quoted names and 
see correct times:

```
root@lede:~# cat /etc/TZ ; date
EET-2EEST,M3.5.0/3,M10.5.0/4
Tue Oct 18 21:46:41 EEST 2016

root@lede:~# echo "UTC" > /etc/TZ
root@lede:~# cat /etc/TZ ; date
UTC
Tue Oct 18 18:46:53 UTC 2016

root@lede:~# echo "<+04>-4" > /etc/TZ
root@lede:~# cat /etc/TZ ; date
<+04>-4
Tue Oct 18 22:47:17 +04 2016

root@lede:~# echo "<+00>0<+02>-2,M3.5.0/1,M10.5.0/3" > /etc/TZ
root@lede:~# cat /etc/TZ ; date
<+00>0<+02>-2,M3.5.0/1,M10.5.0/3
Tue Oct 18 20:48:10 +02 2016

root@lede:~# echo "<-03>3" > /etc/TZ
root@lede:~# cat /etc/TZ ; date
<-03>3
Tue Oct 18 15:48:41 -03 2016
```