Pending patches/issues before 0.9.15 release?

Pending patches/issues before 0.9.15 release?

[Rich Felker]

Hi all,

I know the next release is behind schedule, and I'm trying to get it
ready soon. Please let me know any pending patches or issues that
haven't been addressed which I should review/commit before release.
Below are some I'm already aware of.

Rich



From Timo Teräs/fabled/Alpine:

http://git.alpinelinux.org/cgit/aports/plain/main/musl/1001-add-rfc3678-mcast-structs.patch
http://git.alpinelinux.org/cgit/aports/plain/main/musl/1002-add-linux-tcp-state-enums.patch
http://git.alpinelinux.org/cgit/aports/plain/main/musl/1003-add-basic-dns-record-parsing-functions.patch

From Michael Forney:

- simplify strcmp
- add putspent
- fix putgrent (missing newline, error handling)
- dn_expand fix
- dn_comp stub
- getopt_long abbreviated options

From Paul Schutte:

- adding version reporting to ld.so (and elsewhere?)

From nsz's review of pkgsrc failures:

- getloadavg
- NO_ADDRESS in netdb.h
- c_ispeed and c_ospeed in termios
- termio.h
- net/if_packet.h
- ffsl
- sys/perm.h

(Note that this is just a selection of topics from the thread which
looked like they were easy to address on the musl side and which might
improve practical compatibility, but it's not 100% clear to me that
they should all be added.)

And some misc stuff:

- Revisiting NULL definition

Re: Pending patches/issues before 0.9.15 release?

[Laurent Bercot]

  Hi Rich,

  I think there is a localtime bug related to
  http://www.openwall.com/lists/musl/2012/10/26/11
but I've been unable to expose it in a reproducible way, and I
haven't had time to investigate more.
  But *once*, I got the following results :

$ TZ=CEST ./localtime-musl ; TZ=CEST ./localtime-glibc
Local:  06:13:34
Global: 06:13:34 TUU
Local:  06:13:34
Global: 06:13:34

$ TZ=:right/Europe/Paris ./localtime-musl ; TZ=:right/Europe/Paris ./localtime-glibc
Local:  08:14:10
Global: 06:14:10 TUU
Local:  08:13:45
Global: 06:13:45

  1. That "TUU" looks like some random string is added after the time format, there
seems to be a string length error.
  2. There's a 35 second discrepancy between the glibc and musl results in the
right/ case, which seems to indicate that musl prints UTC when it should print
TAI-10. (This is reproducible.) Does musl support the A. D. Olson "right/" timezones ?

  Thanks,

-- 
  Laurent

Re: Pending patches/issues before 0.9.15 release?

[Rich Felker]

On Thu, Nov 21, 2013 at 09:40:57AM +0000, Laurent Bercot wrote:
>  Hi Rich,
> 
>  I think there is a localtime bug related to
>  http://www.openwall.com/lists/musl/2012/10/26/11
> but I've been unable to expose it in a reproducible way, and I
> haven't had time to investigate more.

Are you using the same source code from that email?

>  But *once*, I got the following results :
> 
> $ TZ=CEST ./localtime-musl ; TZ=CEST ./localtime-glibc
> Local:  06:13:34
> Global: 06:13:34 TUU
> Local:  06:13:34
> Global: 06:13:34
> 
> $ TZ=:right/Europe/Paris ./localtime-musl ; TZ=:right/Europe/Paris ./localtime-glibc
> Local:  08:14:10
> Global: 06:14:10 TUU
> Local:  08:13:45
> Global: 06:13:45
> 
>  1. That "TUU" looks like some random string is added after the time format, there
> seems to be a string length error.

Perhaps strftime is failing to null-terminate under some conditions.
Did you try pre-filling the buffer with junk to see if it makes the
bug reproducable?

>  2. There's a 35 second discrepancy between the glibc and musl results in the
> right/ case, which seems to indicate that musl prints UTC when it should print
> TAI-10. (This is reproducible.) Does musl support the A. D. Olson "right/" timezones ?

I'm not familiar with the "right/" timezones and they're difficult to
google for -- please give me a reference for what they are, if you
can. However, musl only uses zoneinfo for timezone/daylight-time
purposes, not for any sort of adjustment between POSIX/calendar time
and UTC(w/leap-seconds) and/or TAI. Perhaps this is the source of the
discrepency.

Rich

Re: Pending patches/issues before 0.9.15 release?

[Szabolcs Nagy]

* Rich Felker <dalias@aerifal.cx> [2013-11-21 11:39:11 -0500]:
> I'm not familiar with the "right/" timezones and they're difficult to
> google for -- please give me a reference for what they are, if you
> can. However, musl only uses zoneinfo for timezone/daylight-time
> purposes, not for any sort of adjustment between POSIX/calendar time
> and UTC(w/leap-seconds) and/or TAI. Perhaps this is the source of the
> discrepency.

yes it seems that's the difference, from the tzcode makefile:

# If you always want time values interpreted as "seconds since the epoch
# (not counting leap seconds)", use
#       REDO=           posix_only
# below.  If you always want right time values interpreted as "seconds since
# the epoch" (counting leap seconds)", use
#       REDO=           right_only
# below.  If you want both sets of data available, with leap seconds not
# counted normally, use
#       REDO=           posix_right
# below.  If you want both sets of data available, with leap seconds counted
# normally, use
#       REDO=           right_posix
# below.
# POSIX mandates that leap seconds not be counted; for compatibility with it,
# use either "posix_only" or "posix_right".

REDO=           posix_right

Re: Pending patches/issues before 0.9.15 release?

[John Spencer]

Rich Felker wrote:
> Hi all,
> 
> I know the next release is behind schedule, and I'm trying to get it
> ready soon. Please let me know any pending patches or issues that
> haven't been addressed which I should review/commit before release.
> Below are some I'm already aware of.

Since the release is way overdue, i personally would prefer if we only 
deal with bugfixes and postpone any additions till 1.0 (or maybe add 
some of them to an additional 0.9.16 release), so we can get the release 
out ASAP.

so the only things that would qualify are:

- fix putgrent (missing newline, error handling)
- dn_expand fix

Re: Pending patches/issues before 0.9.15 release?

[Laurent Bercot]

> Perhaps strftime is failing to null-terminate under some conditions.
> Did you try pre-filling the buffer with junk to see if it makes the
> bug reproducable?

  I didn't investigate or experiment more, but I will as soon as I get
some time - maybe this week-end.


> I'm not familiar with the "right/" timezones and they're difficult to
> google for -- please give me a reference for what they are, if you
> can. However, musl only uses zoneinfo for timezone/daylight-time
> purposes, not for any sort of adjustment between POSIX/calendar time
> and UTC(w/leap-seconds) and/or TAI. Perhaps this is the source of the
> discrepency.

  The short version of what right/ timezones are is explained here:
  http://www.madore.org/~david/computers/unix-leap-seconds.html#tai-minus-10

  The difference between glibc and musl is 25 seconds, not 35 as I wrote,
which seems to indicate that musl's gmtime()/localtime() does not take leap
second tables, which are defined in right/ timezones, into account.
  I would really love that to be fixed - maybe not for 0.9.15, but for 0.9.16
if possible, because I'm a hardcore TAI-10 user.

-- 
  Laurent

Re: Pending patches/issues before 0.9.15 release?

[Rich Felker]

On Thu, Nov 21, 2013 at 07:27:53PM +0000, Laurent Bercot wrote:
> >I'm not familiar with the "right/" timezones and they're difficult to
> >google for -- please give me a reference for what they are, if you
> >can. However, musl only uses zoneinfo for timezone/daylight-time
> >purposes, not for any sort of adjustment between POSIX/calendar time
> >and UTC(w/leap-seconds) and/or TAI. Perhaps this is the source of the
> >discrepency.
> 
>  The short version of what right/ timezones are is explained here:
>  http://www.madore.org/~david/computers/unix-leap-seconds.html#tai-minus-10

Thanks for the info.

>  The difference between glibc and musl is 25 seconds, not 35 as I wrote,
> which seems to indicate that musl's gmtime()/localtime() does not take leap
> second tables, which are defined in right/ timezones, into account.
>  I would really love that to be fixed - maybe not for 0.9.15, but for 0.9.16
> if possible, because I'm a hardcore TAI-10 user.

I'm a pretty hardcore anti-leapseconds / pro-POSIX-seconds guy, so if
you'd like to make any progress here, maybe you could do some research
into whether/how the features you want could be supported
non-invasively.

Rich

Re: Pending patches/issues before 0.9.15 release?

[Daniel Cegiełka]

2013/11/21 Rich Felker <dalias@aerifal.cx>:

> I'm a pretty hardcore anti-leapseconds / pro-POSIX-seconds guy,

I think the big problem is the lack of proper approach to leapseconds
and unfortunately it ends this way:

http://www.wired.com/wiredenterprise/2012/07/leap-second-glitch-explained/

25 seconds is a lot :/

Daniel

Re: Pending patches/issues before 0.9.15 release?

[Rich Felker]

On Thu, Nov 21, 2013 at 09:07:08PM +0100, Daniel Cegiełka wrote:
> 2013/11/21 Rich Felker <dalias@aerifal.cx>:
> 
> > I'm a pretty hardcore anti-leapseconds / pro-POSIX-seconds guy,
> 
> I think the big problem is the lack of proper approach to leapseconds
> and unfortunately it ends this way:
> 
> http://www.wired.com/wiredenterprise/2012/07/leap-second-glitch-explained/

A glitch which would never have happened without the abominations
which are leap seconds. Really, the core cause was the NTP folks'
agenda in forcing leap seconds onto a POSIX time model where they
don't work, resulting in discontinuous, non-monotonic clock jumps and
time ambiguity. If they'd instead smoothed out the difference between
POSIX seconds and SI seconds over the whole interval between leap
seconds as a tiny clock drift, reliable conversion back to TAI seconds
could be performed as a presentation step without breaking software
doing (valid, per POSIX) interval timing with time_t and
timeval/timespec.

> 25 seconds is a lot :/

25 seconds is a lot in one day. It's not a lot in 43 years. It's much
smaller than the error tolerance for audio and video timing, and
unlikely to matter for anything except physics experiments.

Rich

Re: Pending patches/issues before 0.9.15 release?

[Laurent Bercot]

> A glitch which would never have happened without the abominations
> which are leap seconds. Really, the core cause was the NTP folks'
> agenda in forcing leap seconds onto a POSIX time model where they
> don't work, resulting in discontinuous, non-monotonic clock jumps and
> time ambiguity.

  Looks like we totally agree on this :)


> If they'd instead smoothed out the difference between
> POSIX seconds and SI seconds over the whole interval between leap
> seconds as a tiny clock drift, reliable conversion back to TAI seconds
> could be performed as a presentation step without breaking software
> doing (valid, per POSIX) interval timing with time_t and
> timeval/timespec.

  ... but not on the conclusion. My own conclusion is that a linear
time scale should have been adopted for system clocks, and conversions
from/to UTC should be an entirely userland, only-for-display thing.
System operations should be able to depend on linear time, and only
humans should care about human-oriented time such as UTC. This is why
I always set up my system clock as TAI-10.

-- 
  Laurent

Re: Pending patches/issues before 0.9.15 release?

[Szabolcs Nagy]

* Laurent Bercot <ska-dietlibc@skarnet.org> [2013-11-21 19:27:53 +0000]:
>  The short version of what right/ timezones are is explained here:
>  http://www.madore.org/~david/computers/unix-leap-seconds.html#tai-minus-10

according to the official tzdata code the "right/" directory
is obsolete and different path is used for this now

"The advantage of this scheme is that TAI−10 uniquely refers
to an instant in time, and the difference between two such
values determines the interval between these instants"

is not true, a single time coordinate cannot uniquely refer to
an instant in time (TAI only specifies the frame of reference
since 1977, before that it was largely nonsense and after that
it only measures the time somewhat accurately for observers at
sea level in rest, but eg outside the gravity well of earth it
is off by 22ms/year)

even in classical physics, a clock source with precise ticks is
not usable for representing dates because the rotation of the
earth is not exactly periodic (by which dates are defined)

so using the absolute time coordinate of an observational frame
of reference for date keeping seems to be rather futile

>  The difference between glibc and musl is 25 seconds, not 35 as I wrote,
> which seems to indicate that musl's gmtime()/localtime() does not take leap
> second tables, which are defined in right/ timezones, into account.
>  I would really love that to be fixed - maybe not for 0.9.15, but for 0.9.16
> if possible, because I'm a hardcore TAI-10 user.

knowing that TAI is non-conforming you need to provide a strong
use-case for it

Re: Pending patches/issues before 0.9.15 release?

[Laurent Bercot]

> according to the official tzdata code the "right/" directory
> is obsolete and different path is used for this now

  Oh ? I will definitely investigate this more.


> "The advantage of this scheme is that TAI−10 uniquely refers
> to an instant in time, and the difference between two such
> values determines the interval between these instants"
>
> is not true, a single time coordinate cannot uniquely refer to
> an instant in time (TAI only specifies the frame of reference
> since 1977, before that it was largely nonsense and after that
> it only measures the time somewhat accurately for observers at
> sea level in rest, but eg outside the gravity well of earth it
> is off by 22ms/year)

  Until we can go back to before 1977 or have computers outside the
gravity well of Earth, this is not what matters. What matters is that
we need a linear measure of time to synchronize our system clocks on,
and TAI is exactly this.


> knowing that TAI is non-conforming you need to provide a strong
> use-case for it

  I believe that Linux kernels crashing all over the world because of
a leap second happening makes a pretty strong case for not having the
system clock jump around. There are several solutions for that, but
none of them is simpler than just ignoring the drift at a system level
and perform conversions in userland when needed.
  I'll follow Rich's advice and try to provide a musl patch that's
minimally invasive, when I've found the right way to proceed with
timezones.

-- 
  Laurent

Re: Pending patches/issues before 0.9.15 release?

[Rich Felker]

On Thu, Nov 21, 2013 at 11:54:41PM +0100, Szabolcs Nagy wrote:
> * Laurent Bercot <ska-dietlibc@skarnet.org> [2013-11-21 19:27:53 +0000]:
> >  The short version of what right/ timezones are is explained here:
> >  http://www.madore.org/~david/computers/unix-leap-seconds.html#tai-minus-10
> 
> according to the official tzdata code the "right/" directory
> is obsolete and different path is used for this now
> 
> "The advantage of this scheme is that TAI−10 uniquely refers
> to an instant in time, and the difference between two such
> values determines the interval between these instants"
> 
> is not true, a single time coordinate cannot uniquely refer to
> an instant in time (TAI only specifies the frame of reference
> since 1977, before that it was largely nonsense and after that
> it only measures the time somewhat accurately for observers at
> sea level in rest, but eg outside the gravity well of earth it
> is off by 22ms/year)

Indeed. Relativity is the main reason I find the argument for using SI
seconds as a calendar unit unconvincing. SI seconds are a sort of
"local coordinate system" in a particular inertial frame of reference,
whereas calendar time is a shared reference for coordinating human
(and computer) activity. In the latter, "days" are the basic
observable unit, and seconds are just a particular subdivision of
days. Having the number of seconds a day is divided into vary in a way
that *seems* arbitrary to 99.99999% of users of the system is not a
desirable property from my perspective.

> even in classical physics, a clock source with precise ticks is
> not usable for representing dates because the rotation of the
> earth is not exactly periodic (by which dates are defined)

My understanding is that TAI ignores this. The idea is to measure a
fixed physical time quantity that's not related to the human calendar
except for being closely approximated by the calendar second.

> >  The difference between glibc and musl is 25 seconds, not 35 as I wrote,
> > which seems to indicate that musl's gmtime()/localtime() does not take leap
> > second tables, which are defined in right/ timezones, into account.
> >  I would really love that to be fixed - maybe not for 0.9.15, but for 0.9.16
> > if possible, because I'm a hardcore TAI-10 user.
> 
> knowing that TAI is non-conforming you need to provide a strong
> use-case for it

There seems to be some kernel-space effort for handling TAI, but I
have no idea if it's sane or not. Ideally the kernel would be able to
keep it while exposing smooth POSIX time to userspace via the default
clocks and TAI via the non-POSIX extension TAI clocks.

Rich

Re: Pending patches/issues before 0.9.15 release?

[Szabolcs Nagy]

* Laurent Bercot <ska-dietlibc@skarnet.org> [2013-11-21 23:04:52 +0000]:
> >according to the official tzdata code the "right/" directory
> >is obsolete and different path is used for this now
> 
>  Oh ? I will definitely investigate this more.
> 

check the commens in the makefile
(i dont know what distributions do, but it seems they now have
a zoneinfo-posix and zoneinfo-leap and you have to symlink one
or the other to the zoneinfo dir)

> >knowing that TAI is non-conforming you need to provide a strong
> >use-case for it
> 
>  I believe that Linux kernels crashing all over the world because of
> a leap second happening makes a pretty strong case for not having the

leap second is non-conforming as well

> system clock jump around. There are several solutions for that, but
> none of them is simpler than just ignoring the drift at a system level
> and perform conversions in userland when needed.

you are welcome to try that but my guess is..

synchronizing clocks according to TAI accurately is
probably easier than posix time, especially since GPS
time uses the same reference (with some offset for
whatever reason) and that's widely available

but for "performing the conversion when needed" would
require an extra source of data about the rotation of the
earth (otherwise the date would drift away from reality)
and scale the TAI seconds to earth seconds on a given day
before display.. (so you still need to sync to the
rotation of the earth, but you made date calculations
significantly harder)

Re: Pending patches/issues before 0.9.15 release?

[Rich Felker]

Some updates:

On Wed, Nov 20, 2013 at 08:30:56PM -0500, Rich Felker wrote:
> >From Timo Teräs/fabled/Alpine:
> 
> http://git.alpinelinux.org/cgit/aports/plain/main/musl/1001-add-rfc3678-mcast-structs.patch
> http://git.alpinelinux.org/cgit/aports/plain/main/musl/1002-add-linux-tcp-state-enums.patch
> http://git.alpinelinux.org/cgit/aports/plain/main/musl/1003-add-basic-dns-record-parsing-functions.patch

1001 and 1002 have been committed. 1003 should probably be reviewed
and refactored along with the resolver refactor/overhaul after 1.0.

> >From Michael Forney:
> 
> - simplify strcmp
> - add putspent
> - fix putgrent (missing newline, error handling)
> - dn_expand fix
> - dn_comp stub
> - getopt_long abbreviated options

I'm waiting for minor patch updates for putspent and dn_comp.
getopt_long still needs review. I think the rest have been committed.
There's also a new fgetspent patch that I hope to get included.

> >From Paul Schutte:
> 
> - adding version reporting to ld.so (and elsewhere?)

Still under discussion how best to do it.

> >From nsz's review of pkgsrc failures:
> 
> - getloadavg
> - NO_ADDRESS in netdb.h
> - c_ispeed and c_ospeed in termios
> - termio.h
> - net/if_packet.h
> - ffsl
> - sys/perm.h
> 
> (Note that this is just a selection of topics from the thread which
> looked like they were easy to address on the musl side and which might
> improve practical compatibility, but it's not 100% clear to me that
> they should all be added.)

Some members of the community expressed a sentiment that adding new
things like this should wait until after the release if they're not
critical, even if they're small (like most of the above are).

> And some misc stuff:
> 
> - Revisiting NULL definition

I have not done this yet, but I think it could be done before release
and I'd rather change it now than as a new change in 1.0...

Rich

zoneinfo file parsing in __tz.c

[Laurent Bercot]

  While working on leap seconds support, I find the following line in
do_tzset(), that's supposed to skip the non-supported part of the
zoneinfo binary format:
  size_t skip = zi_dotprod(zi, VEC(1,1,8,5,6,1), 6);

  but zi is the start of the mapped file, and the various numbers
indicating the sections' length are supposed to be at zi+20, not zi.
  Is that a bug or have I missed something ?

-- 
  Laurent

Re: zoneinfo file parsing in __tz.c

[Szabolcs Nagy]

* Laurent Bercot <ska-dietlibc@skarnet.org> [2013-11-24 11:06:16 +0000]:
>  While working on leap seconds support, I find the following line in
> do_tzset(), that's supposed to skip the non-supported part of the
> zoneinfo binary format:
>  size_t skip = zi_dotprod(zi, VEC(1,1,8,5,6,1), 6);
> 
>  but zi is the start of the mapped file, and the various numbers
> indicating the sections' length are supposed to be at zi+20, not zi.
>  Is that a bug or have I missed something ?

i think you are not working with latest git then
this is what it says now:

http://git.musl-libc.org/cgit/musl/tree/src/time/__tz.c#n179