Current status: important changes since 0.9.11

Current status: important changes since 0.9.11

[Rich Felker]

New features:
- Zoneinfo time zone support
- PIE support on arm
- Sub-archs for endian and float abi variants (mipsel, armhf, etc.)
- Better support for non-root installs of dynamic linker
- Make popen/posix_spawn work on qemu app-level emulation
  (working around a difficult-to-fix qemu bug)

Bugs fixed:
- Missing close-on-exec flags (possible fd leaks in threaded programs)
- Failure of make install to install bits headers in some circumstances
- Invalid phdr pointers passed back from dl_iterate_phdr
- Broken static-linked TLS on arm, mips, and powerpc
- Integer overflows in date/time conversion

I'd still like to get a couple more things in before making a release,
especially the symlink direction (this issue still needs discussion)
and Andre's work on ARM-optimized memcpy (should we go with the C
version or the originally-proposed asm?). The CPU affinity stuff is
also still a possible candidate to make it in. Oh, and the new crt1.c
idea could probably go in too, even though it won't be used much yet
except possibly adding PIE support on mips, powerpc, and microblaze.

Rich

Re: Current status: important changes since 0.9.11

[Harald Becker]

Hi Rich !

> Oh, and the new crt1.c idea could probably go in too, even
> though it won't be used much yet except possibly adding PIE
> support on mips, powerpc, and microblaze.

I looked at your new crt1.c and I like it. Why not using it for
main stream purpose if it works for all purposes? Are there any
troubles or caveats except a few bytes more (and some nanoseconds
of extra CPU time on program startup)? This doesn't look so
critical. It seams more to me, you need to look for stuff not
needed to be linked in, especially for carefully written and
statically linked small programs, using only base functions. IMO
there shall be a possibility to get those programs smaller (not
linking in some unused library stuff). The few bytes more of
startup code doesn't really matter for this, and you save a lot
of work, when only using one kind of startup routines.

So I would convert all versions to new startup type, then do
testing and push it in release. With a possible backup directory
(unmaintained) for old startup routines (in case anybody needs).
If no one complains, remove this backup directory in next release
or one after.

Just my cent on this topic. As I'm a programmer who like to
create small statically linked programs.

--
Harald

Re: Current status: important changes since 0.9.11

[Rich Felker]

On Fri, Jul 19, 2013 at 08:39:23PM +0200, Harald Becker wrote:
> Hi Rich !
> 
> > Oh, and the new crt1.c idea could probably go in too, even
> > though it won't be used much yet except possibly adding PIE
> > support on mips, powerpc, and microblaze.
> 
> I looked at your new crt1.c and I like it. Why not using it for
> main stream purpose if it works for all purposes? Are there any

My feeling was just that:

1. We already have "optimized asm" for a few archs, and I didn't see a
   strong argument for removing it.

2. There's a chance to mess up odd arch-specific requirements,
   alignment, etc. in the new asm fragment unless it undergoes some
   careful review.

However I do also agree with you, and think simplicity/consistency
possibly override reason #1 above, and #2 could easily be handled if
some time is put into review and testing of the new code.

Anyone else have opinions on the matter?

Rich

Re: Current status: important changes since 0.9.11

[Harald Becker]

Hi Rich!

19-07-2013 14:53 Rich Felker <dalias@aerifal.cx>:

> On Fri, Jul 19, 2013 at 08:39:23PM +0200, Harald Becker wrote:

> 1. We already have "optimized asm" for a few archs, and I
> didn't see a strong argument for removing it.

The strong argument is single point of failure and equal startup
on all plattforms. If you include the asm versions in main
stream, nobody will use/test the new version on those systems. As
soon as you decide to drop old code you stumble into the trouble,
if such exist. Why not doing a careful and documented break (new
startup) in next release. They should be compatible and tested
for the main lines, so most people will not encounter problems.
And those who catch trouble ... (see below)

> 2. There's a chance to mess up odd arch-specific requirements,
>    alignment, etc. in the new asm fragment unless it undergoes
> some careful review.

Sure. That's the reason why I would include the old asm startup
in the next release or the one there after, as (proposed)
unmaintained code. If one stumbles into a rare corner case, which
may be tracked down to the new startup files, it is possible to
grab the old startup ... but you probably get early response of
this, and may be able to fix new startup to handle things
properly. As long as you get response to startup code, hold old
asm versions in release for fallback usage. The relaese one
version with proposed removal of old startup code, then drop
it ... and you safe a lot of work on different places.

Or do it in this way:

Release one version with new old startup code as main stream and
new one as alternative. Requesting testing, proposing switchover.

If there is no response of startup code failures switch over to
new startup on next release, with fallback to old asm version.

This gives you an extra cycle/slot of testing, but rapid switch
over and saving work for the future.

--
Harald

Re: Current status: important changes since 0.9.11

[Luca Barbato]

On 07/19/2013 08:53 PM, Rich Felker wrote:
> However I do also agree with you, and think simplicity/consistency
> possibly override reason #1 above, and #2 could easily be handled if
> some time is put into review and testing of the new code.
> 
> Anyone else have opinions on the matter?

According to what you said pathological compilers would be the problem here.

Not sure which would be the performance impact of the change on good
compilers though.

The safest route is to keep around the assembly that works or benchmark
and replace if the result doesn't change much.

lu

Re: Current status: important changes since 0.9.11

[Rich Felker]

On Fri, Jul 19, 2013 at 09:49:42PM +0200, Luca Barbato wrote:
> On 07/19/2013 08:53 PM, Rich Felker wrote:
> > However I do also agree with you, and think simplicity/consistency
> > possibly override reason #1 above, and #2 could easily be handled if
> > some time is put into review and testing of the new code.
> > 
> > Anyone else have opinions on the matter?
> 
> According to what you said pathological compilers would be the problem here.

Which comment are you referring to?

> Not sure which would be the performance impact of the change on good
> compilers though.

This is code that runs once at startup and has no loops. There's
really no way for it to be slow. The only issues are size and
correctness.

Rich

Re: Current status: important changes since 0.9.11

[Luca Barbato]

On 07/19/2013 09:54 PM, Rich Felker wrote:
> On Fri, Jul 19, 2013 at 09:49:42PM +0200, Luca Barbato wrote:
>> On 07/19/2013 08:53 PM, Rich Felker wrote:
>>> However I do also agree with you, and think simplicity/consistency
>>> possibly override reason #1 above, and #2 could easily be handled if
>>> some time is put into review and testing of the new code.
>>>
>>> Anyone else have opinions on the matter?
>>
>> According to what you said pathological compilers would be the problem here.
> 
> Which comment are you referring to?

I could be wrong and it wasn't from you. Anyway, I still consider
supporting pathological compilers (that botch the usage of inline asm
badly) the only reason to use full-asm.

> This is code that runs once at startup and has no loops. There's
> really no way for it to be slow. The only issues are size and
> correctness.

We have many real life situations in which we spawn many processes in a
loop. Still I doubt it would be an issue.

lu

Re: Current status: important changes since 0.9.11

[Rich Felker]

On Fri, Jul 19, 2013 at 10:19:11PM +0200, Luca Barbato wrote:
> On 07/19/2013 09:54 PM, Rich Felker wrote:
> > On Fri, Jul 19, 2013 at 09:49:42PM +0200, Luca Barbato wrote:
> >> On 07/19/2013 08:53 PM, Rich Felker wrote:
> >>> However I do also agree with you, and think simplicity/consistency
> >>> possibly override reason #1 above, and #2 could easily be handled if
> >>> some time is put into review and testing of the new code.
> >>>
> >>> Anyone else have opinions on the matter?
> >>
> >> According to what you said pathological compilers would be the problem here.
> > 
> > Which comment are you referring to?
> 
> I could be wrong and it wasn't from you. Anyway, I still consider
> supporting pathological compilers (that botch the usage of inline asm
> badly) the only reason to use full-asm.

One could always pre-generate the asm using GCC or another compiler
that can handle it. Actually even if we wanted to keep using per-arch
hand-written asm, generating the initial draft of the asm for a new
arch based on the C with inline asm would be a good idea..

> > This is code that runs once at startup and has no loops. There's
> > really no way for it to be slow. The only issues are size and
> > correctness.
> 
> We have many real life situations in which we spawn many processes in a
> loop. Still I doubt it would be an issue.

Indeed, this code takes about 1/100 of one percent of the time spent
on exec... :)

Rich

Re: Current status: important changes since 0.9.11

[Luca Barbato]

On 07/19/2013 10:26 PM, Rich Felker wrote:
> On Fri, Jul 19, 2013 at 10:19:11PM +0200, Luca Barbato wrote:
>> On 07/19/2013 09:54 PM, Rich Felker wrote:
>>> On Fri, Jul 19, 2013 at 09:49:42PM +0200, Luca Barbato wrote:
>>>> On 07/19/2013 08:53 PM, Rich Felker wrote:
>>>>> However I do also agree with you, and think simplicity/consistency
>>>>> possibly override reason #1 above, and #2 could easily be handled if
>>>>> some time is put into review and testing of the new code.
>>>>>
>>>>> Anyone else have opinions on the matter?
>>>>
>>>> According to what you said pathological compilers would be the problem here.
>>>
>>> Which comment are you referring to?
>>
>> I could be wrong and it wasn't from you. Anyway, I still consider
>> supporting pathological compilers (that botch the usage of inline asm
>> badly) the only reason to use full-asm.
> 
> One could always pre-generate the asm using GCC or another compiler
> that can handle it. Actually even if we wanted to keep using per-arch
> hand-written asm, generating the initial draft of the asm for a new
> arch based on the C with inline asm would be a good idea..

Exactly.

> Indeed, this code takes about 1/100 of one percent of the time spent
> on exec... :)

I guess nobody should be against this change.

lu

Re: Current status: important changes since 0.9.11

[idunham]

On Fri, Jul 19, 2013 at 12:12:34PM -0400, Rich Felker wrote:
> New features:
> - Zoneinfo time zone support
> - PIE support on arm
> - Sub-archs for endian and float abi variants (mipsel, armhf, etc.)
> - Better support for non-root installs of dynamic linker
> - Make popen/posix_spawn work on qemu app-level emulation
>   (working around a difficult-to-fix qemu bug)

Nice.

> 
> Bugs fixed:
> - Missing close-on-exec flags (possible fd leaks in threaded programs)
> - Failure of make install to install bits headers in some circumstances
> - Invalid phdr pointers passed back from dl_iterate_phdr
> - Broken static-linked TLS on arm, mips, and powerpc
> - Integer overflows in date/time conversion
> 
> I'd still like to get a couple more things in before making a release,
> especially the symlink direction (this issue still needs discussion)

Would this work?
$(DESTDIR)$(LDSO_PATHNAME): lib/libc.so
	install -D -m 755 $< $@ && \
	ln -s $(LDSO_PATHNAME) $(DESTDIR)$(libdir)/libc.so || \
	install -D -m 755 $< $(DESTDIR)$(libdir)/libc.so

> and Andre's work on ARM-optimized memcpy (should we go with the C
> version or the originally-proposed asm?). The CPU affinity stuff is
> also still a possible candidate to make it in. Oh, and the new crt1.c
> idea could probably go in too, even though it won't be used much yet
> except possibly adding PIE support on mips, powerpc, and microblaze.
> 
> Rich

Thanks,
Isaac Dunham