New gcc wrapper to try

New gcc wrapper to try

[Rich Felker]

[-- Attachment #1: Type: text/plain, Size: 466 bytes --]

Hi,
Here's the latest draft of a new concept for the musl-gcc gcc wrapper.
It's almost entirely gcc specfile-based. If it works well, I intend to
move even more of what it does into the specfile, and instead of
generating the specfile at runtime each time it's run, just have it
generated at build time and installed with musl.

Please let me know how it works. (You'll need to edit the first couple
lines to set the paths/arch if you don't use the defaults.)

Rich

[-- Attachment #2: musl-gcc.new --]
[-- Type: text/plain, Size: 728 bytes --]

#!/bin/sh

libc_prefix="/usr/local/musl"
ldso_pathname="/lib/ld-musl-i386.so.1"
gcc=gcc
libc_lib=$libc_prefix/lib
libc_inc=$libc_prefix/include
libc_crt="$libc_lib/crt1.o"
libc_start="$libc_lib/crti.o"
libc_end="$libc_lib/crtn.o"

tmp_specs=$HOME/.specs.tmp.$$
printf '
*link_libgcc:
-L%s

*libgcc:
libgcc.a%%s %%:if-exists(libgcc_eh.a%%s)

*cc1:
%%(cc1_cpu)

*startfile:
%%{!shared: %s} %s crtbegin.o%%s

*endfile:
crtend.o%%s %s

*esp_link:


*esp_options:


*esp_cpp_options:


' \
 "$libc_lib" "$libc_crt" "$libc_start" "$libc_end" > "$tmp_specs" || exit 1

exec 3<"$tmp_specs"
rm -f "$tmp_specs"

exec "$gcc" -specs=/proc/self/fd/3 -nostdinc -isystem "$libc_inc" \
 "$@" -Wl,-dynamic-linker,"$ldso_pathname" -Wl,-nostdlib

Re: New gcc wrapper to try

[Rich Felker]

[-- Attachment #1: Type: text/plain, Size: 128 bytes --]

Here's an updated version which fixes some issues with the previous
(wrong crtbegin/end files when -shared is used, etc.)

Rich

[-- Attachment #2: musl-gcc.new --]
[-- Type: text/plain, Size: 935 bytes --]

#!/bin/sh

libc_prefix="/usr/local/musl"
ldso_pathname="/lib/ld-musl-i386.so.1"
gcc=gcc
libc_lib=$libc_prefix/lib
libc_inc=$libc_prefix/include
libc_crt="$libc_lib/crt1.o"
libc_start="$libc_lib/crti.o"
libc_end="$libc_lib/crtn.o"

tmp_specs=$HOME/.specs.tmp.$$
printf '
%%rename cpp_options old_cpp_options

*cpp_options:
-nostdinc -isystem %s %%(old_cpp_options)

*cc1:
%%(cc1_cpu) -nostdinc -isystem %s 

*link_libgcc:
-L%s

*libgcc:
libgcc.a%%s %%:if-exists(libgcc_eh.a%%s)

*startfile:
%%{!shared: %s} %s %%{shared|pie:crtbeginS.o%%s;:crtbegin.o%%s}

*endfile:
%%{shared|pie:crtendS.o%%s;:crtend.o%%s} %s

%%rename link old_link

*link:
%%(old_link) -dynamic-linker %s -nostdlib

*esp_link:


*esp_options:


*esp_cpp_options:


' \
 "$libc_inc" "$libc_inc" "$libc_lib" "$libc_crt" "$libc_start" "$libc_end" "$ldso_pathname" > "$tmp_specs" || exit 1

exec 3<"$tmp_specs"
rm -f "$tmp_specs"

exec "$gcc" -specs=/proc/self/fd/3 "$@"

Re: New gcc wrapper to try

[Isaac Dunham]

On Sat, 21 Apr 2012 02:49:33 -0400
Rich Felker <dalias@aerifal.cx> wrote:

> Hi,
> Here's the latest draft of a new concept for the musl-gcc gcc wrapper.
> It's almost entirely gcc specfile-based. If it works well, I intend to
> move even more of what it does into the specfile, and instead of
> generating the specfile at runtime each time it's run, just have it
> generated at build time and installed with musl.

> Please let me know how it works. (You'll need to edit the first couple
> lines to set the paths/arch if you don't use the defaults.)

I intend to test it, but have some work to do beforehand.  
I've been using /opt/musl, myself.
By the way, I had an idea for handling musl-gcc -V:
if [ "-V" = "$1" ]
	gcc="gcc -V $2"
	shift 2
fi

-- 
Isaac Dunham <idunham@lavabit.com>

Re: New gcc wrapper to try

[Isaac Dunham]

[-- Attachment #1: Type: text/plain, Size: 544 bytes --]

On Sat, 21 Apr 2012 16:37:51 -0400
Rich Felker <dalias@aerifal.cx> wrote:

> Here's an updated version which fixes some issues with the previous
> (wrong crtbegin/end files when -shared is used, etc.)
> 
Doesn't fix building wireless-tools (I've changed the target directory
to fit my install; no other changes)
In fact, where the old musl-gcc can link the object-file statically,
this one cannot link it at all, under any circumstances.
(This version is saved as muslcc, if you want to examine the logs...)
Using make iwmulticall

Isaac Dunham

[-- Attachment #2: make.muslcc --]
[-- Type: application/octet-stream, Size: 2529 bytes --]

muslcc -Os -W -Wall -Wstrict-prototypes -I. -DIW_USES_ITS_OWN_DOGFOOD -I/opt/musl/linux-headers/include -D_GNU_SOURCE  -MMD     -c iwmulticall.c
In file included from iwmulticall.c:96:
iwgetid.c: In function ‘print_one_device’:
iwlib.c:2161: warning: dereferencing pointer ‘({anonymous})’ does break strict-aliasing rules
iwlib.c:2161: warning: dereferencing pointer ‘({anonymous})’ does break strict-aliasing rules
iwlib.c:2162: warning: dereferencing pointer ‘({anonymous})’ does break strict-aliasing rules
iwlib.c:2162: warning: dereferencing pointer ‘({anonymous})’ does break strict-aliasing rules
iwlib.c:2163: warning: dereferencing pointer ‘({anonymous})’ does break strict-aliasing rules
iwlib.c:2163: warning: dereferencing pointer ‘({anonymous})’ does break strict-aliasing rules
iwgetid.c:178: note: initialized from here
In file included from iwmulticall.c:74:
iwlist.c: In function ‘print_scanning_info’:
iwlib.c:2163: warning: dereferencing pointer ‘({anonymous})’ does break strict-aliasing rules
iwlib.c:2163: warning: dereferencing pointer ‘({anonymous})’ does break strict-aliasing rules
iwlib.c:2162: warning: dereferencing pointer ‘({anonymous})’ does break strict-aliasing rules
iwlib.c:2162: warning: dereferencing pointer ‘({anonymous})’ does break strict-aliasing rules
iwlib.c:2161: warning: dereferencing pointer ‘({anonymous})’ does break strict-aliasing rules
iwlib.c:2161: warning: dereferencing pointer ‘({anonymous})’ does break strict-aliasing rules
iwlib-private.h:121: note: initialized from here
muslcc  -Wl,-s -Os -W -Wall -Wstrict-prototypes -I. -DIW_USES_ITS_OWN_DOGFOOD -I/opt/musl/linux-headers/include -D_GNU_SOURCE  -MMD     -o iwmulticall iwmulticall.o -lm
iwmulticall.o: In function `set_commit_info':
iwmulticall.c:(.text+0x11d): undefined reference to `__stack_chk_fail'
iwmulticall.o: In function `iw_check_if_addr_type':
iwmulticall.c:(.text+0x21a): undefined reference to `__stack_chk_fail'
iwmulticall.o: In function `iw_check_mac_addr_type':
iwmulticall.c:(.text+0x295): undefined reference to `__stack_chk_fail'
iwmulticall.o: In function `print_iface_version_info':
iwmulticall.c:(.text+0x3a0): undefined reference to `__stack_chk_fail'
iwmulticall.o: In function `set_sens_info':
iwmulticall.c:(.text+0x426): undefined reference to `__stack_chk_fail'
iwmulticall.o:iwmulticall.c:(.text+0xcff): more undefined references to `__stack_chk_fail' follow
collect2: ld returned 1 exit status
make: *** [iwmulticall] Error 1

Re: New gcc wrapper to try

[Rich Felker]

On Sun, Apr 22, 2012 at 10:57:04PM -0700, Isaac Dunham wrote:
> On Sat, 21 Apr 2012 16:37:51 -0400
> Rich Felker <dalias@aerifal.cx> wrote:
> 
> > Here's an updated version which fixes some issues with the previous
> > (wrong crtbegin/end files when -shared is used, etc.)
> > 
> Doesn't fix building wireless-tools (I've changed the target directory
> to fit my install; no other changes)
> In fact, where the old musl-gcc can link the object-file statically,
> this one cannot link it at all, under any circumstances.
> (This version is saved as muslcc, if you want to examine the logs...)
> Using make iwmulticall

Somebody has enabled stack protector. Either your distro (what distro
are you using) has hacked gcc in a way different from what musl-gcc
expects, or wireless-tools' makefile is adding -fstack-protector...

Rich

Re: New gcc wrapper to try

[Isaac Dunham]

[-- Attachment #1: Type: text/plain, Size: 1169 bytes --]

On Mon, 23 Apr 2012 05:19:05 -0400
Rich Felker <dalias@aerifal.cx> wrote:

> On Sun, Apr 22, 2012 at 10:57:04PM -0700, Isaac Dunham wrote:
> > On Sat, 21 Apr 2012 16:37:51 -0400
> > Rich Felker <dalias@aerifal.cx> wrote:
> > 
> > > Here's an updated version which fixes some issues with the
> > > previous (wrong crtbegin/end files when -shared is used, etc.)
> > > 
> > Doesn't fix building wireless-tools (I've changed the target
> > directory to fit my install; no other changes)
> > In fact, where the old musl-gcc can link the object-file statically,
> > this one cannot link it at all, under any circumstances.
> > (This version is saved as muslcc, if you want to examine the
> > logs...) Using make iwmulticall
> 
> Somebody has enabled stack protector. Either your distro (what distro
> are you using) has hacked gcc in a way different from what musl-gcc
> expects, or wireless-tools' makefile is adding -fstack-protector...

There's no -fstack-protector, so it's Ubuntu's fault.
Now (as hacked) builds iwmulticall.
Attached is my version, with manual disable for stack protector (and
also -V support, in case I want to use both gcc-4.1 and 4.4 )

Isaac Dunham

[-- Attachment #2: muslcc --]
[-- Type: application/octet-stream, Size: 1011 bytes --]

#!/bin/sh

libc_prefix="/opt/musl"
ldso_pathname="/lib/ld-musl-i386.so.1"
gcc=gcc
libc_lib=$libc_prefix/lib
libc_inc=$libc_prefix/include
libc_crt="$libc_lib/crt1.o"
libc_start="$libc_lib/crti.o"
libc_end="$libc_lib/crtn.o"

if [ "-V" == "$1" ]
	then
		gcc="$gcc $1 $2"
		shift 2
	fi

tmp_specs=$HOME/.specs.tmp.$$
printf '
%%rename cpp_options old_cpp_options

*cpp_options:
-nostdinc -isystem %s %%(old_cpp_options)

*cc1:
%%(cc1_cpu) -nostdinc -isystem %s 

*link_libgcc:
-L%s

*libgcc:
libgcc.a%%s %%:if-exists(libgcc_eh.a%%s)

*startfile:
%%{!shared: %s} %s %%{shared|pie:crtbeginS.o%%s;:crtbegin.o%%s}

*endfile:
%%{shared|pie:crtendS.o%%s;:crtend.o%%s} %s

%%rename link old_link

*link:
%%(old_link) -dynamic-linker %s -nostdlib

*esp_link:


*esp_options:


*esp_cpp_options:


' \
 "$libc_inc" "$libc_inc" "$libc_lib" "$libc_crt" "$libc_start" "$libc_end" "$ldso_pathname" > "$tmp_specs" || exit 1

exec 3<"$tmp_specs"
rm -f "$tmp_specs"

exec "$gcc" -specs=/proc/self/fd/3 -fno-stack-protector "$@"

Re: New gcc wrapper to try

[Isaac Dunham]

[-- Attachment #1: Type: text/plain, Size: 429 bytes --]

On Tue, 24 Apr 2012 12:28:28 -0700
Isaac Dunham <idunham@lavabit.com> wrote:


> Now (as hacked) builds iwmulticall.
> Attached is my version, with manual disable for stack protector (and
> also -V support, in case I want to use both gcc-4.1 and 4.4 )

Sorry, the above version doesn't work properly with -V, due to space
preservation.
Here's a version that builds iwmulticall with CC="muslcc -V 4.1" or
CC=muslcc.

Isaac Dunham

[-- Attachment #2: muslcc --]
[-- Type: application/octet-stream, Size: 1018 bytes --]

#!/bin/sh

libc_prefix="/opt/musl"
ldso_pathname="/lib/ld-musl-i386.so.1"
gcc=gcc
libc_lib=$libc_prefix/lib
libc_inc=$libc_prefix/include
libc_crt="$libc_lib/crt1.o"
libc_start="$libc_lib/crti.o"
libc_end="$libc_lib/crtn.o"

if [ "-V" == "$1" ]
	then
		gccver="$1$2"
		shift 2
	fi

tmp_specs=$HOME/.specs.tmp.$$
printf '
%%rename cpp_options old_cpp_options

*cpp_options:
-nostdinc -isystem %s %%(old_cpp_options)

*cc1:
%%(cc1_cpu) -nostdinc -isystem %s 

*link_libgcc:
-L%s

*libgcc:
libgcc.a%%s %%:if-exists(libgcc_eh.a%%s)

*startfile:
%%{!shared: %s} %s %%{shared|pie:crtbeginS.o%%s;:crtbegin.o%%s}

*endfile:
%%{shared|pie:crtendS.o%%s;:crtend.o%%s} %s

%%rename link old_link

*link:
%%(old_link) -dynamic-linker %s -nostdlib

*esp_link:


*esp_options:


*esp_cpp_options:


' \
 "$libc_inc" "$libc_inc" "$libc_lib" "$libc_crt" "$libc_start" "$libc_end" "$ldso_pathname" > "$tmp_specs" || exit 1

exec 3<"$tmp_specs"
rm -f "$tmp_specs"

exec "$gcc" "$gccver" -specs=/proc/self/fd/3 -fno-stack-protector "$@"

Re: New gcc wrapper to try

[Solar Designer]

Rich, Isaac -

On Tue, Apr 24, 2012 at 12:28:28PM -0700, Isaac Dunham wrote:
> On Mon, 23 Apr 2012 05:19:05 -0400 Rich Felker <dalias@aerifal.cx> wrote:
> > Somebody has enabled stack protector. Either your distro (what distro
> > are you using) has hacked gcc in a way different from what musl-gcc
> > expects, or wireless-tools' makefile is adding -fstack-protector...
> 
> There's no -fstack-protector, so it's Ubuntu's fault.

Several distros do it these days (and Owl will start doing it soon as
well), so it's becoming standard and needs to be supported.
Technically, -fno-stack-protector in the musl-gcc wrapper would probably
make things appear to work, but it's not necessarily a good thing to do
since it might violate reasonable expectations of the user.  I think
it'd be better for musl to start providing the necessary symbols.

Alexander

Re: New gcc wrapper to try

[Rich Felker]

On Wed, Apr 25, 2012 at 01:28:40AM +0400, Solar Designer wrote:
> Rich, Isaac -
> 
> On Tue, Apr 24, 2012 at 12:28:28PM -0700, Isaac Dunham wrote:
> > On Mon, 23 Apr 2012 05:19:05 -0400 Rich Felker <dalias@aerifal.cx> wrote:
> > > Somebody has enabled stack protector. Either your distro (what distro
> > > are you using) has hacked gcc in a way different from what musl-gcc
> > > expects, or wireless-tools' makefile is adding -fstack-protector...
> > 
> > There's no -fstack-protector, so it's Ubuntu's fault.
> 
> Several distros do it these days (and Owl will start doing it soon as
> well), so it's becoming standard and needs to be supported.
> Technically, -fno-stack-protector in the musl-gcc wrapper would probably
> make things appear to work, but it's not necessarily a good thing to do
> since it might violate reasonable expectations of the user.  I think
> it'd be better for musl to start providing the necessary symbols.

The problem is that the default gcc build incorporates glibc ABI
knowledge (layout of the thread structure) and the assumption that the
thread pointer has been initialized into any binary built with stack
protector. Just adding symbols will not fix anything.

If gcc's stack protector support is built without TLS support so that
it uses a global variable for the canary, instead of a thread-local
variable, then we could support it with no work at all. Unfortunately
I don't believe there's any way to override the type of canary access
gcc does via command line options; you have to rebuild gcc. Thus, for
the time being (until/unless we do TLS and mimic glibc's structure
layout), disabling stack protector is the only option for the wrapper
to use. For our own native gcc builds, we can already support it if I
just add the symbol it needs.

Rich

Re: New gcc wrapper to try

[Rich Felker]

On Tue, Apr 24, 2012 at 05:41:27PM -0400, Rich Felker wrote:
> On Wed, Apr 25, 2012 at 01:28:40AM +0400, Solar Designer wrote:
> > Rich, Isaac -
> > 
> > On Tue, Apr 24, 2012 at 12:28:28PM -0700, Isaac Dunham wrote:
> > > On Mon, 23 Apr 2012 05:19:05 -0400 Rich Felker <dalias@aerifal.cx> wrote:
> > > > Somebody has enabled stack protector. Either your distro (what distro
> > > > are you using) has hacked gcc in a way different from what musl-gcc
> > > > expects, or wireless-tools' makefile is adding -fstack-protector...
> > > 
> > > There's no -fstack-protector, so it's Ubuntu's fault.
> > 
> > Several distros do it these days (and Owl will start doing it soon as
> > well), so it's becoming standard and needs to be supported.
> > Technically, -fno-stack-protector in the musl-gcc wrapper would probably
> > make things appear to work, but it's not necessarily a good thing to do
> > since it might violate reasonable expectations of the user.  I think
> > it'd be better for musl to start providing the necessary symbols.
> 
> The problem is that the default gcc build incorporates glibc ABI
> knowledge (layout of the thread structure) and the assumption that the
> thread pointer has been initialized into any binary built with stack
> protector. Just adding symbols will not fix anything.
> 
> If gcc's stack protector support is built without TLS support so that
> it uses a global variable for the canary, instead of a thread-local
> variable, then we could support it with no work at all. Unfortunately
> I don't believe there's any way to override the type of canary access
> gcc does via command line options; you have to rebuild gcc. Thus, for
> the time being (until/unless we do TLS and mimic glibc's structure
> layout), disabling stack protector is the only option for the wrapper
> to use. For our own native gcc builds, we can already support it if I
> just add the symbol it needs.

......but since you requested it, I'm working on trying to make it
work anyway. We'll see how it goes. Preliminary support was just
committed.

Rich

Re: New gcc wrapper to try

[Solar Designer]

Rich -

On Tue, Apr 24, 2012 at 06:10:40PM -0400, Rich Felker wrote:
> On Tue, Apr 24, 2012 at 05:41:27PM -0400, Rich Felker wrote:
> > The problem is that the default gcc build incorporates glibc ABI
> > knowledge (layout of the thread structure) and the assumption that the
> > thread pointer has been initialized into any binary built with stack
> > protector. Just adding symbols will not fix anything.

I did not realize that.

> ......but since you requested it, I'm working on trying to make it
> work anyway. We'll see how it goes. Preliminary support was just
> committed.

Wow.  At this time, I suggested it as being good for the musl project
rather than requested it for my own use (although I imagine that I might
want to use it at some point), but this does not make me any less
impressed by your work on it.

Thank you!

Alexander

Re: New gcc wrapper to try

[Szabolcs Nagy]

* Solar Designer <solar@openwall.com> [2012-04-25 02:14:23 +0400]:
> On Tue, Apr 24, 2012 at 06:10:40PM -0400, Rich Felker wrote:
> > On Tue, Apr 24, 2012 at 05:41:27PM -0400, Rich Felker wrote:
> > > The problem is that the default gcc build incorporates glibc ABI
> > > knowledge (layout of the thread structure) and the assumption that the
> > > thread pointer has been initialized into any binary built with stack
> > > protector. Just adding symbols will not fix anything.
> 
> I did not realize that.
> 
> > ......but since you requested it, I'm working on trying to make it
> > work anyway. We'll see how it goes. Preliminary support was just
> > committed.
> 
> Wow.  At this time, I suggested it as being good for the musl project
> rather than requested it for my own use (although I imagine that I might
> want to use it at some point), but this does not make me any less
> impressed by your work on it.
> 

btw i noticed that with the modified gcc where
-fstack-protector is enabled by default, the
stricter -fstack-protector-all does not work
(at least here with gcc 4.4.3)

the workaround is to turn it off and on again:
-fno-stack-protector -fstack-protector-all
works

this might be good to know when someone tries to
catch stack corruption issues


actually it would be quite useful if the compiler
could emit proper bounds check for stack access

address-sanitizer does something like that
https://code.google.com/p/address-sanitizer/

stack corruption bugs can be annoying to debug
without such a tool

Re: New gcc wrapper to try

[Solar Designer]

On Wed, Apr 25, 2012 at 01:53:39AM +0200, Szabolcs Nagy wrote:
> btw i noticed that with the modified gcc where
> -fstack-protector is enabled by default, the
> stricter -fstack-protector-all does not work
> (at least here with gcc 4.4.3)

Ouch.  I brought this up on owl-dev so that we test for it when getting
this change into Owl:

http://www.openwall.com/lists/owl-dev/2012/04/25/1

Thanks,

Alexander