Asterisk 16 function redefines

Asterisk 16 function redefines

[Sebastian Kemper]

Hi again list,

I got 2 more doubts. In previous mail chain regarding Asterisk 16 I mentioned a
header file (astmm.h) which is now included (previously it was only used when
doing things like malloc debugging AFAIK).

astmm.h
(https://github.com/asterisk/asterisk/blob/16.2/include/asterisk/astmm.h)
redefines functions. Currently, to be able to compile asterisk against
gcc 7.4.0/musl 1.1.21 at all, I have to patch the header to not redefine
anything. Otherwise the build bombs.

How astmm.h redefines functions depends on preprocessor variable
ASTMM_LIBC. For normal compiles it is either set to ASTMM_BLOCK,
ASTMM_IGNORE or ASTMM_REDIRECT, depending on which part of asterisk is
currently compiled.

ASTMM_IGNORE doesn't redefine anything. I don't think there's a problem
when this is set.

Here come my 2 questions.

1. Why does the compile fail when set to ASTMM_BLOCK?
2. Why does the compile fail when set to ASTMM_REDIRECT?

I mean, I see the errors but I don't understand where exactly they're
coming from. I compiled the same asterisk using glibc and uclibc and
there is no error when astmm.h redefines functions.

Regarding ASTMM_REDIRECT: redefines libc functions to asterisk's versions. Examples:

#define calloc(nmemb, size) \
	__ast_repl_calloc(nmemb, size, __FILE__, __LINE__, __PRETTY_FUNCTION__)
#define malloc(size) \
	__ast_repl_malloc(size, __FILE__, __LINE__, __PRETTY_FUNCTION__)
#define free(ptr) \
	__ast_free(ptr, __FILE__, __LINE__, __PRETTY_FUNCTION__)

Compile output example:

mips-openwrt-linux-musl-gcc -o aelparse.o -c aelparse.c -MD -MT aelparse.o -MF .aelparse.o.d -MP -pthread -I/home/sk/tmp/openwrt/build_dir/target-mips_24kc_musl/asterisk-16.2.1/include -Os -pipe -mno-branch-likely -mips32r2 -mtune=24kc -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -msoft-float -mips16 -minterlink-mips16 -iremap/home/sk/tmp/openwrt/build_dir/target-mips_24kc_musl/asterisk-16.2.1:asterisk-16.2.1 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -I/home/sk/tmp/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libiconv-stub/include -I/home/sk/tmp/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libintl-stub/include -I/home/sk/tmp/openwrt/staging_dir/target-mips_24kc_musl/usr/include -I/home/sk/tmp/openwrt/staging_dir/target-mips_24kc_musl/include -I/home/sk/tmp/openwrt/staging_dir/toolchain-mips_24kc_gcc-7.4.0_musl/usr/include -I/home/sk/tmp/openwrt/staging_dir/toolchain-mips_24kc_gcc-7.4.0_musl/include/fortify -I/home/sk/tmp/openwrt/staging_dir/toolchain-mips_24kc_gcc-7.4.0_musl/include -I/home/sk/tmp/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libiconv-stub/include -I/home/sk/tmp/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libintl-stub/include    -I/home/sk/tmp/openwrt/staging_dir/target-mips_24kc_musl/usr/include/libxml2  -Wall -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations     -DSTANDALONE   -I/home/sk/tmp/openwrt/build_dir/target-mips_24kc_musl/asterisk-16.2.1/res -Wno-unused -Wno-format-truncation  
/home/sk/tmp/openwrt/staging_dir/toolchain-mips_24kc_gcc-7.4.0_musl/include/sched.h:76:28: error: expected declaration specifiers or '...' before string constant
 void *calloc(size_t, size_t);
                            ^
/home/sk/tmp/openwrt/staging_dir/toolchain-mips_24kc_gcc-7.4.0_musl/include/sched.h:76:28: error: expected declaration specifiers or '...' before numeric constant
 void *calloc(size_t, size_t);
                            ^
In file included from /home/sk/tmp/openwrt/build_dir/target-mips_24kc_musl/asterisk-16.2.1/include/asterisk.h:23:0,
                 from aelparse.c:2:
/home/sk/tmp/openwrt/build_dir/target-mips_24kc_musl/asterisk-16.2.1/include/asterisk/astmm.h:138:53: error: expected declaration specifiers or '...' before '__PRETTY_FUNCTION__'
  __ast_repl_calloc(nmemb, size, __FILE__, __LINE__, __PRETTY_FUNCTION__)
                                                     ^
/home/sk/tmp/openwrt/staging_dir/toolchain-mips_24kc_gcc-7.4.0_musl/include/sched.h:77:17: error: expected declaration specifiers or '...' before string constant
 void free(void *);
                 ^
/home/sk/tmp/openwrt/staging_dir/toolchain-mips_24kc_gcc-7.4.0_musl/include/sched.h:77:17: error: expected declaration specifiers or '...' before numeric constant
 void free(void *);
                 ^
/home/sk/tmp/openwrt/build_dir/target-mips_24kc_musl/asterisk-16.2.1/include/asterisk/astmm.h:142:38: error: expected declaration specifiers or '...' before '__PRETTY_FUNCTION__'
  __ast_free(ptr, __FILE__, __LINE__, __PRETTY_FUNCTION__)
                                      ^

Regarding ASTMM_BLOCK: redefines libc functions to cause compile errors. Examples:

#define calloc(a, b) \
        Do_not_use_calloc__use_ast_calloc->fail(a, b)
#define malloc(a) \
        Do_not_use_malloc__use_ast_malloc->fail(a)
#define free(a) \
        Do_not_use_free__use_ast_free_or_ast_std_free_for_remotely_allocated_memory->fail(a)

Compile output example:

mips-openwrt-linux-musl-gcc -o chan_pjsip.o -c chan_pjsip.c -MD -MT chan_pjsip.o -MF .chan_pjsip.o.d -MP -pthread -I/home/sk/tmp/openwrt/build_dir/target-mips_24kc_musl/asterisk-16.2.1/include -Os -pipe -mno-branch-likely -mips32r2 -mtune=24kc -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -msoft-float -mips16 -minterlink-mips16 -iremap/home/sk/tmp/openwrt/build_dir/target-mips_24kc_musl/asterisk-16.2.1:asterisk-16.2.1 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -I/home/sk/tmp/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libiconv-stub/include -I/home/sk/tmp/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libintl-stub/include -I/home/sk/tmp/openwrt/staging_dir/target-mips_24kc_musl/usr/include -I/home/sk/tmp/openwrt/staging_dir/target-mips_24kc_musl/include -I/home/sk/tmp/openwrt/staging_dir/toolchain-mips_24kc_gcc-7.4.0_musl/usr/include -I/home/sk/tmp/openwrt/staging_dir/toolchain-mips_24kc_gcc-7.4.0_musl/include/fortify -I/home/sk/tmp/openwrt/staging_dir/toolchain-mips_24kc_gcc-7.4.0_musl/include -I/home/sk/tmp/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libiconv-stub/include -I/home/sk/tmp/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libintl-stub/include    -I/home/sk/tmp/openwrt/staging_dir/target-mips_24kc_musl/usr/include/libxml2  -Wall -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations       -fPIC -DAST_MODULE=\"chan_pjsip\" -DAST_MODULE_SELF_SYM=__internal_chan_pjsip_self  -DPJ_AUTOCONF=1 -DPJ_IS_BIG_ENDIAN=1 -DPJ_IS_LITTLE_ENDIAN=0 -fPIC -I/home/sk/tmp/openwrt/staging_dir/target-mips_24kc_musl/usr/include  
In file included from /home/sk/tmp/openwrt/build_dir/target-mips_24kc_musl/asterisk-16.2.1/include/asterisk.h:23:0,
                 from chan_pjsip.c:35:
/home/sk/tmp/openwrt/build_dir/target-mips_24kc_musl/asterisk-16.2.1/include/asterisk/astmm.h:158:35: error: expected '=', ',', ';', 'asm' or '__attribute__' before '->' token
  Do_not_use_calloc__use_ast_calloc->fail(a, b)
                                   ^
/home/sk/tmp/openwrt/build_dir/target-mips_24kc_musl/asterisk-16.2.1/include/asterisk/astmm.h:162:77: error: expected '=', ',', ';', 'asm' or '__attribute__' before '->' token
  Do_not_use_free__use_ast_free_or_ast_std_free_for_remotely_allocated_memory->fail(a)
                                                                             ^mips-openwrt-linux-musl-gcc -o chan_pjsip.o -c chan_pjsip.c -MD -MT chan_pjsip.o -MF .chan_pjsip.o.d -MP -pthread -I/home/sk/tmp/openwrt/build_dir/target-mips_24kc_musl/asterisk-16.2.1/include -Os -pipe -mno-branch-likely -mips32r2 -mtune=24kc -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -msoft-float -mips16 -minterlink-mips16 -iremap/home/sk/tmp/openwrt/build_dir/target-mips_24kc_musl/asterisk-16.2.1:asterisk-16.2.1 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -I/home/sk/tmp/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libiconv-stub/include -I/home/sk/tmp/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libintl-stub/include -I/home/sk/tmp/openwrt/staging_dir/target-mips_24kc_musl/usr/include -I/home/sk/tmp/openwrt/staging_dir/target-mips_24kc_musl/include -I/home/sk/tmp/openwrt/staging_dir/toolchain-mips_24kc_gcc-7.4.0_musl/usr/include -I/home/sk/tmp/openwrt/staging_dir/toolchain-mips_24kc_gcc-7.4.0_musl/include/fortify -I/home/sk/tmp/openwrt/staging_dir/toolchain-mips_24kc_gcc-7.4.0_musl/include -I/home/sk/tmp/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libiconv-stub/include -I/home/sk/tmp/openwrt/staging_dir/target-mips_24kc_musl/usr/lib/libintl-stub/include    -I/home/sk/tmp/openwrt/staging_dir/target-mips_24kc_musl/usr/include/libxml2  -Wall -Wstrict-prototypes -Wmissing-prototypes -Wmissing-declarations       -fPIC -DAST_MODULE=\"chan_pjsip\" -DAST_MODULE_SELF_SYM=__internal_chan_pjsip_self  -DPJ_AUTOCONF=1 -DPJ_IS_BIG_ENDIAN=1 -DPJ_IS_LITTLE_ENDIAN=0 -fPIC -I/home/sk/tmp/openwrt/staging_dir/target-mips_24kc_musl/usr/include  
In file included from /home/sk/tmp/openwrt/build_dir/target-mips_24kc_musl/asterisk-16.2.1/include/asterisk.h:23:0,
                 from chan_pjsip.c:35:
/home/sk/tmp/openwrt/build_dir/target-mips_24kc_musl/asterisk-16.2.1/include/asterisk/astmm.h:158:35: error: expected '=', ',', ';', 'asm' or '__attribute__' before '->' token
  Do_not_use_calloc__use_ast_calloc->fail(a, b)
                                   ^
/home/sk/tmp/openwrt/build_dir/target-mips_24kc_musl/asterisk-16.2.1/include/asterisk/astmm.h:162:77: error: expected '=', ',', ';', 'asm' or '__attribute__' before '->' token
  Do_not_use_free__use_ast_free_or_ast_std_free_for_remotely_allocated_memory->fail(a)
                                                                             ^

To me it looks like the mere definition sets off these errors for some
reason. Is it about the syntax? To me it looks fine. I mean it looks
like the syntax used for function-like macro definitions mentioned in
the gcc manual.

Again, any help appreciated.

Kind regards,
Seb

Re: Asterisk 16 function redefines

[Markus Wichmann]

On Tue, Mar 05, 2019 at 08:28:50PM +0100, Sebastian Kemper wrote:
> #define calloc(nmemb, size) \
> 	__ast_repl_calloc(nmemb, size, __FILE__, __LINE__, __PRETTY_FUNCTION__)
> #define malloc(size) \
> 	__ast_repl_malloc(size, __FILE__, __LINE__, __PRETTY_FUNCTION__)
> #define free(ptr) \
> 	__ast_free(ptr, __FILE__, __LINE__, __PRETTY_FUNCTION__)
> 
> Compile output example:
> 
> /home/sk/tmp/openwrt/staging_dir/toolchain-mips_24kc_gcc-7.4.0_musl/include/sched.h:76:28: error: expected declaration specifiers or '...' before string constant
>  void *calloc(size_t, size_t);
>                             ^

Welcome to the wonderful world of the preprocessor. The define above is
in effect here, so the compiler never sees the valid function prototype,
the compiler sees

void *__ast_repl_calloc(size_t, size_t, "sched.h", 76, "");

And can't deal with that. Maybe they should include the replacements as
the very last thing.

Nitpick, depending on how you feel about it: The name
"__ast_repl_calloc" starts with two underscores and is therefore in the
implementation namespace. So asterisk should keep their fingers off it!
They should just define the function without the underscores; what's so
hard about that?

> Regarding ASTMM_BLOCK: redefines libc functions to cause compile errors. [...]

And they cause compile errors. Mission accomplished.

I guess the code is tripping over the prototypes in sched.h.

> To me it looks like the mere definition sets off these errors for some
> reason. Is it about the syntax? To me it looks fine. I mean it looks
> like the syntax used for function-like macro definitions mentioned in
> the gcc manual.
> 
> Again, any help appreciated.
> 

These macros can't deal with prototypes. That is their failing. I don't
know how to do what they want to do, but this is probably not it. I'd
suggest going with ASTMM_IGNORE for now.

> Kind regards,
> Seb

Ciao,
Markus

Re: Asterisk 16 function redefines

[Szabolcs Nagy]

* Markus Wichmann <nullplan@gmx.net> [2019-03-05 21:05:30 +0100]:
> On Tue, Mar 05, 2019 at 08:28:50PM +0100, Sebastian Kemper wrote:
> > #define calloc(nmemb, size) \
> > 	__ast_repl_calloc(nmemb, size, __FILE__, __LINE__, __PRETTY_FUNCTION__)
> > #define malloc(size) \
> > 	__ast_repl_malloc(size, __FILE__, __LINE__, __PRETTY_FUNCTION__)
> > #define free(ptr) \
> > 	__ast_free(ptr, __FILE__, __LINE__, __PRETTY_FUNCTION__)
> > 
> > Compile output example:
> > 
> > /home/sk/tmp/openwrt/staging_dir/toolchain-mips_24kc_gcc-7.4.0_musl/include/sched.h:76:28: error: expected declaration specifiers or '...' before string constant
> >  void *calloc(size_t, size_t);
> >                             ^
> 
> Welcome to the wonderful world of the preprocessor. The define above is
> in effect here, so the compiler never sees the valid function prototype,
> the compiler sees
> 
> void *__ast_repl_calloc(size_t, size_t, "sched.h", 76, "");
> 
> And can't deal with that. Maybe they should include the replacements as
> the very last thing.

yeah that's one solution: don't include libc
headers after these macros (it may require
significant changes though, a quick workaround
hack is to preinclude sched.h before the
macros)

(in principle these names are reserved for the
implementation so such macros are not required
to work, in practice they work if there is no
later declaration of these functions.

normally stdlib.h declares these functions, so
if that's included before the macro definitions
then it works, however with _GNU_SOURCE defined
musl also declares calloc in sched.h.)

Re: Asterisk 16 function redefines

[Sebastian Kemper]

On Tue, Mar 05, 2019 at 09:05:30PM +0100, Markus Wichmann wrote:
> On Tue, Mar 05, 2019 at 08:28:50PM +0100, Sebastian Kemper wrote:
> > #define calloc(nmemb, size) \
> > 	__ast_repl_calloc(nmemb, size, __FILE__, __LINE__, __PRETTY_FUNCTION__)
> > #define malloc(size) \
> > 	__ast_repl_malloc(size, __FILE__, __LINE__, __PRETTY_FUNCTION__)
> > #define free(ptr) \
> > 	__ast_free(ptr, __FILE__, __LINE__, __PRETTY_FUNCTION__)
> > 
> > Compile output example:
> > 
> > /home/sk/tmp/openwrt/staging_dir/toolchain-mips_24kc_gcc-7.4.0_musl/include/sched.h:76:28: error: expected declaration specifiers or '...' before string constant
> >  void *calloc(size_t, size_t);
> >                             ^
> 
> Welcome to the wonderful world of the preprocessor. The define above is
> in effect here, so the compiler never sees the valid function prototype,
> the compiler sees
> 
> void *__ast_repl_calloc(size_t, size_t, "sched.h", 76, "");
> 
> And can't deal with that. Maybe they should include the replacements as
> the very last thing.

Hello Markus,

It is indeed wonderful ;-)

What wonders me the most is that the same preprocessor define works fine
with the same compiler using a different libc. I tried

gcc 7.4.0 + musl 1.1.20:	NOK
gcc 7.1.1 + uclibc 1.0.31:	OK
gcc 8.2.0 + glibc 2.27:		OK

Different gcc versions, but still.

Just read Szabolcs reply. Now it makes a bit more sense to me.

> > Regarding ASTMM_BLOCK: redefines libc functions to cause compile errors. [...]
> 
> And they cause compile errors. Mission accomplished.

I don't think exactly that failure is what they had in mind :D

Thanks!
Seb

Re: Asterisk 16 function redefines

[Sebastian Kemper]

On Tue, Mar 05, 2019 at 09:56:01PM +0100, Szabolcs Nagy wrote:
> * Markus Wichmann <nullplan@gmx.net> [2019-03-05 21:05:30 +0100]:
> > On Tue, Mar 05, 2019 at 08:28:50PM +0100, Sebastian Kemper wrote:
> > > #define calloc(nmemb, size) \
> > > 	__ast_repl_calloc(nmemb, size, __FILE__, __LINE__, __PRETTY_FUNCTION__)
> > > #define malloc(size) \
> > > 	__ast_repl_malloc(size, __FILE__, __LINE__, __PRETTY_FUNCTION__)
> > > #define free(ptr) \
> > > 	__ast_free(ptr, __FILE__, __LINE__, __PRETTY_FUNCTION__)
> > > 
> > > Compile output example:
> > > 
> > > /home/sk/tmp/openwrt/staging_dir/toolchain-mips_24kc_gcc-7.4.0_musl/include/sched.h:76:28: error: expected declaration specifiers or '...' before string constant
> > >  void *calloc(size_t, size_t);
> > >                             ^
> > 
> > Welcome to the wonderful world of the preprocessor. The define above is
> > in effect here, so the compiler never sees the valid function prototype,
> > the compiler sees
> > 
> > void *__ast_repl_calloc(size_t, size_t, "sched.h", 76, "");
> > 
> > And can't deal with that. Maybe they should include the replacements as
> > the very last thing.
> 
> yeah that's one solution: don't include libc
> headers after these macros (it may require
> significant changes though, a quick workaround
> hack is to preinclude sched.h before the
> macros)

Hello Szabolcs,

That indeed solves the compile problem. Thank you very much! When I read
your reply it started to make sense :) Thanks to Markus as well, of
course. I needed the explicit instruction ("include sched.h") to get the
picture.

Kind regards,
Seb

Re: Asterisk 16 function redefines

[Markus Wichmann]

On Tue, Mar 05, 2019 at 10:05:17PM +0100, Sebastian Kemper wrote:
> What wonders me the most is that the same preprocessor define works fine
> with the same compiler using a different libc. I tried
> 
> gcc 7.4.0 + musl 1.1.20:	NOK
> gcc 7.1.1 + uclibc 1.0.31:	OK
> gcc 8.2.0 + glibc 2.27:		OK
> 

At least for glibc, this is easy. And I suppose, uclibc will do it a
similar way.

<sched.h> has to define the macro CPU_ALLOC(). And it has to be usable
by just including <sched.h>. In musl, CPU_ALLOC() is defined in terms of
calloc(). In order for CPU_ALLOC() to be usable, calloc() must be
declared in <sched.h> as well. However, including <stdlib.h> from
<sched.h> would add a lot more names to the namespace than simply
redeclaring the functions. So that's why there's a calloc() prototype in
musl's <sched.h>.

In glibc, CPU_ALLOC() is defined in terms of __sched_cpualloc(). Which
is probably just a wrapper around the normal allocators, but it gets
<sched.h> out of having to declare the allocator functions. So no
calloc() prototype in glibc's <sched.h>, and thus the above hack works
in glibc. But only by accident.

Ciao,
Markus