musl perf, 20% slower than native build?

musl perf, 20% slower than native build?

[John Mudd]

[-- Attachment #1: Type: text/plain, Size: 3350 bytes --]

On March 13 I raised a concern about performance, Subject "musl 14x slower?
". It now looks like most of that issue had to do with my application code.
But here's a more focused look at how musl compares to native build.
Granted this is crude, limited, but it looks like musl is 20% slower.

I built Python both native and with musl. It's not apples to apples, I
actually used a newer version of gcc to build musl 1.1.8 and musl Python.
This may and may not be an advantage fro the musl version. I also used -O3
when building musl and musl Python. I assume this should help the musl
version. The musl version also uses a newer version of Python. That may or
may not be helpful. I can redo with a more consistent builds if this is
worthwhile. Maybe you can suggest a straight C benchmark instead.

I ran a standard Python benchmark. The code is available here:
http://goo.gl/UyLDYC
I picked this at random. It is not all encompassing.

I've also never used "perf" before. Feel free to advise if this needs
improvement.


musl version:

$ python
Python 2.7.9 (default, Apr  2 2015, 15:16:16)
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
$ perf stat python spitfire_bigtable.py
StringIO                                      507.98 ms
cStringIO                                     189.18 ms
list concat                                    61.59 ms

 Performance counter stats for
'/home/mudd/multicorn_ctree/spitfire_bigtable.py':

        810.537826 task-clock (msec)         #    0.971 CPUs utilized
               297 context-switches          #    0.366 K/sec
                11 cpu-migrations            #    0.014 K/sec
             5,977 page-faults               #    0.007 M/sec
     2,151,830,012 cycles                    #    2.655 GHz
    [50.64%]
   <not supported> stalled-cycles-frontend
   <not supported> stalled-cycles-backend
     3,106,074,350 instructions              #    1.44  insns per cycle
    [74.86%]
       677,389,217 branches                  #  835.728 M/sec
    [74.56%]
        13,710,101 branch-misses             #    2.02% of all branches
    [75.09%]

       0.834844640 seconds time elapsed
$



native:

$ python
Python 2.7.5 (default, Aug 19 2013, 15:23:53)
[GCC 4.7.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>
$ perf stat ~/multicorn_ctree/spitfire_bigtable.py
StringIO                                      402.63 ms
cStringIO                                     132.62 ms
list concat                                    46.89 ms

 Performance counter stats for
'/home/mudd/multicorn_ctree/spitfire_bigtable.py':

        626.547364 task-clock (msec)         #    0.982 CPUs utilized
               169 context-switches          #    0.270 K/sec
                19 cpu-migrations            #    0.030 K/sec
             5,773 page-faults               #    0.009 M/sec
     1,663,247,805 cycles                    #    2.655 GHz
    [49.94%]
   <not supported> stalled-cycles-frontend
   <not supported> stalled-cycles-backend
     2,573,617,826 instructions              #    1.55  insns per cycle
    [75.03%]
       554,357,437 branches                  #  884.781 M/sec
    [75.49%]
        10,851,258 branch-misses             #    1.96% of all branches
    [74.83%]

       0.638252827 seconds time elapsed
$

[-- Attachment #2: Type: text/html, Size: 4438 bytes --]

Re: musl perf, 20% slower than native build?

[Alexander Monakov]

On Wed, 8 Apr 2015, John Mudd wrote:

> I've also never used "perf" before. Feel free to advise if this needs
> improvement.

Yes, can you please also provide perf record/report output?  Plain 'perf stat'
is good for a very high-level overview, but insufficient to understand where
performance is lost.  Running `perf record python ...` will create 'perf.data'
file in current directory.  Use `perf report` to view it, or `perf report
--stdio` to retrieve pasteable output (top 25 or so lines should do).

Thanks for looking into it.

Alexander

Re: musl perf, 20% slower than native build?

[Szabolcs Nagy]

20% is tiny measurement noise compared to the huge
variance in the environments you are comparing

try to figure out the libc api calls that are slower in
musl and why (perf can collect stats about hot paths)

(eg. if python uses regex that would explain the
difference because musl uses a slow but correct
regex algorithm unlike glibc..)

Re: musl perf, 20% slower than native build?

[John Mudd]

[-- Attachment #1: Type: text/plain, Size: 4077 bytes --]

Here's a fresh native compile of the same version of Python, same gcc. Now
the musl version is only slightly slower, maybe 5%. BTW, I'm not
complaining. I use musl for portability, not speed.

$ python
Python 2.7.9 (default, Apr  8 2015, 14:29:14)
[GCC 4.8.2] on linux2
>>>

$ perf stat ~/multicorn_ctree/spitfire_bigtable.py
StringIO                                      523.62 ms
cStringIO                                     144.32 ms
list concat                                    55.12 ms

 Performance counter stats for
'/home/mudd/multicorn_ctree/spitfire_bigtable.py':

        769.874633 task-clock (msec)         #    0.977 CPUs utilized
               269 context-switches          #    0.349 K/sec
                 6 cpu-migrations            #    0.008 K/sec
             5,997 page-faults               #    0.008 M/sec
     2,043,153,669 cycles                    #    2.654 GHz
    [50.74%]
   <not supported> stalled-cycles-frontend
   <not supported> stalled-cycles-backend
     2,993,940,382 instructions              #    1.47  insns per cycle
    [75.11%]
       673,064,696 branches                  #  874.252 M/sec
    [74.59%]
        15,486,299 branch-misses             #    2.30% of all branches
    [74.71%]

       0.787704322 seconds time elapsed
$


Here's output from perf record/report for libc. This looks consistent with
the 5% longer run time.

native:
     2.20%   python  libc-2.19.so         [.] __memcpy_ssse3
     0.85%   python  libc-2.19.so         [.] __x86.get_pc_thunk.bx
     0.72%   python  libc-2.19.so         [.] _int_malloc
     0.56%   python  libc-2.19.so         [.] __memset_sse2
     0.47%   python  libc-2.19.so         [.] _int_free
     0.38%   python  libc-2.19.so         [.] malloc
     0.25%   python  libc-2.19.so         [.] realloc
     0.25%   python  libc-2.19.so         [.] __ctype_b_loc
     0.10%   python  libc-2.19.so         [.] free
     0.04%   python  libc-2.19.so         [.] __strchr_sse2_bsf
     0.03%   python  libc-2.19.so         [.] __memcpy_ia32
     0.03%   python  libc-2.19.so         [.] __sbrk
     0.03%   python  libc-2.19.so         [.] vfprintf
     0.03%   python  libc-2.19.so         [.] mremap_chunk
     0.03%   python  libc-2.19.so         [.] __strncpy_ssse3
     0.03%   python  libc-2.19.so         [.] __strlen_sse2_bsf
     0.03%   python  libc-2.19.so         [.] __x86.get_pc_thunk.cx

musl:
     4.74%   python  libc.so              [.] memcpy
     2.05%   python  libc.so              [.] free
     1.17%   python  libc.so              [.] malloc
     1.05%   python  libc.so              [.] unbin
     0.90%   python  libc.so              [.] a_and_64
     0.81%   python  libc.so              [.] a_or_64
     0.68%   python  libc.so              [.] memset
     0.31%   python  libc.so              [.] bin_index_up
     0.22%   python  libc.so              [.] bin_index
     0.22%   python  libc.so              [.] a_ctz_64
     0.16%   python  libc.so              [.] realloc
     0.16%   python  libc.so              [.] __x86.get_pc_thunk.bx
     0.14%   python  libc.so              [.] strlen
     0.12%   python  libc.so              [.] trim
     0.12%   python  libc.so              [.] strcmp
     0.09%   python  libc.so              [.] adjust_size
     0.06%   python  libc.so              [.] __strerror_l
     0.06%   python  libc.so              [.] __stpncpy
     0.03%   python  libc.so              [.] first_set
     0.03%   python  libc.so              [.] .L80
     0.03%   python  libc.so              [.] remap_rel
     0.03%   python  libc.so              [.] find_sym
     0.03%   python  libc.so              [.] sysv_hash
     0.03%   python  libc.so              [.] fclose
     0.03%   python  libc.so              [.] do_relocs
     0.03%   python  libc.so              [.] __aio_close
     0.03%   python  libc.so              [.] sysv_lookup




On Wed, Apr 8, 2015 at 12:05 PM, Szabolcs Nagy <nsz@port70.net> wrote:

> 20% is tiny measurement noise compared to the huge
> variance in the environments you are comparing
>
>

[-- Attachment #2: Type: text/html, Size: 6536 bytes --]

Re: musl perf, 20% slower than native build?

[Daniel Cegiełka]

2015-04-08 21:10 GMT+02:00 John Mudd <johnbmudd@gmail.com>:

> Here's output from perf record/report for libc. This looks consistent with
> the 5% longer run time.
>
> native:
>      2.20%   python  libc-2.19.so         [.] __memcpy_ssse3

>
> musl:
>      4.74%   python  libc.so              [.] memcpy

I was able to get twice speed-up (in my code) just by replacing memcpy
in the musl.

Daniel

Re: musl perf, 20% slower than native build?

[Alexander Monakov]

That a_* functions appear in this profile implies they were not inlined, which
is quite strange.  Are you positive -O3 was used for your musl compile?  Can
you check the list of symbols in libc.so (e.g. using nm or `readelf -a`) to
double check?

Thanks.

Alexander

Re: musl perf, 20% slower than native build?

[John Mudd]

[-- Attachment #1: Type: text/plain, Size: 1996 bytes --]

I don't see any of those "a_*" names in the output of 'nm' or 'readelf -a'
on musl-1.1.8/lib/libc.so.

Here's my configure command:

./configure --prefix=$(pwd).install --syslibdir=/usr/rx30/musl/lib/
CFLAGS=-O3 CC=gcc

Here's a sample of the make output:

gcc -std=c99 -nostdinc -ffreestanding -fexcess-precision=standard
-frounding-math -D_XOPEN_SOURCE=700 -I./arch/i386 -I./src/internal
-I./include  -pipe -fomit-frame-pointer -fno-unwind-tables
-fno-asynchronous-unwind-tables -Wa,--noexecstack -march=i486
-mtune=generic -Werror=implicit-function-declaration -Werror=implicit-int
-Werror=pointer-sign -Werror=pointer-arith -fno-stack-protector -O3 -c -o
crt/crt1.o crt/i386/crt1.s
cp crt/crt1.o lib/crt1.o
gcc -std=c99 -nostdinc -ffreestanding -fexcess-precision=standard
-frounding-math -D_XOPEN_SOURCE=700 -I./arch/i386 -I./src/internal
-I./include  -pipe -fomit-frame-pointer -fno-unwind-tables
-fno-asynchronous-unwind-tables -Wa,--noexecstack -march=i486
-mtune=generic -Werror=implicit-function-declaration -Werror=implicit-int
-Werror=pointer-sign -Werror=pointer-arith -fno-stack-protector -O3 -fPIC
-c -o crt/Scrt1.o crt/i386/Scrt1.s
cp crt/Scrt1.o lib/Scrt1.o
gcc -std=c99 -nostdinc -ffreestanding -fexcess-precision=standard
-frounding-math -D_XOPEN_SOURCE=700 -I./arch/i386 -I./src/internal
-I./include  -pipe -fomit-frame-pointer -fno-unwind-tables
-fno-asynchronous-unwind-tables -Wa,--noexecstack -march=i486
-mtune=generic -Werror=implicit-function-declaration -Werror=implicit-int
-Werror=pointer-sign -Werror=pointer-arith -fno-stack-protector -O3 -c -o
crt/crti.o crt/i386/crti.s
cp crt/crti.o lib/crti.o

On Wed, Apr 8, 2015 at 3:33 PM, Alexander Monakov <amonakov@ispras.ru>
wrote:

> That a_* functions appear in this profile implies they were not inlined,
> which
> is quite strange.  Are you positive -O3 was used for your musl compile?
> Can
> you check the list of symbols in libc.so (e.g. using nm or `readelf -a`) to
> double check?
>
> Thanks.
>
> Alexander
>
>

[-- Attachment #2: Type: text/html, Size: 2566 bytes --]

Re: musl perf, 20% slower than native build?

[John Mudd]

[-- Attachment #1: Type: text/plain, Size: 280 bytes --]

Ignore my previous email. The symbols are there. I'll rebuild again and
check the speed...

On Wed, Apr 8, 2015 at 4:16 PM, John Mudd <johnbmudd@gmail.com> wrote:

> I don't see any of those "a_*" names in the output of 'nm' or 'readelf -a'
> on musl-1.1.8/lib/libc.so.
>
>>
>>
>

[-- Attachment #2: Type: text/html, Size: 908 bytes --]

Re: musl perf, 20% slower than native build?

[Paul Schutte]

[-- Attachment #1: Type: text/plain, Size: 634 bytes --]

Hi Daniel,

Pardon my stupidity, but with what did you replace the memcpy ?

Regards
Paul

On Wed, Apr 8, 2015 at 9:28 PM, Daniel Cegiełka <daniel.cegielka@gmail.com>
wrote:

> 2015-04-08 21:10 GMT+02:00 John Mudd <johnbmudd@gmail.com>:
>
> > Here's output from perf record/report for libc. This looks consistent
> with
> > the 5% longer run time.
> >
> > native:
> >      2.20%   python  libc-2.19.so         [.] __memcpy_ssse3
>
> >
> > musl:
> >      4.74%   python  libc.so              [.] memcpy
>
> I was able to get twice speed-up (in my code) just by replacing memcpy
> in the musl.
>
> Daniel
>

[-- Attachment #2: Type: text/html, Size: 1212 bytes --]

Re: musl perf, 20% slower than native build?

[John Mudd]

[-- Attachment #1: Type: text/plain, Size: 2503 bytes --]

Reran benchmark using -O3 musl library and now musl and native have
basically the same run times.

native:
     2.08%   python  libc-2.19.so         [.] __memcpy_ssse3
     0.81%   python  libc-2.19.so         [.] __x86.get_pc_thunk.bx
     0.56%   python  libc-2.19.so         [.] __memset_sse2
     0.53%   python  libc-2.19.so         [.] _int_malloc
     0.45%   python  libc-2.19.so         [.] _int_free
     0.40%   python  libc-2.19.so         [.] malloc
     0.22%   python  libc-2.19.so         [.] realloc
     0.21%   python  libc-2.19.so         [.] __ctype_b_loc
     0.12%   python  libc-2.19.so         [.] __x86.get_pc_thunk.cx
     0.12%   python  libc-2.19.so         [.] free
     0.01%   python  libc-2.19.so         [.] __strlen_sse2_bsf
     0.01%   python  libc-2.19.so         [.] malloc_consolidate
     0.01%   python  libc-2.19.so         [.] __memcpy_ia32
     0.00%   python  libc-2.19.so         [.] _IO_getc
     0.00%   python  libc-2.19.so         [.] mremap_chunk
     0.00%   python  libc-2.19.so         [.] _int_realloc
     0.00%   python  libc-2.19.so         [.] __strcmp_ssse3


musl:
     5.37%   python  libc.so              [.] memcpy
     1.96%   python  libc.so              [.] malloc
     1.93%   python  libc.so              [.] free
     0.60%   python  libc.so              [.] memset
     0.20%   python  libc.so              [.] __x86.get_pc_thunk.bx
     0.12%   python  libc.so              [.] realloc
     0.03%   python  libc.so              [.] alloc_fwd
     0.02%   python  libc.so              [.] alloc_rev
     0.01%   python  libc.so              [.] find_sym
     0.01%   python  libc.so              [.] strcmp
     0.00%   python  libc.so              [.] memcmp
     0.00%   python  libc.so              [.] __vm_lock
     0.00%   python  libc.so              [.] __strerror_l
     0.00%   python  libc.so              [.] __cp_end
     0.00%   python  libc.so              [.] .L350
     0.00%   python  libc.so              [.] strlen
     0.00%   python  libc.so              [.] memchr
     0.00%   python  libc.so              [.] pthread_rwlock_trywrlock




On Wed, Apr 8, 2015 at 4:24 PM, John Mudd <johnbmudd@gmail.com> wrote:

> Ignore my previous email. The symbols are there. I'll rebuild again and
> check the speed...
>
> On Wed, Apr 8, 2015 at 4:16 PM, John Mudd <johnbmudd@gmail.com> wrote:
>
>> I don't see any of those "a_*" names in the output of 'nm' or 'readelf
>> -a' on musl-1.1.8/lib/libc.so.
>>
>>>
>>>
>>
>

[-- Attachment #2: Type: text/html, Size: 4685 bytes --]

Re: musl perf, 20% slower than native build?

[Daniel Cegiełka]

2015-04-08 22:59 GMT+02:00 Paul Schutte <sjpschutte@gmail.com>:
> Hi Daniel,
>
> Pardon my stupidity, but with what did you replace the memcpy ?

I use memcpy more suited to my CPU. memcpy latency was very important
for me because it had a big impact on the total latency (in my code).
I suppose that most of the problems with latency will have its cause
in musl's memcpy. This is quite a complex topic, because the memcpy's
optimal code depends on how large blocks of memory will be copied.
Sometimes faster will be SSE2 and sometimes AVX2, but heavily
optimized code is not portable (eg AVX2) and this is a problem. Fast
memcpy implementations usualy uses CPUID to choose the right code, but
such code is blown and ugly.

Daniel


> Regards
> Paul
>
> On Wed, Apr 8, 2015 at 9:28 PM, Daniel Cegiełka <daniel.cegielka@gmail.com>
> wrote:
>>
>> 2015-04-08 21:10 GMT+02:00 John Mudd <johnbmudd@gmail.com>:
>>
>> > Here's output from perf record/report for libc. This looks consistent
>> > with
>> > the 5% longer run time.
>> >
>> > native:
>> >      2.20%   python  libc-2.19.so         [.] __memcpy_ssse3
>>
>> >
>> > musl:
>> >      4.74%   python  libc.so              [.] memcpy
>>
>> I was able to get twice speed-up (in my code) just by replacing memcpy
>> in the musl.
>>
>> Daniel
>
>

Execinfo.h, backtrace is needed.

[Jean-Marc Pigeon]

[-- Attachment #1: Type: text/plain, Size: 2084 bytes --]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,


Noticed today, musl doesn't provide execinfo.h
nor backtrace function.

Reading mailling list as backtrace is a not needed
function as "gdb can do it".

I beg to disagree.

There is many time where backtrace capabilities
are handy.
Ex: within a daemon you want to detect and trace
a memory leak origin. This can not be done within
a testbench, as the leak is small and occur
once in a (very long) while.

I understand there is no "standard" defining
the way to implement it.
What is the musl designers position,
never ever backtrace to be implemented or
this could be good for real production
program.

May be there is a way to get backtrace
capabilities I overlooked within musl?

Comment?


- -- 

A bientôt
===========================================================
Jean-Marc Pigeon                        E-Mail: jmp@safe.ca
SAFE Inc.                             Phone: (514) 493-4280
  Clement, 'a kiss solution' to get rid of SPAM (at last)
     Clement' Home base <"http://www.clement.safe.ca">
===========================================================
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJVJrbXAAoJEAtgwJ4bBQU5/JkP/iM5s755X2gC7YgMwRtBwUi2
VnZMPeUgDlq9cn9+etvu0T5SUuYPkYI8YsNoU56Ui75VlV/VTdKfmTxQr3YGRNrQ
FaDH2o1Mkub4d9DDVkJegVuJaN7nYHjK2SM8hDsmGXMH0a7ZLPBzvRNgucoIx6Or
lDF8KnfCaExtCvEyfbTavZSkFnW5g37lSudPCdPt3ANvHYwe+rTk0QEh9kt+VifN
ug2cj5F7vnlMaFYxgpIWXL2UPv2I1Bdh6ZqMo1M+iwBYw9TFT1VwgEZFxWqAtnXe
1eLxiGiI1Y0z7iOu+ap33W0xvcRkxsgqjn/PQP46e2Q+nPIuiRVb4fMOXFaUkjQN
BEBOhKww2LmryTJvYXVcvp0z8Fh3rWOx9zXGiJp476EpHNTXIlTKshaM1RKKTIvR
Dwy9tsyAHpzKekzDuHrwUs7LR/0Q3S0vlFd/mu/BqMRkO1dZXYYYicfXDsqxiIaE
pf5IT1O4lqteiHnvZWdzqSe5Dzc2j4AtCgioA3+r792OwwAbzJakKJiShLC3TBbs
vodsj9ppx/1xWsOSf3JFm6fq7qgrwdeh5oqeEc7oJGK67YTdHKzpHvGcOaXHMA1n
myGdsd6gB7fGKIBJFxHguyBPoGFSgh/ElJo7HbdnPgqDQHpxt27lcGUfVooYVQ6H
81DkHs2uFLc2s42zKKeG
=fZW+
-----END PGP SIGNATURE-----


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4242 bytes --]

Re: Execinfo.h, backtrace is needed.

[Rich Felker]

On Thu, Apr 09, 2015 at 01:28:55PM -0400, Jean-Marc Pigeon wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Hello,
> 
> 
> Noticed today, musl doesn't provide execinfo.h
> nor backtrace function.
> 
> Reading mailling list as backtrace is a not needed
> function as "gdb can do it".
> 
> I beg to disagree.
> 
> There is many time where backtrace capabilities
> are handy.
> Ex: within a daemon you want to detect and trace
> a memory leak origin. This can not be done within
> a testbench, as the leak is small and occur
> once in a (very long) while.
> 
> I understand there is no "standard" defining
> the way to implement it.
> What is the musl designers position,
> never ever backtrace to be implemented or
> this could be good for real production
> program.
> 
> May be there is a way to get backtrace
> capabilities I overlooked within musl?

I believe there's a third-party libbacktrace that might work with
little or no modification. The function is not appropriate for
inclusion in musl both for the reasons discussed in the past and for
important technical reasons, including at least:

- It would require musl either static link with gcc's libgcc_eh or
  libunwind, neither of which is acceptable, or that we implement our
  own dwarf interpreter, which is major scope creep.

- It would require that at least part of musl be built with eh_frame
  unwind information.

With a third-party library all of these issues are isolated from musl
and only affect people using the backtrace library.

Rich

Re: Execinfo.h, backtrace is needed.

[Alexander Monakov]

On Thu, 9 Apr 2015, Rich Felker wrote:
> I believe there's a third-party libbacktrace that might work with
> little or no modification.

One specific example would be libbacktrace from GCC, which is available under
a BSD-style license: https://gcc.gnu.org/git/?p=gcc.git;a=tree;f=libbacktrace

Building GCC's libbacktrace separately from the rest of the compiler is not
straighforward; some time ago I worked on that when importing libbacktrace
for use in apitrace project; you can view the result here:
https://github.com/apitrace/apitrace/tree/master/thirdparty/libbacktrace
(it uses cmake simply because apitrace itself is using cmake).

Please note: libbacktrace may be overkill if you don't need DWARF parsing at
runtime. For example, if you can perform symbol resolution separately as
post-processing, and capturing only stack frame addresses at runtime suffices,
libunwind may be the tool for the job.  Also, libunwind provides backtrace()
as a weak alias to its own unw_backtrace(), so in simple cases it's a drop-in
replacement.

Hope that helps.
Alexander

Re: Execinfo.h, backtrace is needed.

[Jean-Marc Pigeon]

[-- Attachment #1: Type: text/plain, Size: 2090 bytes --]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

On 04/09/2015 01:57 PM, Alexander Monakov wrote:
[...]
> Please note: libbacktrace may be overkill if you don't need DWARF 
> parsing at runtime. For example, if you can perform symbol 
> resolution separately as post-processing, and capturing only stack 
> frame addresses at runtime suffices, libunwind may be the tool for 
> the job.  Also, libunwind provides backtrace() as a weak alias to 
> its own unw_backtrace(), so in simple cases it's a drop-in 
> replacement.
> 
> Hope that helps. Alexander

Many thanks for your help,
installed and using now libunwind, it is exactly what was
needed, just replacing backtrace by unw_backtrace.
Still using addr2line to display the exact "memory leak
origine".



- -- 

A bientôt
===========================================================
Jean-Marc Pigeon                        E-Mail: jmp@safe.ca
SAFE Inc.                             Phone: (514) 493-4280
  Clement, 'a kiss solution' to get rid of SPAM (at last)
     Clement' Home base <"http://www.clement.safe.ca">
===========================================================
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJVJ95/AAoJEAtgwJ4bBQU5XXwP/2rJrGvgiohS7DVe9ivKDOYK
Q/v0txAIq+RRdwxV2rPX3nFvtHWNvkMb/HQ7wdVZUkVSFzL09+q/E+HxEK5lcMP3
SHvIcy7CiECRMtnui2CHb84COKGH27T6M5L5iPs7MqjFNcbMEu0vITl6QeY6l5IQ
5qCotHYGfPBTxJz8ZYPfcTdwko6emSunKdmgzQMkFi6VKyv2OriJ5v0nm+2qajR2
G7rO97yS3eq0gdw+e7M3o5nGnqJWaz1NMCA2V259AWfElvCYFFIyQAkMRUjGC6AX
2Z9fMxg7JtVI1evxi9LUr/wQNQKJwr08GygWvdErcXgrJXJ0R/WOIJ46JSjAszUq
XDsOJIEMDPurSi5cxV5GyVx7RtncRgM2AW+Pe4jFQLg0plXiMNCpHv6SVon268e5
5dzDwvfAVnP80Ed2j/L3cOquO16D7SxwnlIN8lKD6swTbtl/UnMmyOjU8iAwzXTX
AgpXzYdGpE9nnV1pGAdGTeqD2TURAtA1Qv3GSTi0O06aAfuUIJOyxs/pNIlVSQip
JuyGBdanDsJ4WflFCzmbosMRTo50/qlXMVCDm94bgjtPKAha+y9vOipDFDcTGzc6
HL+Z+AjzIjdE02ea0ZVL2wBLOM+cdfqSqIGx47Y0V4WZKFfYnJpUji1w8/bt82q5
YEQB5mEdkineURcW0q5g
=GYIL
-----END PGP SIGNATURE-----


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4242 bytes --]