From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/13546 Path: news.gmane.org!.POSTED!not-for-mail From: Arkadiusz Sienkiewicz Newsgroups: gmane.linux.lib.musl.general Subject: Re: aio_cancel segmentation fault for in progress write requests Date: Mon, 10 Dec 2018 10:05:05 +0100 Message-ID: References: <20181207154419.GD23599@brightrain.aerifal.cx> <20181207165217.GE23599@brightrain.aerifal.cx> <54b4d253-1660-3207-5d59-f23f1c25b2b9@adelielinux.org> <20181207182650.GF23599@brightrain.aerifal.cx> <87h8fpaypx.fsf@oldenburg2.str.redhat.com> <20181207201453.GH23599@brightrain.aerifal.cx> <87in049em2.fsf@oldenburg2.str.redhat.com> Reply-To: musl@lists.openwall.com NNTP-Posting-Host: blaine.gmane.org Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="00000000000074439c057ca743dc" X-Trace: blaine.gmane.org 1544432613 2592 195.159.176.226 (10 Dec 2018 09:03:33 GMT) X-Complaints-To: usenet@blaine.gmane.org NNTP-Posting-Date: Mon, 10 Dec 2018 09:03:33 +0000 (UTC) Cc: dalias@libc.org To: musl@lists.openwall.com Original-X-From: musl-return-13562-gllmg-musl=m.gmane.org@lists.openwall.com Mon Dec 10 10:03:29 2018 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by blaine.gmane.org with smtp (Exim 4.84_2) (envelope-from ) id 1gWHTI-0000X0-OX for gllmg-musl@m.gmane.org; Mon, 10 Dec 2018 10:03:24 +0100 Original-Received: (qmail 21924 invoked by uid 550); 10 Dec 2018 09:05:30 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Original-Received: (qmail 21888 invoked from network); 10 Dec 2018 09:05:28 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=GoiPO+Tlu216Hn5Vggk2U9IB19qnER1DfnODgYbNxVU=; b=OtcFL4C+x2ObavWNcnYWy5+OBtskW5HRYO9lm5Pkpy5VlgNiDqaXqMhrA1xU4R80gm jvJ9YE/UOCunWtepTnm0D+4uIHR6dnFxJ3JztZKIBwDyBM302eVYR7V8rL73kptH74no bjrV3GWZNYgkX8re2jo+u0l6mvNyuqog+Lh1jGQVxcbPYwCX3gFOE9UPc682MoEuVK9/ LTcSIRZ2NMu26Trm5Q3e6i8VB43p4wrCNWKGB1gz0hiqdioItvi+KN0/xVB1joe1kxHp T3VD6EpquPKzjxZYzMEqJS9OrgjG2S+vJb5TyJdRRQgVz4A3aOCXRexuzrhY+4HgTTrW ZuKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=GoiPO+Tlu216Hn5Vggk2U9IB19qnER1DfnODgYbNxVU=; b=Zn/5wIaRVbG3PcRsBqfACsaU2V34gJgssJUcd/Dbd+iH9EoTViQKBfUt6RRerXG2VV Bi+0vlAHHCai3wEPj0DFxRWz/HDOS742Rr7bEsR82B/QIWHnEWtJw82yYmNhdFeGIehM 3FNiclgyWv87hhc6boyFozPn4dmFqi7i3Rbg5vFxO1bl9b1zS6cAZ6kHAK3DDqFpourR 3nVeLLqAxTF41TLpXNLP9c42XdjA3OyKiDsJM2zJZ3pcyFzwVbqkMLzpIg6Y/M9goeOk uXBVlh6Ad2U9F4pq8dROod4kVIiKLQEQovXd3zf6skRq/zMOHzFiALuyLD6e/gfO2E0V BZYQ== X-Gm-Message-State: AA+aEWaNfIgDZW2sYjwD/QHUA5Um8Cvphu+3ybOP0uZ9KeuoLmx0IkSD Axmmjap5iZR6acHaOPKIE4GIc38MLLhT6+BhY2DlmU2G X-Google-Smtp-Source: AFSGD/UYLOQak0uXPtNjP91y3YKQtDUy/snt9U/B1kgDbJyR8RMCmP6jNSkweCDH2CiJy1YPNV99FvpKpdwWt7hFtw8= X-Received: by 2002:ac8:668c:: with SMTP id d12mr10751002qtp.242.1544432716627; Mon, 10 Dec 2018 01:05:16 -0800 (PST) In-Reply-To: <87in049em2.fsf@oldenburg2.str.redhat.com> Xref: news.gmane.org gmane.linux.lib.musl.general:13546 Archived-At: --00000000000074439c057ca743dc Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Here are answers to some question directed to me earlier: > Could you attach the log from "strace -f -o strace.log ~/aioWrite"? Sorry, can't do that. strace is not installed and I don't have root access. If this is still needed I will ask admin to add strace. > Do the other machines have the same kernel (4.15.0-20-generic)? No, other machines use kernel 4.15.0-39-generic. > Have you tried running the binary built on a successful machine on the problematic machine? Yes, same effect - segmentation fault. bt from gdb is identical too. > valgrind might also be a good idea. alpine-tmp-0:~$ strace -f ./aioWrite -sh: strace: not found alpine-tmp-0:~$ valgrind valgrind valgrind-di-server valgrind-listener alpine-tmp-0:~$ valgrind ./aioWrite =3D=3D70339=3D=3D Memcheck, a memory error detector =3D=3D70339=3D=3D Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward = et al. =3D=3D70339=3D=3D Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyr= ight info =3D=3D70339=3D=3D Command: ./aioWrite =3D=3D70339=3D=3D =3D=3D70339=3D=3D Invalid free() / delete / delete[] / realloc() =3D=3D70339=3D=3D at 0x4C92B0E: free (vg_replace_malloc.c:530) =3D=3D70339=3D=3D by 0x4020248: reclaim_gaps (dynlink.c:478) =3D=3D70339=3D=3D by 0x4020CD0: map_library (dynlink.c:674) =3D=3D70339=3D=3D by 0x4021818: load_library (dynlink.c:980) =3D=3D70339=3D=3D by 0x4022607: load_preload (dynlink.c:1075) =3D=3D70339=3D=3D by 0x4022607: __dls3 (dynlink.c:1585) =3D=3D70339=3D=3D by 0x4021EDB: __dls2 (dynlink.c:1389) =3D=3D70339=3D=3D by 0x401FC8E: ??? (in /lib/ld-musl-x86_64.so.1) =3D=3D70339=3D=3D Address 0x4e9a180 is in a rw- mapped file /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so segment =3D=3D70339=3D=3D =3D=3D70339=3D=3D Can't extend stack to 0x4087948 during signal delivery fo= r thread 2: =3D=3D70339=3D=3D no stack segment =3D=3D70339=3D=3D =3D=3D70339=3D=3D Process terminating with default action of signal 11 (SIG= SEGV): dumping core =3D=3D70339=3D=3D Access not within mapped region at address 0x4087948 =3D=3D70339=3D=3D at 0x4016834: __syscall3 (syscall_arch.h:29) =3D=3D70339=3D=3D by 0x4016834: __wake (pthread_impl.h:133) =3D=3D70339=3D=3D by 0x4016834: cleanup (aio.c:154) =3D=3D70339=3D=3D by 0x40167B0: io_thread_func (aio.c:255) =3D=3D70339=3D=3D by 0x4054292: start (pthread_create.c:145) =3D=3D70339=3D=3D by 0x4053071: ??? (clone.s:21) =3D=3D70339=3D=3D by 0x4053071: ??? (clone.s:21) =3D=3D70339=3D=3D by 0x4053071: ??? (clone.s:21) =3D=3D70339=3D=3D by 0x4053071: ??? (clone.s:21) =3D=3D70339=3D=3D by 0x4053071: ??? (clone.s:21) =3D=3D70339=3D=3D by 0x4053071: ??? (clone.s:21) =3D=3D70339=3D=3D by 0x4053071: ??? (clone.s:21) =3D=3D70339=3D=3D by 0x4053071: ??? (clone.s:21) =3D=3D70339=3D=3D by 0x4053071: ??? (clone.s:21) =3D=3D70339=3D=3D If you believe this happened as a result of a stack =3D=3D70339=3D=3D overflow in your program's main thread (unlikely but =3D=3D70339=3D=3D possible), you can try to increase the size of the =3D=3D70339=3D=3D main thread stack using the --main-stacksize=3D flag. =3D=3D70339=3D=3D The main thread stack size used in this run was 8388608. =3D=3D70339=3D=3D =3D=3D70339=3D=3D HEAP SUMMARY: =3D=3D70339=3D=3D in use at exit: 81,051 bytes in 9 blocks =3D=3D70339=3D=3D total heap usage: 9 allocs, 3 frees, 81,051 bytes alloc= ated =3D=3D70339=3D=3D =3D=3D70339=3D=3D LEAK SUMMARY: =3D=3D70339=3D=3D definitely lost: 0 bytes in 0 blocks =3D=3D70339=3D=3D indirectly lost: 0 bytes in 0 blocks =3D=3D70339=3D=3D possibly lost: 0 bytes in 0 blocks =3D=3D70339=3D=3D still reachable: 81,051 bytes in 9 blocks =3D=3D70339=3D=3D suppressed: 0 bytes in 0 blocks =3D=3D70339=3D=3D Rerun with --leak-check=3Dfull to see details of leaked m= emory =3D=3D70339=3D=3D =3D=3D70339=3D=3D For counts of detected and suppressed errors, rerun with:= -v =3D=3D70339=3D=3D ERROR SUMMARY: 3 errors from 1 contexts (suppressed: 0 fr= om 0) Killed sob., 8 gru 2018 o 17:18 Florian Weimer napisa=C5=82(a= ): > * Rich Felker: > > > On Fri, Dec 07, 2018 at 09:06:18PM +0100, Florian Weimer wrote: > >> * Rich Felker: > >> > >> > I don't think so. I'm concerned that it's a stack overflow, and that > >> > somehow the kernel folks have managed to break the MINSIGSTKSZ ABI. > >> > >> Probably: > >> > >> > >> > >> > >> It's a nasty CPU backwards compatibility problem. Some of the > >> suggestions I made to work around this are simply wrong; don't take th= em > >> too seriously. > >> > >> Nowadays, the kernel has a way to disable the %zmm registers, but it > >> unfortunately does not reduce the save area size. > > > > How large is the saved context with the %zmm junk? I measured just > > ~768 bytes on normal x86_64 without it, and since 2048 is rounded up > > to a whole page (4096), overflow should not happen until the signal > > context is something like 3.5k (allowing ~512 bytes for TCB (~128) and > > 2 simple call frames). > > I wrote a test to do some measurements: > > > > The signal handler context is quite large on x86-64 with AVX-512F, > indeed around 3.5 KiB. It is even larger on ppc64 and ppc64el > (~4.5 KiB), which I find somewhat surprising. > > The cancellation test also includes stack usage from the libgcc > unwinder. Its stack usage likely differs between versions, so I should > have included that in the reported results. > > Thanks, > Florian > --00000000000074439c057ca743dc Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Here are answers to= some question directed to me earlier:

> Could = you attach the log from "strace -f -o strace.log ~/aioWrite"?
Sorry, can't do that. strace is not installed and I don't ha= ve root access. If this is still needed I will ask admin to add strace.
=

> Do the other machines have the same kernel (= 4.15.0-20-generic)?
No, other machines use kernel 4.15.0-39-gener= ic.

> Have you tried running the binary built on a successful machine on
the problematic machine?

Yes, same effect - segmen= tation fault. bt from gdb is identical too.

> v= algrind might also be a good idea.

alpine-tmp-0:~$= strace -f ./aioWrite
-sh: strace: not found
alpine-tmp-0:~$ valgrind=
valgrind=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 valgrind-di-server=C2=A0 valgrind-listener
alpine-tmp-0:~$ valgrind = ./aioWrite
=3D=3D70339=3D=3D Memcheck, a memory error detector
=3D=3D= 70339=3D=3D Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et= al.
=3D=3D70339=3D=3D Using Valgrind-3.11.0 and LibVEX; rerun with -h f= or copyright info
=3D=3D70339=3D=3D Command: ./aioWrite
=3D=3D70339= =3D=3D
=3D=3D70339=3D=3D Invalid free() / delete / delete[] / realloc()=
=3D=3D70339=3D=3D=C2=A0=C2=A0=C2=A0 at 0x4C92B0E: free (vg_replace_mall= oc.c:530)
=3D=3D70339=3D=3D=C2=A0=C2=A0=C2=A0 by 0x4020248: reclaim_gaps= (dynlink.c:478)
=3D=3D70339=3D=3D=C2=A0=C2=A0=C2=A0 by 0x4020CD0: map_l= ibrary (dynlink.c:674)
=3D=3D70339=3D=3D=C2=A0=C2=A0=C2=A0 by 0x4021818:= load_library (dynlink.c:980)
=3D=3D70339=3D=3D=C2=A0=C2=A0=C2=A0 by 0x4= 022607: load_preload (dynlink.c:1075)
=3D=3D70339=3D=3D=C2=A0=C2=A0=C2= =A0 by 0x4022607: __dls3 (dynlink.c:1585)
=3D=3D70339=3D=3D=C2=A0=C2=A0= =C2=A0 by 0x4021EDB: __dls2 (dynlink.c:1389)
=3D=3D70339=3D=3D=C2=A0=C2= =A0=C2=A0 by 0x401FC8E: ??? (in /lib/ld-musl-x86_64.so.1)
=3D=3D70339=3D= =3D=C2=A0 Address 0x4e9a180 is in a rw- mapped file /usr/lib/valgrind/vgpre= load_memcheck-amd64-linux.so segment
=3D=3D70339=3D=3D
=3D=3D70339= =3D=3D Can't extend stack to 0x4087948 during signal delivery for threa= d 2:
=3D=3D70339=3D=3D=C2=A0=C2=A0 no stack segment
=3D=3D70339=3D=3D=
=3D=3D70339=3D=3D Process terminating with default action of signal 11= (SIGSEGV): dumping core
=3D=3D70339=3D=3D=C2=A0 Access not within mappe= d region at address 0x4087948
=3D=3D70339=3D=3D=C2=A0=C2=A0=C2=A0 at 0x4= 016834: __syscall3 (syscall_arch.h:29)
=3D=3D70339=3D=3D=C2=A0=C2=A0=C2= =A0 by 0x4016834: __wake (pthread_impl.h:133)
=3D=3D70339=3D=3D=C2=A0=C2= =A0=C2=A0 by 0x4016834: cleanup (aio.c:154)
=3D=3D70339=3D=3D=C2=A0=C2= =A0=C2=A0 by 0x40167B0: io_thread_func (aio.c:255)
=3D=3D70339=3D=3D=C2= =A0=C2=A0=C2=A0 by 0x4054292: start (pthread_create.c:145)
=3D=3D70339= =3D=3D=C2=A0=C2=A0=C2=A0 by 0x4053071: ??? (clone.s:21)
=3D=3D70339=3D= =3D=C2=A0=C2=A0=C2=A0 by 0x4053071: ??? (clone.s:21)
=3D=3D70339=3D=3D= =C2=A0=C2=A0=C2=A0 by 0x4053071: ??? (clone.s:21)
=3D=3D70339=3D=3D=C2= =A0=C2=A0=C2=A0 by 0x4053071: ??? (clone.s:21)
=3D=3D70339=3D=3D=C2=A0= =C2=A0=C2=A0 by 0x4053071: ??? (clone.s:21)
=3D=3D70339=3D=3D=C2=A0=C2= =A0=C2=A0 by 0x4053071: ??? (clone.s:21)
=3D=3D70339=3D=3D=C2=A0=C2=A0= =C2=A0 by 0x4053071: ??? (clone.s:21)
=3D=3D70339=3D=3D=C2=A0=C2=A0=C2= =A0 by 0x4053071: ??? (clone.s:21)
=3D=3D70339=3D=3D=C2=A0=C2=A0=C2=A0 b= y 0x4053071: ??? (clone.s:21)
=3D=3D70339=3D=3D=C2=A0 If you believe thi= s happened as a result of a stack
=3D=3D70339=3D=3D=C2=A0 overflow in yo= ur program's main thread (unlikely but
=3D=3D70339=3D=3D=C2=A0 possi= ble), you can try to increase the size of the
=3D=3D70339=3D=3D=C2=A0 ma= in thread stack using the --main-stacksize=3D flag.
=3D=3D70339=3D=3D=C2= =A0 The main thread stack size used in this run was 8388608.
=3D=3D70339= =3D=3D
=3D=3D70339=3D=3D HEAP SUMMARY:
=3D=3D70339=3D=3D=C2=A0=C2=A0= =C2=A0=C2=A0 in use at exit: 81,051 bytes in 9 blocks
=3D=3D70339=3D=3D= =C2=A0=C2=A0 total heap usage: 9 allocs, 3 frees, 81,051 bytes allocated=3D=3D70339=3D=3D
=3D=3D70339=3D=3D LEAK SUMMARY:
=3D=3D70339=3D=3D= =C2=A0=C2=A0=C2=A0 definitely lost: 0 bytes in 0 blocks
=3D=3D70339=3D= =3D=C2=A0=C2=A0=C2=A0 indirectly lost: 0 bytes in 0 blocks
=3D=3D70339= =3D=3D=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 possibly lost: 0 bytes in 0 blocks
= =3D=3D70339=3D=3D=C2=A0=C2=A0=C2=A0 still reachable: 81,051 bytes in 9 bloc= ks
=3D=3D70339=3D=3D=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sup= pressed: 0 bytes in 0 blocks
=3D=3D70339=3D=3D Rerun with --leak-check= =3Dfull to see details of leaked memory
=3D=3D70339=3D=3D
=3D=3D7033= 9=3D=3D For counts of detected and suppressed errors, rerun with: -v
=3D= =3D70339=3D=3D ERROR SUMMARY: 3 errors from 1 contexts (suppressed: 0 from = 0)
Killed


=
sob., 8 gru 2018 o 17:18=C2=A0Florian Weimer <fweimer@redhat.com> napisa=C5=82(a):<= br>
* Rich Felker:
> On Fri, Dec 07, 2018 at 09:06:18PM +0100, Florian Weimer wrote:
>> * Rich Felker:
>>
>> > I don't think so. I'm concerned that it's a stack= overflow, and that
>> > somehow the kernel folks have managed to break the MINSIGSTKS= Z ABI.
>>
>> Probably:
>>
>>=C2=A0 =C2=A0<https://sourceware.o= rg/bugzilla/show_bug.cgi?id=3D20305>
>>=C2=A0 =C2=A0<https://sourceware.o= rg/bugzilla/show_bug.cgi?id=3D22636>
>>
>> It's a nasty CPU backwards compatibility problem.=C2=A0 Some o= f the
>> suggestions I made to work around this are simply wrong; don't= take them
>> too seriously.
>>
>> Nowadays, the kernel has a way to disable the %zmm registers, but = it
>> unfortunately does not reduce the save area size.
>
> How large is the saved context with the %zmm junk? I measured just
> ~768 bytes on normal x86_64 without it, and since 2048 is rounded up > to a whole page (4096), overflow should not happen until the signal > context is something like 3.5k (allowing ~512 bytes for TCB (~128) and=
> 2 simple call frames).

I wrote a test to do some measurements:

=C2=A0 <https://sourceware.org/ml/libc-= alpha/2018-12/msg00271.html>

The signal handler context is quite large on x86-64 with AVX-512F,
indeed around 3.5 KiB.=C2=A0 It is even larger on ppc64 and ppc64el
(~4.5 KiB), which I find somewhat surprising.

The cancellation test also includes stack usage from the libgcc
unwinder.=C2=A0 Its stack usage likely differs between versions, so I shoul= d
have included that in the reported results.

Thanks,
Florian
--00000000000074439c057ca743dc--