Need to zero pads in msghdr

Need to zero pads in msghdr

[Andrei Vagin]

Hi,

In this patch
http://git.musl-libc.org/cgit/musl/commit/arch/x86_64/bits/socket.h?id=7168790763cdeb794df52be6e3b39fbb021c5a64
you suppose that the kernel ignores the upper 32 bits of msg_iovlen,
but it doesn't, so pads in msghdr structures have to be zeroed before
calling sendmsg and recvmsg syscalls.

https://github.com/xemul/criu/issues/276

Thanks,
Andrei

Re: Need to zero pads in msghdr

[Rich Felker]

On Wed, Jan 25, 2017 at 08:42:52AM -0800, Andrei Vagin wrote:
> Hi,
> 
> In this patch
> http://git.musl-libc.org/cgit/musl/commit/arch/x86_64/bits/socket.h?id=7168790763cdeb794df52be6e3b39fbb021c5a64
> you suppose that the kernel ignores the upper 32 bits of msg_iovlen,
> but it doesn't, so pads in msghdr structures have to be zeroed before
> calling sendmsg and recvmsg syscalls.
> 
> https://github.com/xemul/criu/issues/276

sendmsg and recvmsg perform this zeroing. If it's not working there
must be some more subtle cause.

Rich

Re: Need to zero pads in msghdr

[Andrei Vagin]

On Wed, Jan 25, 2017 at 8:42 AM, Andrei Vagin <avagin@gmail.com> wrote:
> Hi,
>
> In this patch
> http://git.musl-libc.org/cgit/musl/commit/arch/x86_64/bits/socket.h?id=7168790763cdeb794df52be6e3b39fbb021c5a64
> you suppose that the kernel ignores the upper 32 bits of msg_iovlen,
> but it doesn't, so pads in msghdr structures have to be zeroed before
> calling sendmsg and recvmsg syscalls.

Actually the problem is a bit different. In CRIU we use the msghdr
structure from musl-libc, but in some cases we have to call raw system
calls. We don't expect to have pads in structures and so we don't zero
them.

I understand that it is not a bug for many users of mucl-libc, but in
case of CRIU we see this issue.


> https://github.com/xemul/criu/issues/276
>
> Thanks,
> Andrei

Re: Re: Need to zero pads in msghdr

[Szabolcs Nagy]

* Andrei Vagin <avagin@gmail.com> [2017-01-25 10:56:22 -0800]:
> On Wed, Jan 25, 2017 at 8:42 AM, Andrei Vagin <avagin@gmail.com> wrote:
> > In this patch
> > http://git.musl-libc.org/cgit/musl/commit/arch/x86_64/bits/socket.h?id=7168790763cdeb794df52be6e3b39fbb021c5a64
> > you suppose that the kernel ignores the upper 32 bits of msg_iovlen,
> > but it doesn't, so pads in msghdr structures have to be zeroed before
> > calling sendmsg and recvmsg syscalls.
> 
> Actually the problem is a bit different. In CRIU we use the msghdr
> structure from musl-libc, but in some cases we have to call raw system
> calls. We don't expect to have pads in structures and so we don't zero
> them.

why do you need a raw syscall?

(i think if you do raw syscalls you should use
your own linux syscall wrappers including typedefs
and macro defines, not libc ones, because the libc
can and does do all sorts of remapping of things to
workaround various mismatches between the posix
library api it provides and the linux syscall abi)

Re: Re: Need to zero pads in msghdr

[Andrei Vagin]

On Wed, Jan 25, 2017 at 11:40 AM, Szabolcs Nagy <nsz@port70.net> wrote:
> * Andrei Vagin <avagin@gmail.com> [2017-01-25 10:56:22 -0800]:
>> On Wed, Jan 25, 2017 at 8:42 AM, Andrei Vagin <avagin@gmail.com> wrote:
>> > In this patch
>> > http://git.musl-libc.org/cgit/musl/commit/arch/x86_64/bits/socket.h?id=7168790763cdeb794df52be6e3b39fbb021c5a64
>> > you suppose that the kernel ignores the upper 32 bits of msg_iovlen,
>> > but it doesn't, so pads in msghdr structures have to be zeroed before
>> > calling sendmsg and recvmsg syscalls.
>>
>> Actually the problem is a bit different. In CRIU we use the msghdr
>> structure from musl-libc, but in some cases we have to call raw system
>> calls. We don't expect to have pads in structures and so we don't zero
>> them.
>
> why do you need a raw syscall?

We inject our code into processes which are going to be dumped:
https://criu.org/Parasite_code

And on restore we have to unmap old libc to restore process mappings.

>
> (i think if you do raw syscalls you should use
> your own linux syscall wrappers including typedefs
> and macro defines, not libc ones, because the libc
> can and does do all sorts of remapping of things to
> workaround various mismatches between the posix
> library api it provides and the linux syscall abi)

We know about this risk, but before this day we executed out test for
glibc and it worked for everyone. Now we need think how to resolve the
problem.

Re: Re: Need to zero pads in msghdr

[Rich Felker]

On Wed, Jan 25, 2017 at 11:46:44AM -0800, Andrei Vagin wrote:
> On Wed, Jan 25, 2017 at 11:40 AM, Szabolcs Nagy <nsz@port70.net> wrote:
> > * Andrei Vagin <avagin@gmail.com> [2017-01-25 10:56:22 -0800]:
> >> On Wed, Jan 25, 2017 at 8:42 AM, Andrei Vagin <avagin@gmail.com> wrote:
> >> > In this patch
> >> > http://git.musl-libc.org/cgit/musl/commit/arch/x86_64/bits/socket.h?id=7168790763cdeb794df52be6e3b39fbb021c5a64
> >> > you suppose that the kernel ignores the upper 32 bits of msg_iovlen,
> >> > but it doesn't, so pads in msghdr structures have to be zeroed before
> >> > calling sendmsg and recvmsg syscalls.
> >>
> >> Actually the problem is a bit different. In CRIU we use the msghdr
> >> structure from musl-libc, but in some cases we have to call raw system
> >> calls. We don't expect to have pads in structures and so we don't zero
> >> them.
> >
> > why do you need a raw syscall?
> 
> We inject our code into processes which are going to be dumped:
> https://criu.org/Parasite_code
> 
> And on restore we have to unmap old libc to restore process mappings.
> 
> >
> > (i think if you do raw syscalls you should use
> > your own linux syscall wrappers including typedefs
> > and macro defines, not libc ones, because the libc
> > can and does do all sorts of remapping of things to
> > workaround various mismatches between the posix
> > library api it provides and the linux syscall abi)
> 
> We know about this risk, but before this day we executed out test for
> glibc and it worked for everyone. Now we need think how to resolve the
> problem.

While you may hit problems elsewhere, I think there's a clean solution
here, just pre-zeroing the structure so that any possible padding is
zero. It's unfortunate that this is needed but it comes from the
kernel getting the structure definition wrong.

Rich

Re: Re: Need to zero pads in msghdr

[Szabolcs Nagy]

* Andrei Vagin <avagin@gmail.com> [2017-01-25 11:46:44 -0800]:
> On Wed, Jan 25, 2017 at 11:40 AM, Szabolcs Nagy <nsz@port70.net> wrote:
> > why do you need a raw syscall?
> 
> We inject our code into processes which are going to be dumped:
> https://criu.org/Parasite_code
> 
> And on restore we have to unmap old libc to restore process mappings.

if you static link to musl for the parasite then
i don't see why the syscalls have to be raw..

what you may worry about is process global
state that the libc takes control of
(libc internal signal handler, brk pointer,
doing things to fd 0/1/2, etc), but that you
cannot prevent with raw syscalls.

Re: Re: Need to zero pads in msghdr

[Rich Felker]

On Thu, Jan 26, 2017 at 12:00:46AM +0100, Szabolcs Nagy wrote:
> * Andrei Vagin <avagin@gmail.com> [2017-01-25 11:46:44 -0800]:
> > On Wed, Jan 25, 2017 at 11:40 AM, Szabolcs Nagy <nsz@port70.net> wrote:
> > > why do you need a raw syscall?
> > 
> > We inject our code into processes which are going to be dumped:
> > https://criu.org/Parasite_code
> > 
> > And on restore we have to unmap old libc to restore process mappings.
> 
> if you static link to musl for the parasite then
> i don't see why the syscalls have to be raw..
> 
> what you may worry about is process global
> state that the libc takes control of
> (libc internal signal handler, brk pointer,
> doing things to fd 0/1/2, etc), but that you
> cannot prevent with raw syscalls.

My impression is that the parasite code does not link any libc, in
which case it should be fine.

On further consideration, though, it probably makes more sense to use
kernel headers for the syscall-argument structure defs in the parasite
code if it's making direct syscalls. Using the libc headers to get
these structs will break if we ever have a musl2 abi (longterm idea,
no idea if it will ever happen) that abstracts away all the linux
kernel types behind clean, arch-independent, extensible definitions
with translation in the libc wrapper functions.

Rich