openssh and sendmsg() problem

openssh and sendmsg() problem

[orc]

Has anyone got openssh to work with privilege separation?
I tried today, openssh refused with strange reason:

error: mm_send_fd: sendmsg(4): Invalid argument

strace shows that sendmsg system call returns EINVAL:

sendmsg(5, {msg_name(0)=NULL, msg_iov(1)=[{"\0", 1}], msg_controllen=24, {cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_
type=SCM_RIGHTS, {4, 385875968}}, msg_flags=0}, 0) = -1 EINVAL (Invalid argument)

If needed, I can send full strace log.

Re: openssh and sendmsg() problem

[Szabolcs Nagy]

* orc <orc@sibserver.ru> [2012-06-26 19:01:46 +0800]:
> strace shows that sendmsg system call returns EINVAL:
> 
> sendmsg(5, {msg_name(0)=NULL, msg_iov(1)=[{"\0", 1}], msg_controllen=24, {cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_
> type=SCM_RIGHTS, {4, 385875968}}, msg_flags=0}, 0) = -1 EINVAL (Invalid argument)
> 

src/network/sendmsg.c has an ifdef for long>int case

i don't know what it's supposed to do
you may want to change the struct msghdr definition
in arch/*/bits/socket.h
(eg to the one in your /usr/include/bits/socket.h)
and remove the ifdef from sendmsg.c

Re: openssh and sendmsg() problem

[Rich Felker]

On Tue, Jun 26, 2012 at 02:00:17PM +0200, Szabolcs Nagy wrote:
> * orc <orc@sibserver.ru> [2012-06-26 19:01:46 +0800]:
> > strace shows that sendmsg system call returns EINVAL:
> > 
> > sendmsg(5, {msg_name(0)=NULL, msg_iov(1)=[{"\0", 1}], msg_controllen=24, {cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_
> > type=SCM_RIGHTS, {4, 385875968}}, msg_flags=0}, 0) = -1 EINVAL (Invalid argument)
> > 
> 
> src/network/sendmsg.c has an ifdef for long>int case
> 
> i don't know what it's supposed to do

It's there because the kernel expects certain fields to be long but
POSIX requires them to be int. Thus the syscall wrapper has to copy
the userspace struct (with int and padding) to a kernel-format struct
(with the padding filled with 0's so it's a valid long).

> you may want to change the struct msghdr definition
> in arch/*/bits/socket.h
> (eg to the one in your /usr/include/bits/socket.h)
> and remove the ifdef from sendmsg.c

Nope, this is not valid. See above.

Rich

Re: openssh and sendmsg() problem

[orc]

On Tue, 26 Jun 2012 12:01:06 -0400
Rich Felker <dalias@aerifal.cx> wrote:

> On Tue, Jun 26, 2012 at 02:00:17PM +0200, Szabolcs Nagy wrote:
> > * orc <orc@sibserver.ru> [2012-06-26 19:01:46 +0800]:
> > > strace shows that sendmsg system call returns EINVAL:
> > > 
> > > sendmsg(5, {msg_name(0)=NULL, msg_iov(1)=[{"\0", 1}],
> > > msg_controllen=24, {cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_
> > > type=SCM_RIGHTS, {4, 385875968}}, msg_flags=0}, 0) = -1 EINVAL
> > > (Invalid argument)
> > > 
> > 
> > src/network/sendmsg.c has an ifdef for long>int case
> > 
> > i don't know what it's supposed to do
> 
> It's there because the kernel expects certain fields to be long but
> POSIX requires them to be int. Thus the syscall wrapper has to copy
> the userspace struct (with int and padding) to a kernel-format struct
> (with the padding filled with 0's so it's a valid long).

And how to deal with this? Should it be openssh workaround or fix in
musl itself?

Re: openssh and sendmsg() problem

[Rich Felker]

On Wed, Jun 27, 2012 at 01:24:41AM +0800, orc wrote:
> On Tue, 26 Jun 2012 12:01:06 -0400
> Rich Felker <dalias@aerifal.cx> wrote:
> 
> > On Tue, Jun 26, 2012 at 02:00:17PM +0200, Szabolcs Nagy wrote:
> > > * orc <orc@sibserver.ru> [2012-06-26 19:01:46 +0800]:
> > > > strace shows that sendmsg system call returns EINVAL:
> > > > 
> > > > sendmsg(5, {msg_name(0)=NULL, msg_iov(1)=[{"\0", 1}],
> > > > msg_controllen=24, {cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_
> > > > type=SCM_RIGHTS, {4, 385875968}}, msg_flags=0}, 0) = -1 EINVAL
> > > > (Invalid argument)
> > > > 
> > > 
> > > src/network/sendmsg.c has an ifdef for long>int case
> > > 
> > > i don't know what it's supposed to do
> > 
> > It's there because the kernel expects certain fields to be long but
> > POSIX requires them to be int. Thus the syscall wrapper has to copy
> > the userspace struct (with int and padding) to a kernel-format struct
> > (with the padding filled with 0's so it's a valid long).
> 
> And how to deal with this? Should it be openssh workaround or fix in
> musl itself?

There's nothing to be dealt with here; musl already works around the
kernel bug. Something is wrong, possibly in this code or elsewhere,
but I don't know what and I don't see anything obviously wrong with
the syscall. Can you compare the strace to what happens on glibc?

Rich

Re: openssh and sendmsg() problem

[orc]

On Tue, 26 Jun 2012 13:32:05 -0400
Rich Felker <dalias@aerifal.cx> wrote:

> On Wed, Jun 27, 2012 at 01:24:41AM +0800, orc wrote:
> > On Tue, 26 Jun 2012 12:01:06 -0400
> > Rich Felker <dalias@aerifal.cx> wrote:
> > 
> > > On Tue, Jun 26, 2012 at 02:00:17PM +0200, Szabolcs Nagy wrote:
> > > > * orc <orc@sibserver.ru> [2012-06-26 19:01:46 +0800]:
> > > > > strace shows that sendmsg system call returns EINVAL:
> > > > > 
> > > > > sendmsg(5, {msg_name(0)=NULL, msg_iov(1)=[{"\0", 1}],
> > > > > msg_controllen=24, {cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_
> > > > > type=SCM_RIGHTS, {4, 385875968}}, msg_flags=0}, 0) = -1 EINVAL
> > > > > (Invalid argument)
> > > > > 
> > > > 
> > > > src/network/sendmsg.c has an ifdef for long>int case
> > > > 
> > > > i don't know what it's supposed to do
> > > 
> > > It's there because the kernel expects certain fields to be long
> > > but POSIX requires them to be int. Thus the syscall wrapper has
> > > to copy the userspace struct (with int and padding) to a
> > > kernel-format struct (with the padding filled with 0's so it's a
> > > valid long).
> > 
> > And how to deal with this? Should it be openssh workaround or fix in
> > musl itself?
> 
> There's nothing to be dealt with here; musl already works around the
> kernel bug. Something is wrong, possibly in this code or elsewhere,
> but I don't know what and I don't see anything obviously wrong with
> the syscall. Can you compare the strace to what happens on glibc?
> 
> Rich

strace from glibc system on this syscall:

sendmsg(5, {msg_name(0)=NULL, msg_iov(1)=[{"\0", 1}], msg_controllen=24, {cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, {4}}, msg_flags=0}, 0) = 1
sendmsg(5, {msg_name(0)=NULL, msg_iov(1)=[{"\0", 1}], msg_controllen=24, {cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, {6}}, msg_flags=0}, 0) = 1

Re: openssh and sendmsg() problem

[John Spencer]

On 06/26/2012 07:24 PM, orc wrote:
> On Tue, 26 Jun 2012 12:01:06 -0400
> Rich Felker<dalias@aerifal.cx>  wrote:
>
>> On Tue, Jun 26, 2012 at 02:00:17PM +0200, Szabolcs Nagy wrote:
>>> * orc<orc@sibserver.ru>  [2012-06-26 19:01:46 +0800]:
>>>> strace shows that sendmsg system call returns EINVAL:
>>>>
>>>> sendmsg(5, {msg_name(0)=NULL, msg_iov(1)=[{"\0", 1}],
>>>> msg_controllen=24, {cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_
>>>> type=SCM_RIGHTS, {4, 385875968}}, msg_flags=0}, 0) = -1 EINVAL
>>>> (Invalid argument)
>>>>
>>> src/network/sendmsg.c has an ifdef for long>int case
>>>
>>> i don't know what it's supposed to do
>> It's there because the kernel expects certain fields to be long but
>> POSIX requires them to be int. Thus the syscall wrapper has to copy
>> the userspace struct (with int and padding) to a kernel-format struct
>> (with the padding filled with 0's so it's a valid long).
> And how to deal with this? Should it be openssh workaround or fix in
> musl itself?
>

you might want to try this recipe, which is known to work
https://github.com/rofl0r/sabotage/blob/master/pkg/openssh

Re: openssh and sendmsg() problem

[orc]

On Tue, 26 Jun 2012 20:32:27 +0200
John Spencer <maillist-musl@barfooze.de> wrote:

> On 06/26/2012 07:24 PM, orc wrote:
> > On Tue, 26 Jun 2012 12:01:06 -0400
> > Rich Felker<dalias@aerifal.cx>  wrote:
> >
> >> On Tue, Jun 26, 2012 at 02:00:17PM +0200, Szabolcs Nagy wrote:
> >>> * orc<orc@sibserver.ru>  [2012-06-26 19:01:46 +0800]:
> >>>> strace shows that sendmsg system call returns EINVAL:
> >>>>
> >>>> sendmsg(5, {msg_name(0)=NULL, msg_iov(1)=[{"\0", 1}],
> >>>> msg_controllen=24, {cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_
> >>>> type=SCM_RIGHTS, {4, 385875968}}, msg_flags=0}, 0) = -1 EINVAL
> >>>> (Invalid argument)
> >>>>
> >>> src/network/sendmsg.c has an ifdef for long>int case
> >>>
> >>> i don't know what it's supposed to do
> >> It's there because the kernel expects certain fields to be long but
> >> POSIX requires them to be int. Thus the syscall wrapper has to copy
> >> the userspace struct (with int and padding) to a kernel-format
> >> struct (with the padding filled with 0's so it's a valid long).
> > And how to deal with this? Should it be openssh workaround or fix in
> > musl itself?
> >
> 
> you might want to try this recipe, which is known to work
> https://github.com/rofl0r/sabotage/blob/master/pkg/openssh
> 

Problem still persists.

Re: openssh and sendmsg() problem

[Rich Felker]

On Wed, Jun 27, 2012 at 02:16:40AM +0800, orc wrote:
> On Tue, 26 Jun 2012 13:32:05 -0400
> Rich Felker <dalias@aerifal.cx> wrote:
> 
> > On Wed, Jun 27, 2012 at 01:24:41AM +0800, orc wrote:
> > > On Tue, 26 Jun 2012 12:01:06 -0400
> > > Rich Felker <dalias@aerifal.cx> wrote:
> > > 
> > > > On Tue, Jun 26, 2012 at 02:00:17PM +0200, Szabolcs Nagy wrote:
> > > > > * orc <orc@sibserver.ru> [2012-06-26 19:01:46 +0800]:
> > > > > > strace shows that sendmsg system call returns EINVAL:
> > > > > > 
> > > > > > sendmsg(5, {msg_name(0)=NULL, msg_iov(1)=[{"\0", 1}],
> > > > > > msg_controllen=24, {cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_
> > > > > > type=SCM_RIGHTS, {4, 385875968}}, msg_flags=0}, 0) = -1 EINVAL
> > > > > > (Invalid argument)
> > > > > > 
> 
> strace from glibc system on this syscall:
> 
> sendmsg(5, {msg_name(0)=NULL, msg_iov(1)=[{"\0", 1}], msg_controllen=24, {cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, {4}}, msg_flags=0}, 0) = 1
> sendmsg(5, {msg_name(0)=NULL, msg_iov(1)=[{"\0", 1}], msg_controllen=24, {cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, {6}}, msg_flags=0}, 0) = 1

I don't quite understand this from the strace, but it looks like
there's an issue with the size of cmsg structures on 64-bit machines
due to another buggy kernel socklen_t/size_t mismatch. I'm not sure
how it can be worked around, but musl's failure to work around it is a
flaw that needs to be fixed. glibc just violates POSIX and defines
the wrong type.

Rich

Re: openssh and sendmsg() problem

[orc]

On Tue, 26 Jun 2012 15:08:23 -0400
Rich Felker <dalias@aerifal.cx> wrote:

> On Wed, Jun 27, 2012 at 02:16:40AM +0800, orc wrote:
> > On Tue, 26 Jun 2012 13:32:05 -0400
> > Rich Felker <dalias@aerifal.cx> wrote:
> > 
> > > On Wed, Jun 27, 2012 at 01:24:41AM +0800, orc wrote:
> > > > On Tue, 26 Jun 2012 12:01:06 -0400
> > > > Rich Felker <dalias@aerifal.cx> wrote:
> > > > 
> > > > > On Tue, Jun 26, 2012 at 02:00:17PM +0200, Szabolcs Nagy wrote:
> > > > > > * orc <orc@sibserver.ru> [2012-06-26 19:01:46 +0800]:
> > > > > > > strace shows that sendmsg system call returns EINVAL:
> > > > > > > 
> > > > > > > sendmsg(5, {msg_name(0)=NULL, msg_iov(1)=[{"\0", 1}],
> > > > > > > msg_controllen=24, {cmsg_len=20, cmsg_level=SOL_SOCKET,
> > > > > > > cmsg_ type=SCM_RIGHTS, {4, 385875968}}, msg_flags=0}, 0)
> > > > > > > = -1 EINVAL (Invalid argument)
> > > > > > > 
> > 
> > strace from glibc system on this syscall:
> > 
> > sendmsg(5, {msg_name(0)=NULL, msg_iov(1)=[{"\0", 1}],
> > msg_controllen=24, {cmsg_len=20, cmsg_level=SOL_SOCKET,
> > cmsg_type=SCM_RIGHTS, {4}}, msg_flags=0}, 0) = 1 sendmsg(5,
> > {msg_name(0)=NULL, msg_iov(1)=[{"\0", 1}], msg_controllen=24,
> > {cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, {6}},
> > msg_flags=0}, 0) = 1
> 
> I don't quite understand this from the strace, but it looks like
> there's an issue with the size of cmsg structures on 64-bit machines
> due to another buggy kernel socklen_t/size_t mismatch. I'm not sure
> how it can be worked around, but musl's failure to work around it is a
> flaw that needs to be fixed. glibc just violates POSIX and defines
> the wrong type.
> 
> Rich

And kernel too.

Both they define incompatible msghdr and cmsghdr structs.
The problem sits in cmsghdr struct, changing it's first member type to
'size_t' solves openssh issue.

Re: openssh and sendmsg() problem

[Szabolcs Nagy]

* orc <orc@sibserver.ru> [2012-06-27 03:39:34 +0800]:
> On Tue, 26 Jun 2012 15:08:23 -0400
> Rich Felker <dalias@aerifal.cx> wrote:
> > On Wed, Jun 27, 2012 at 02:16:40AM +0800, orc wrote:
> > > > > On Tue, 26 Jun 2012 12:01:06 -0400
> > > > > Rich Felker <dalias@aerifal.cx> wrote:
> > > > > > > * orc <orc@sibserver.ru> [2012-06-26 19:01:46 +0800]:
> > > > > > > > strace shows that sendmsg system call returns EINVAL:
> > > > > > > > 
> > > > > > > > sendmsg(5, {msg_name(0)=NULL, msg_iov(1)=[{"\0", 1}],
> > > > > > > > msg_controllen=24, {cmsg_len=20, cmsg_level=SOL_SOCKET,
> > > > > > > > cmsg_ type=SCM_RIGHTS, {4, 385875968}}, msg_flags=0}, 0)
> > > > > > > > = -1 EINVAL (Invalid argument)
> > > > > > > > 
> > > 
> > > strace from glibc system on this syscall:
> > > 
> > > sendmsg(5, {msg_name(0)=NULL, msg_iov(1)=[{"\0", 1}],
> > > msg_controllen=24, {cmsg_len=20, cmsg_level=SOL_SOCKET,
> > > cmsg_type=SCM_RIGHTS, {4}}, msg_flags=0}, 0) = 1 sendmsg(5,
> > > {msg_name(0)=NULL, msg_iov(1)=[{"\0", 1}], msg_controllen=24,
> > > {cmsg_len=20, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, {6}},
> > > msg_flags=0}, 0) = 1
> > 
> > I don't quite understand this from the strace, but it looks like
> > there's an issue with the size of cmsg structures on 64-bit machines
> > due to another buggy kernel socklen_t/size_t mismatch. I'm not sure
> > how it can be worked around, but musl's failure to work around it is a
> > flaw that needs to be fixed. glibc just violates POSIX and defines
> > the wrong type.
> > 
> > Rich
> 
> And kernel too.
> 
> Both they define incompatible msghdr and cmsghdr structs.
> The problem sits in cmsghdr struct, changing it's first member type to
> 'size_t' solves openssh issue.

so the issue is that the kernel and glibc uses size_t for socklen_t
in struct msghdr and struct cmsghdr as well

musl fixes msghdr but not cmsghdr
(by using int + padding)

where is it specified in posix that socklen_t should be unsigned int?
it seems using size_t would make things easier..

i only see a recommendation that
"To forestall portability problems, it is recommended that applications
not use values larger than 2^31 -1 for the socklen_t type."

Re: openssh and sendmsg() problem

[Rich Felker]

On Tue, Jun 26, 2012 at 10:40:16PM +0200, Szabolcs Nagy wrote:
> > And kernel too.
> > 
> > Both they define incompatible msghdr and cmsghdr structs.
> > The problem sits in cmsghdr struct, changing it's first member type to
> > 'size_t' solves openssh issue.
> 
> so the issue is that the kernel and glibc uses size_t for socklen_t
> in struct msghdr and struct cmsghdr as well
> 
> musl fixes msghdr but not cmsghdr
> (by using int + padding)
> 
> where is it specified in posix that socklen_t should be unsigned int?
> it seems using size_t would make things easier..

Yes it would be easier, but unfortunately it's part of the LSB types
ABI and presumably the Linux kernel ABI...

Rich

Re: openssh and sendmsg() problem

[orc]

On Tue, 26 Jun 2012 17:13:19 -0400
Rich Felker <dalias@aerifal.cx> wrote:

> On Tue, Jun 26, 2012 at 10:40:16PM +0200, Szabolcs Nagy wrote:
> > > And kernel too.
> > > 
> > > Both they define incompatible msghdr and cmsghdr structs.
> > > The problem sits in cmsghdr struct, changing it's first member
> > > type to 'size_t' solves openssh issue.
> > 
> > so the issue is that the kernel and glibc uses size_t for socklen_t
> > in struct msghdr and struct cmsghdr as well
> > 
> > musl fixes msghdr but not cmsghdr
> > (by using int + padding)
> > 
> > where is it specified in posix that socklen_t should be unsigned
> > int? it seems using size_t would make things easier..
> 
> Yes it would be easier, but unfortunately it's part of the LSB types
> ABI and presumably the Linux kernel ABI...
> 
> Rich

As a temporary workaround, can I safely change the type until final fix?