Re: a problem with linux 2.6.11 and sa

Re: a problem with linux 2.6.11 and sa

[George Georgalis]

On Tue, Mar 08, 2005 at 01:37:03PM +0000, Nix wrote:
>On Thu, 3 Mar 2005, George Georgalis uttered the following:
>> I recall a problem a while back with a pipe from
>> /proc/kmsg that was sent by root to a program with a
>> user uid. The fix was to run the logging program as
>> root. Has that protected pipe method been extended
>> since 2.6.8.1?
>
>The entire implementation of pipes has been radically revised between
>2.6.10 and 2.6.11: see, e.g., <http://lwn.net/Articles/118750/> and
><http://lwn.net/Articles/119682/>.
>
>Bugs have been spotted in this area in 2.6.10: this may be
>another one.

Thanks, my issue is clearly between 2.6.10 and 2.6.11; though I won't be
able to drill down anything more specific, for a while. The links
do look relevant but I cannot say for sure.

Here's what I'm doing that is broken. I use tcpserver (functionally
similar to inetd) to receive an incoming smtp connection. While the
smtp session is still open, the message is piped to a temp file which
is then scanned for spam, if it passes the temp file is piped to my
local delivery program. If it doesn't pass the spam test or the delivery
program fails (disk full etc), the respective error code, if any,
is passed to tcpserver. The corresponding accepted, temporary reject or
permanently reject signal is passed to the remote sender.

The temp file is then removed or, for spam, it is cataloged for
statistics and/or abuse reporting. An additional copy is kept in a
traditional maildir to check for false positives.

#!/bin/bash
# exit 31 = permanently refuse
# exit 71 = temporarily refuse
# pwd is /var/qmail
echo $0 # for the logs
scq="spamc-queue" # a maildir with qmaild write perms
tmp="${scq}/`safecat "${scq}/tmp" "${scq}" </dev/stdin`" \
	|| { echo "Error $?"; exit 71; } # put the pipeline to disk, if possible
	# ${scq}/tmp is a temp for this function ${scq} is temp for this program
score=`spamc -x -c <"$tmp"` # score it with spamd
sce=$?
echo $score # for the logs
case $sce in
0) # ham
	host=`cat control/me`
	formail -f -A "X-spamc: ${score} by ${host}; `date -R`" \
		-A "X-tcpremoteip: $TCPREMOTEIP" <"$tmp" \
		| bin/qmail-queue # mark it and pass to the regular queue
	qqe=$?
	rm "$tmp"
	exit $qqe # return whatever qmail-queue exits as
;;
1) # spam 
	sipd="$scq/IP/`echo $TCPREMOTEIP | sed 's|\.|/|g'`"
	mkdir -p $sipd/{new,cur,tmp} # make a spam ip maildir, if needed
	printf "$TCPREMOTEIP\t`date`\n" >>$sipd/date # keep track of when they came
	maildir "${sipd}" >/dev/null <"$tmp" # keep a copy for reporting
	maildir "${scq}"  >/dev/null <"$tmp" # save it to verify no falseys
	rm "$tmp"
	exit 31
;;
*) # spamc error, 
	echo "$0 error, spamc exit $sce"
	exit 71
esac
exit 81 # Internal bug



>If you can reproduce it consistently, *please* report
>this to the linux-kernel list!

I did, but have had no response to my followup:

Date: Fri, 4 Mar 2005 15:58:43 -0500
Subject: Re: problem with linux 2.6.11 and sa


>(I don't see what you mean by `a pipe rom /proc/kmsg', though:
>pipes connect processes, not files. File redirections are
>quite different and should work unchanged in 2.6.11.)


An interesting technique that allows a program (such as a log writer)
to run as an unprivileged user, while receiving privileged data. (taken
almost verbatim from Gerrit Pape's socklog)

#!/bin/sh
exec </proc/kmsg
exec 2>&1
exec softlimit -m 2000000 setuidgid nobody socklog ucspi

This script, run by root takes its stdin from /proc/kmsg then combines
its stdout and stderr, and exec-switches to the socklog program run
as an ucspi application listening to the domain stream socket, as
nobody:nogroup, with memory consumption limited to 2Mb. (and sends
log to stdout)

It worked flawlessly until several kernel revs back when the kernel
started protecting kmsg and wouldn't allow the user program to receive
it, result: nothing sent to the logging program and no error. The fix
was to run socklog as root instead of nobody.

// George



-- 
George Georgalis, systems architect, administrator Linux BSD IXOYE
http://galis.org/george/ cell:646-331-2027 mailto:george@galis.org

Re: a problem with linux 2.6.11 and sa

[George Georgalis]

On Tue, Mar 08, 2005 at 11:58:14AM -0500, George Georgalis wrote:
>On Tue, Mar 08, 2005 at 01:37:03PM +0000, Nix wrote:
>>On Thu, 3 Mar 2005, George Georgalis uttered the following:
>>> I recall a problem a while back with a pipe from
>>> /proc/kmsg that was sent by root to a program with a
>>> user uid. The fix was to run the logging program as
>>> root. Has that protected pipe method been extended
>>> since 2.6.8.1?
>>
>>The entire implementation of pipes has been radically revised between
>>2.6.10 and 2.6.11: see, e.g., <http://lwn.net/Articles/118750/> and
>><http://lwn.net/Articles/119682/>.
>>
>>Bugs have been spotted in this area in 2.6.10: this may be
>>another one.
>
>Thanks, my issue is clearly between 2.6.10 and 2.6.11; though I won't be
>able to drill down anything more specific, for a while. The links
>do look relevant but I cannot say for sure.

My last post didn't actually describe what the problem is, which is
spamassassin always scores 0/0 under 2.6.11 but functions properly
(scoring x/5) under 2.6.10.

More details are in the thread of this post.
http://lkml.org/lkml/2005/3/3/513

// George

-- 
George Georgalis, systems architect, administrator Linux BSD IXOYE
http://galis.org/george/ cell:646-331-2027 mailto:george@galis.org

Re: a problem with linux 2.6.11 and sa

[George Georgalis]

On Tue, 8 Mar 2005 12:19:53 -0500, George Georgalis <george@galis.org> wrote:
> On Tue, Mar 08, 2005 at 11:58:14AM -0500, George Georgalis wrote:
> >On Tue, Mar 08, 2005 at 01:37:03PM +0000, Nix wrote:
> >>On Thu, 3 Mar 2005, George Georgalis uttered the following:
> >>> I recall a problem a while back with a pipe from
> >>> /proc/kmsg that was sent by root to a program with a
> >>> user uid. The fix was to run the logging program as
> >>> root. Has that protected pipe method been extended
> >>> since 2.6.8.1?
> >>
> >>The entire implementation of pipes has been radically revised between
> >>2.6.10 and 2.6.11: see, e.g., <http://lwn.net/Articles/118750/> and
> >><http://lwn.net/Articles/119682/>.
> >>
> >>Bugs have been spotted in this area in 2.6.10: this may be
> >>another one.
> >
> >Thanks, my issue is clearly between 2.6.10 and 2.6.11; though I won't be
> >able to drill down anything more specific, for a while. The links
> >do look relevant but I cannot say for sure.

Here is a problem with 2.6.10:

while read file; do mplayer $file ; done <mediafiles.txt

or

tail -n93  mediafiles.txt | while read file; do mplayer $file ; done

for each file path in that text file I get:

Failed to open /dev/rtc: Permission denied (it should be readable by the user.)

In addition the audio pcm level is set to zero (presumably by mplayer).

This does work:
for file in `cat mediafiles.txt`; do mplayer $file ; done

but discovering and fixing code now broke will be unpleasent.
What exactly is going on? 

// George

-- 
George Georgalis, systems architect, administrator Linux BSD IXOYE
http://galis.org/george/ cell:646-331-2027 mailto:george@galis.org

Re: a problem with linux 2.6.11 and sa

[Andre Tomt]

George Georgalis wrote:
> Here is a problem with 2.6.10:
> 
> while read file; do mplayer $file ; done <mediafiles.txt
> 
> or
> 
> tail -n93  mediafiles.txt | while read file; do mplayer $file ; done
> 
> for each file path in that text file I get:
> 
> Failed to open /dev/rtc: Permission denied (it should be readable by the user.)

^- This is also emitted by mplayer. It usually does this on any standard 
system.

> In addition the audio pcm level is set to zero (presumably by mplayer).
> 
> This does work:
> for file in `cat mediafiles.txt`; do mplayer $file ; done
> 
> but discovering and fixing code now broke will be unpleasent.
> What exactly is going on? 
> 
> // George
> 

Re: a problem with linux 2.6.11 and sa

[Nix]

On Tue, 8 Mar 2005, George Georgalis announced authoritatively:
> Here's what I'm doing that is broken. I use tcpserver (functionally
> similar to inetd) to receive an incoming smtp connection. While the
> smtp session is still open, the message is piped to a temp file which
> is then scanned for spam, if it passes the temp file is piped to my

Both of these sound like redirection, not piping.

>>(I don't see what you mean by `a pipe rom /proc/kmsg', though:
>>pipes connect processes, not files. File redirections are
>>quite different and should work unchanged in 2.6.11.)
> 
> An interesting technique that allows a program (such as a log writer)
> to run as an unprivileged user, while receiving privileged data. (taken
> almost verbatim from Gerrit Pape's socklog)
> 
> #!/bin/sh
> exec </proc/kmsg
> exec 2>&1
> exec softlimit -m 2000000 setuidgid nobody socklog ucspi
> 
> This script, run by root takes its stdin from /proc/kmsg then combines
> its stdout and stderr, and exec-switches to the socklog program run
> as an ucspi application listening to the domain stream socket, as
> nobody:nogroup, with memory consumption limited to 2Mb. (and sends
> log to stdout)

This is definitely redirection, not piping. As far as I know the
implementation of redirection in the kernel remains unchanged: certainly
the need to buffer piped data doesn't exist in this case, and since the
redesign was of the buffering, this is probably not your problem :)

> It worked flawlessly until several kernel revs back when the kernel
> started protecting kmsg and wouldn't allow the user program to receive
> it,

Indeed.

>       result: nothing sent to the logging program and no error. The fix
> was to run socklog as root instead of nobody.

You should be able to open it as root and read from it as another user:
i.e., your technique above shouldn't break. (I'd hope.)

-- 
> ...Hires Root Beer...
What we need these days is a stable, fast, anti-aliased root beer
with dynamic shading. Not that you can let just anybody have root.
 --- John M. Ford

Re: a problem with linux 2.6.11 and sa

[George Georgalis]

On Wed, Mar 09, 2005 at 01:06:11PM +0000, Nix wrote:

>> An interesting technique that allows a program (such as a log writer)
>> to run as an unprivileged user, while receiving privileged data. (taken
>> almost verbatim from Gerrit Pape's socklog)
>> 
>> #!/bin/sh
>> exec </proc/kmsg
>> exec 2>&1
>> exec softlimit -m 2000000 setuidgid nobody socklog ucspi
>> 
>> This script, run by root takes its stdin from /proc/kmsg then combines
>> its stdout and stderr, and exec-switches to the socklog program run
>> as an ucspi application listening to the domain stream socket, as
>> nobody:nogroup, with memory consumption limited to 2Mb. (and sends
>> log to stdout)
>
>This is definitely redirection, not piping. As far as I know the
>implementation of redirection in the kernel remains unchanged: certainly
>the need to buffer piped data doesn't exist in this case, and since the
>redesign was of the buffering, this is probably not your problem :)
>
>> It worked flawlessly until several kernel revs back when the kernel
>> started protecting kmsg and wouldn't allow the user program to receive
>> it,
>
>Indeed.
>
>>       result: nothing sent to the logging program and no error. The fix
>> was to run socklog as root instead of nobody.
>
>You should be able to open it as root and read from it as another user:
>i.e., your technique above shouldn't break. (I'd hope.)

Here is a nice proof that kmsg did become a problem around 2.6.0
http://article.gmane.org/gmane.comp.misc.pape.general/595
http://thread.gmane.org/gmane.comp.misc.pape.general/590


It (Gerrit Pape's technique) very defiantly stopped working a few revs
back (2.6.7?). I'm seeing a similar failed read from /dev/rtc and
mplayer with 2.6.10, now too.

http://lkml.org/lkml/2005/3/8/226

while read file; do mplayer $file ; done <mediafiles.txt

Failed to open /dev/rtc: Permission denied

for file in `cat mediafiles.txt`; do mplayer $file ; done

works.

// George

-- 
George Georgalis, systems architect, administrator Linux BSD IXOYE
http://galis.org/george/ cell:646-331-2027 mailto:george@galis.org

Re: a problem with linux 2.6.11 and sa

[Paul Jarc]

"George Georgalis" <george@galis.org> wrote:
> It (Gerrit Pape's technique) very defiantly stopped working a few revs
> back (2.6.7?). I'm seeing a similar failed read from /dev/rtc and
> mplayer with 2.6.10, now too.

The /proc/kmsg problem happens because the kernel now checks for
permission at read() instead of open().  The /dev/rtc problem seems to
be a different beast.

> while read file; do mplayer $file ; done <mediafiles.txt
>
> Failed to open /dev/rtc: Permission denied
>
> for file in `cat mediafiles.txt`; do mplayer $file ; done
>
> works.

To simplify, what about these two:
mplayer foo.mpg
mplayer foo.mpg < mediafiles.txt

You might try strace'ing both cases and see how they compare.


paul

Re: a problem with linux 2.6.11 and sa

[Nix]

On Wed, 09 Mar 2005, Paul Jarc uttered the following:
> "George Georgalis" <george@galis.org> wrote:
>> It (Gerrit Pape's technique) very defiantly stopped working a few revs
>> back (2.6.7?). I'm seeing a similar failed read from /dev/rtc and
>> mplayer with 2.6.10, now too.
> 
> The /proc/kmsg problem happens because the kernel now checks for
> permission at read() instead of open().

Am I the only person who thinks `eee-ick, how utterly non-POSIX' at
that?

-- 
> ...Hires Root Beer...
What we need these days is a stable, fast, anti-aliased root beer
with dynamic shading. Not that you can let just anybody have root.
 --- John M. Ford

Re: a problem with linux 2.6.11 and sa

[George Georgalis]

On Wed, Mar 09, 2005 at 06:28:35PM -0500, Paul Jarc wrote:
>"George Georgalis" <george@galis.org> wrote:
>> It (Gerrit Pape's technique) very defiantly stopped working a few revs
>> back (2.6.7?). I'm seeing a similar failed read from /dev/rtc and
>> mplayer with 2.6.10, now too.
>
>The /proc/kmsg problem happens because the kernel now checks for
>permission at read() instead of open().  The /dev/rtc problem seems to
>be a different beast.

Thanks for the kmsg clairfication, Paul.

>> while read file; do mplayer $file ; done <mediafiles.txt
>>
>> Failed to open /dev/rtc: Permission denied
>>
>> for file in `cat mediafiles.txt`; do mplayer $file ; done
>>
>> works.
>
>To simplify, what about these two:
>mplayer foo.mpg
>mplayer foo.mpg < mediafiles.txt
>
>You might try strace'ing both cases and see how they compare.

The particular host does not have X support so mpg is out.
I'm not sure that that test would work as mplayer requires filenames
as command arguments not stdin (exclusivly, I think); my guess
is mplayer would try to decode stdin.

this works fine
mplayer `cat zz.mtest `

Then I tried
mplayer /dev/stdin <zz.mtest

I got
Failed to open /dev/rtc: Permission denied (it should be readable by the user.)

so what the heck, I changed it...
$ ls -l /dev/rtc 
crw-rw----    1 root     root      10, 135 Mar 14  2002 /dev/rtc
chmod o+r /dev/rtc

Then I tried
while read file; do mplayer "$file" ; done <zz.mtest

and got
Linux RTC init error in ioctl (rtc_irqp_set 1024): Permission denied
Try adding "echo 1024 > /proc/sys/dev/rtc/max-user-freq" to your system startup 
scripts.

the file almost played though...
Playing /usr/nfs/sandbox/media/audio/_the-party-has-just-begun/Lebanese_Blonde.ogg.
Ogg file format detected.
...

But it seemed to take keyboard commands from the binary
No bind found for key _                                                         
A:   0.1 (00.1) ??,?%                                                           
No bind found for key R                         
A:   0.8 (00.8)  4.2%                                                           

and quit.  I tried the sysctl suggestion, no change, whenever a file list
is redirected to stdin, and a filename argument is given to mplayer, eg

while read file; do mplayer "$file" ; done <zz.mtest

now I don't have rtc errors but mplayer is getting strange input it
doesn't grok.

Once again, this works fine without the changed rtc perms or the sysctl
echo:

mplayer `cat zz.mtest`

I've not had a chance to properly test - I still think there is a new
kernel bug/feature but cant find time to properly track it down.

// George


-- 
George Georgalis, systems architect, administrator Linux BSD IXOYE
http://galis.org/george/ cell:646-331-2027 mailto:george@galis.org

Re: a problem with linux 2.6.11 and sa

[Paul Jarc]

"George Georgalis" <george@galis.org> wrote:
> On Wed, Mar 09, 2005 at 06:28:35PM -0500, Paul Jarc wrote:
>> To simplify, what about these two:
>> mplayer foo.mpg
>> mplayer foo.mpg < mediafiles.txt
>
> The particular host does not have X support so mpg is out.

Well, use any one of the files listed in mediafiles.txt.  I expect the
first one would behave the same as your for loop, and the second would
behave the same as your while loop.

> I'm not sure that that test would work as mplayer requires filenames
> as command arguments not stdin (exclusivly, I think);

Note that I said to redirect input from mediafiles.txt, not from any
of the filenames listed in it, but one of the files listed in it
should also be passed ion the command line in both cases.

Your test also had mplayer's stdin connected to mediafiles.txt.  It
was just less explicit - mplayer inherits stdin from surrounding loop.
So I'm suggesting simplifying the test so that stdin is the *only*
difference between the two cases, and that will show whether it's
relevant.  OTOH, if you can't reproduce the problem with the
simplified pair of tests, then some interaction with the shell loops
must be involved.

> this works fine
> mplayer `cat zz.mtest `
>
> Then I tried
> mplayer /dev/stdin <zz.mtest

In the first case, mplayer is processing the files listed in
zz.mtest.  In the second case, it's processing zz.mtest itself.  So
it's not surprising that you get different results here.

> Then I tried
> while read file; do mplayer "$file" ; done <zz.mtest

What's in zz.mtest?  E.g., if it contains a line "-", then that will
tell mplayer to play the file on stdin, which in this case is
zz.mtest.  Choosing one of the listed files and testing with that, as
I suggested above, will eliminate this possibility.


paul

Re: a problem with linux 2.6.11 and sa

[George Georgalis]

On Wed, Mar 16, 2005 at 05:37:59PM -0500, Paul Jarc wrote:
>"George Georgalis" <george@galis.org> wrote:
>> On Wed, Mar 09, 2005 at 06:28:35PM -0500, Paul Jarc wrote:
>>> To simplify, what about these two:
>>> mplayer foo.mpg
>>> mplayer foo.mpg < mediafiles.txt
>>
>> The particular host does not have X support so mpg is out.
>
>Well, use any one of the files listed in mediafiles.txt.  I expect the
>first one would behave the same as your for loop, and the second would
>behave the same as your while loop.

zz.mtest contains the full path to 3 ogg files on 3 lines, no funny
characters, the following is one of them.

$ mplayer /usr/nfs/sandbox/media/audio/_the-party-has-just-begun/Lebanese_Blonde.ogg

plays fine, as expected. however, per your test:

$ mplayer /usr/nfs/sandbox/media/audio/_the-party-has-just-begun/Lebanese_Blonde.ogg <zz.mtest 
MPlayer 1.0pre6-2.95.4 (C) 2000-2004 MPlayer Team
CPU: IDT/Centaur/VIA C3 Nehemiah (Family: 6, Stepping: 5)
Detected cache-line size is 32 bytes
CPUflags:  MMX: 1 MMX2: 1 3DNow: 0 3DNow2: 0 SSE: 1 SSE2: 0
Compiled for x86 CPU with extensions: MMX MMX2 SSE


Playing /usr/nfs/sandbox/media/audio/_the-party-has-just-begun/Lebanese_Blonde.ogg.
Ogg file format detected.
==========================================================================
Opening audio decoder: [libvorbis] Ogg/Vorbis audio decoder
AUDIO: 44100 Hz, 2 ch, 16 bit (0x10), ratio: 14001->176400 (112.0 kbit)
Selected audio codec: [vorbis] afm:libvorbis (OggVorbis Audio Decoder)
==========================================================================
Checking audio filter chain for 44100Hz/2ch/16bit -> 44100Hz/2ch/16bit...
AF_pre: af format: 2 bps, 2 ch, 44100 hz, little endian signed int 
AF_pre: 44100Hz 2ch Signed 16-bit (Little-Endian)
AO: [oss] 44100Hz 2ch Signed 16-bit (Little-Endian) (2 bps)
Building audio filter chain for 44100Hz/2ch/16bit -> 44100Hz/2ch/16bit...
Video: no video
Starting playback...
No bind found for key _                                                         
A:   0.1 (00.1) ??,?%                                                           
No bind found for key L                         
No bind found for key _                         
No bind found for key B                         
No bind found for key l                         
A:   0.8 (00.8)  4.3%                                                           
  =====  PAUSE  =====

Exiting... (End of file)


program crashes quickly, without any keyboard interaction.

>> I'm not sure that that test would work as mplayer requires filenames
>> as command arguments not stdin (exclusivly, I think);
>
>Note that I said to redirect input from mediafiles.txt, not from any
>of the filenames listed in it, but one of the files listed in it
>should also be passed ion the command line in both cases.
>
>Your test also had mplayer's stdin connected to mediafiles.txt.  It
>was just less explicit - mplayer inherits stdin from surrounding loop.
>So I'm suggesting simplifying the test so that stdin is the *only*
>difference between the two cases, and that will show whether it's
>relevant.  OTOH, if you can't reproduce the problem with the
>simplified pair of tests, then some interaction with the shell loops
>must be involved.

per above, the problem is reproduced with your example.

v>> this works fine
>> mplayer `cat zz.mtest `
>
>> Then I tried
>> while read file; do mplayer "$file" ; done <zz.mtest
>
>What's in zz.mtest?  E.g., if it contains a line "-", then that will
>tell mplayer to play the file on stdin, which in this case is
>zz.mtest.  Choosing one of the listed files and testing with that, as
>I suggested above, will eliminate this possibility.

zz.mtest is just 3 ogg files like the one above in my first run. Me
throws up hands, I know it is kernel api change, me thinks Linux is not
posix anymore (per lkml followup). Big concern is not my ability to play
songs, but *complex* scripts to check spam during smtp are broke in
2.6.11 (rc?) and forward.

tmp="${scq}/`safecat "${scq}/tmp" "${scq}" </dev/stdin`" \
        || { echo "Error $?"; exit 71; } # put the pipeline to disk, if possible
        # ${scq}/tmp is a temp for this function ${scq} is temp for this program
score=`spamc -x -c <"$tmp"` # score it with spamd
sce=$?

... Who knows what else won't be working.

// George


-- 
George Georgalis, systems architect, administrator Linux BSD IXOYE
http://galis.org/george/ cell:646-331-2027 mailto:george@galis.org