problems with QMAILQUEUE and reading stdin

problems with QMAILQUEUE and reading stdin

[George Georgalis]

I'm dealing with a frustrating problem where my QMAILQUEUE program
is not working because it cannot read stdin.

2006-05-29_00:58:31.79998 tcpsvd: info: status 3/20
2006-05-29_00:58:31.80038 tcpsvd: info: pid 10614 from 204.152.190.11
2006-05-29_00:58:31.80098 tcpsvd: info: start 10614 sta.galis.org:69.31.23.138 ::204.152.190.11:51459 ./peers.cdb/0
2006-05-29_00:58:32.04704 /var/qmail/bin/prequeue: cannot open /dev/stdin: error 9
2006-05-29_00:58:32.15075 tcpsvd: info: end 10614 exit 0

This is the line from /var/qmail/bin/prequeue that's causing the
error:

tmp="$pq/`/usr/pkg/bin/safecat "$pq/tmp" "$pq" </dev/stdin`" || exit 71 # put message to disk, if possible

my dev file seems fine (and works on command line)

crw-rw-rw-  1 root  wheel  22, 0 Feb 12 14:02 /dev/stdin

and if I set QMAILQUEUE="/var/qmail/bin/qmail-queue" it works fine
too.

I've insured the directories safecat writes to are writable by
the prequeue program and I've even tried making prequeue SUID
qmaild, and even qmailq (with matching perms on its working dir).
(in the past my QMAILQUEUE program worked with mode 775, and qmaild
write perms on the working dir)

Help! what's missing here, why cannot I read stdin from _my_
QMAILQUEUE program yet qmail-queue has no problem running via
QMAILQUEUE=/var/qmail/bin/qmail-queue?

// George

-- 
George Georgalis, systems architect, administrator <IXOYE><
http://galis.org/ cell:646-331-2027 mailto:george@galis.org

Re: problems with QMAILQUEUE and reading stdin

[B S Srinidhi]

Hi,

On Sun, 2006-05-28 at 21:58 -0400, George Georgalis wrote:
> I'm dealing with a frustrating problem where my QMAILQUEUE program
> is not working because it cannot read stdin.
> 
> 2006-05-29_00:58:31.79998 tcpsvd: info: status 3/20
> 2006-05-29_00:58:31.80038 tcpsvd: info: pid 10614 from 204.152.190.11
> 2006-05-29_00:58:31.80098 tcpsvd: info: start 10614 sta.galis.org:69.31.23.138 ::204.152.190.11:51459 ./peers.cdb/0
> 2006-05-29_00:58:32.04704 /var/qmail/bin/prequeue: cannot open /dev/stdin: error 9
> 2006-05-29_00:58:32.15075 tcpsvd: info: end 10614 exit 0
> 

'error 9' means "Bad file number". So, its nothing to with your
permissions or anything else.

> This is the line from /var/qmail/bin/prequeue that's causing the
> error:
> 
> tmp="$pq/`/usr/pkg/bin/safecat "$pq/tmp" "$pq" </dev/stdin`" || exit 71 # put message to disk, if possible
> 

hmm... I don't know what exactly should happen here, but wouldn't it be
possible to do this:

`cat /dev/stdin | /usr/pkg/bin/safecat "$pq/tmp" "$pq"`

??

Hope this helps.

Srinidhi.
-- 
ASCII ribbon campaign ( )         B S Srinidhi
 - against HTML email  X          http://srinidhi.deeproot.co.in
             & vCards / \         DeepRoot Linux

Re: problems with QMAILQUEUE and reading stdin

[Charlie Brady]

On Mon, 29 May 2006, B S Srinidhi wrote:

>> This is the line from /var/qmail/bin/prequeue that's causing the
>> error:
>>
>> tmp="$pq/`/usr/pkg/bin/safecat "$pq/tmp" "$pq" </dev/stdin`" || exit 71 # put message to disk, if possible
>>
>
> hmm... I don't know what exactly should happen here, but wouldn't it be
> possible to do this:
>
> `cat /dev/stdin | /usr/pkg/bin/safecat "$pq/tmp" "$pq"`

Why run cat unnecessarily (and depend on /dev/stdin which only sometimes 
exists)?

Try:

tmp=$pq/$(/usr/pkg/bin/safecat "$pq/tmp" "$pq")

Re: problems with QMAILQUEUE and reading stdin

[B S Srinidhi]

On Mon, 2006-05-29 at 08:11 -0400, Charlie Brady wrote:
> Why run cat unnecessarily (and depend on /dev/stdin which only sometimes 
> exists)?
> 
> Try:
> 
> tmp=$pq/$(/usr/pkg/bin/safecat "$pq/tmp" "$pq")

hmm... Yeah you are right. :) But is the above thing portable? I mean,
would this work with shells other than BASH?

Srinidhi.
-- 
ASCII ribbon campaign ( )         B S Srinidhi
 - against HTML email  X          http://srinidhi.deeproot.co.in
             & vCards / \         DeepRoot Linux

Re: problems with QMAILQUEUE and reading stdin

[George Georgalis]

On Mon, May 29, 2006 at 05:49:02PM +0530, B S Srinidhi wrote:
>On Mon, 2006-05-29 at 08:11 -0400, Charlie Brady wrote:
>> Why run cat unnecessarily (and depend on /dev/stdin which only sometimes 
>> exists)?
>> 
>> Try:
>> 
>> tmp=$pq/$(/usr/pkg/bin/safecat "$pq/tmp" "$pq")

Thank you! I had tried all sorts of ways to read stdin; but I was
always referencing the dev file directly (or cat -). The implicit
stdin worked on first try. :)

>hmm... Yeah you are right. :) But is the above thing portable? I mean,
>would this work with shells other than BASH?

well I'm using it with NetBSD /bin/sh which I understand is the
closest thing to the original /bin/sh available. (the ash shell
is the closest in Linux).

I'd prefer to specify stdin as an argument, I'd like to retain
that style for helping people (including myself) interpret the
code.  I'm going to take it up on some netbsd list, they are
pretty good at fixing things if they aren't working right.

Thanks again!

// George

-- 
George Georgalis, systems architect, administrator <IXOYE><
http://galis.org/ cell:646-331-2027 mailto:george@galis.org

Re: problems with QMAILQUEUE and reading stdin

[Charlie Brady]

On Mon, 29 May 2006, George Georgalis wrote:

> I'd prefer to specify stdin as an argument, I'd like to retain
> that style for helping people (including myself) interpret the
> code.

Why? If someone is reading code and they don't know that programs which 
read stdin by default actually read stdin if you don't do anything 
special, then they can't interpret much code at all, can they?

Add a comment about reading stdin if you really think you need it.

Re: problems with QMAILQUEUE and reading stdin

[George Georgalis]

On Mon, May 29, 2006 at 10:38:19AM -0400, Charlie Brady wrote:
>
>On Mon, 29 May 2006, George Georgalis wrote:
>
>>I'd prefer to specify stdin as an argument, I'd like to retain
>>that style for helping people (including myself) interpret the
>>code.
>
>Why? If someone is reading code and they don't know that programs which 
>read stdin by default actually read stdin if you don't do anything 
>special, then they can't interpret much code at all, can they?

yeah, that's what I was thinking but did not say.

>Add a comment about reading stdin if you really think you need it.

yeah. that's what I did. I'll post the new ipsvd/QMAILQUEUE
program here after some more light mods and testing.

but I'd also like to get to the bottom of what looks like a
/bin/sh or kernel bug.

// George


-- 
George Georgalis, systems architect, administrator <IXOYE><
http://galis.org/ cell:646-331-2027 mailto:george@galis.org

Re: problems with QMAILQUEUE and reading stdin

[Charlie Brady]

On Mon, 29 May 2006, George Georgalis wrote:

> but I'd also like to get to the bottom of what looks like a
> /bin/sh or kernel bug.

/dev/stdin is just a hack to let you read stdin on programs which insist 
on reading from a filename. I don't see its use as appropriate here.

If you really have time to satisfy your curiosity as to what is going on 
here, then strace (or BSD equiv) is your friend.

Re: problems with QMAILQUEUE and reading stdin

[George Georgalis]

On Mon, May 29, 2006 at 11:49:32AM -0400, George Georgalis wrote:
>On Mon, May 29, 2006 at 10:38:19AM -0400, Charlie Brady wrote:
>>Add a comment about reading stdin if you really think you need it.
>
>yeah. that's what I did. I'll post the new ipsvd/QMAILQUEUE
>program here after some more light mods and testing.

Well here it is, http://galis.org/script/prequeue
not yet added to a version control management yet
and there is not a lot of mail on this Memorial Day,
so testing is incomplete, but this is a rewrite
of the program I've been using for active smtp
filtering, for the past few years, so it may be
okay.  Comments welcome and encouraged.

I tried or looked at qpsmtpd, simscan, qmail-qfilter
and qmail-spp; but for one reason or another stuck
with my own script.

http://smtpd.develooper.com/
http://www.inter7.com/simscan/
http://untroubled.org/qmail-qfilter/
http://qmail-spp.sourceforge.net

I'd really like it if qmail-spp made it easy to make
"entire email" filter plugins, that would probably
be the best way.

Sorry, no doc on entire deployment (that would be a
big project), but maybe if you are on this list you
can figure it out. Besides the djbdns lookups of RBL
lists, I'm using ipsvd (a daemontools replacement)
under runsvdir and maintain a peers.cdb to run qmail
with clamd and spamassassin; and there is an openbsd
spamd front end.

http://smarden.org/ipsvd/
http://www.openbsd.org/papers/bsdcan05-spamd/

I have a patch to make spamd more politically correct,
but it's not even applied to my own production yet,
eventually it will show up somewhere in here
http://galis.org/mkinst/patch/

That's about it. Enjoy. Peace.

// George

#!/bin/sh
#
# $Id$
# $GeorgalisG: prequeue$
#
# This script functions as a QMAILQUEUE program for "in SMTP" (active)
# filtering of email. It accepts stdin from qmail-smtpd, and expects
# associated environmentals. After it tests with clamav, it then tests
# the email with RBLs and spamassassin. Addition or removal of tests is
# a simple mod. Rejected messages are saved in a maildir (easy enough
# to disable).  There may be better way (such as umask and supplementary
# groups), but for now I run the various scanners as user qmaild so all
# programs have the read/write access they need.
#
# As root, run this once, to initialize
# pq="prequeue"
# install -o qmaild -g qmail -m 2770 -d ~qmaild/$pq ~qmaild/$pq/new ~qmaild/$pq/tmp  ~qmaild/$pq/cur
#
# LICENSE: <george@galis.org> wrote this file. As long as you retain this
# notice, you can do anything with it or buy me a beer -- George Georgalis
#
# exit 31 = permanently refuse
# exit 71 = temporarily refuse
#
# TODO deliver failures, with modified header to, and only to, valid users.

set -e # exit on internal error

ptr () { # reverse a dotted quad or subnet
 rev="$(echo "$1" | cut -d\. -f1).$2" ; ip="$(echo "$1" | cut -d\. -f2-)"
 [ "$ip" = "$1" ] && echo "${rev}" || ptr $ip $rev ;}

failforward () { # update ipsvd-instruct(5), cdb regenerated separately
 umask 002 ; echo "$peerm $opinion" >$peerd/$TCPREMOTEIP
 echo "$(basename $0): failforward: $opinion" 1>&2
 rm "$tmp" ; exit 31 ;} # permanently refuse

fail () { # mark the message with failure report and refuse
 formail -f -b -A "$opinion" <"$tmp" | maildir "$pq" >/dev/null # save in maildir for manual delete
 rm "$tmp" ; exit 31 ;} # permanently refuse

drop () {
 echo "$(basename $0): deny: $opinion" 1>&2
 rm "$tmp" ; exit 31 ;} # permanently refuse

warn () { # error, mark the message, save and refuse
 formail -f -b -A "$opinion" <"$tmp" >"${tmp}-$$"
 echo "$(basename $0): warn: $opinion" 1>&2
 mv "${tmp}-$$" ./ && rm "$tmp" # save for review
 # should monitor $PWD, or notify when warn is run...
 exit 71 ;} # temporarily refuse

pass () { # mark it and pass to the regular queue
 formail -f -b -A "$opinion" <"$tmp" | ./bin/qmail-queue ; testexit=$?
 rm "$tmp" ; exit $testexit ;} # return whatever qmail-queue exits as
 # somehow qmail-queue gets descriptor 1 from qmail-smtpd....

cd /var/qmail
host=$(cat control/me)
ptrip=$(ptr ${TCPREMOTEIP})
now="$(date "+%x %r %Z")"
pq="prequeue" # a maildir with qmaild write perms
peerd="supervise/qmail-smtpd/peers" # prepare to update ipsvd-cdb(8) config
peerm='#!/bin/sh\necho  "220 smtp port"\necho  "250 smtp host"\necho  "550'
# $pq/tmp is a tmp for this operation, $pq is tmp for this program
# $pq is also a maildir for messages rejected by this program
tmp="$pq/$(/usr/pkg/bin/safecat $pq/tmp $pq)" || exit 71 # </dev/stdin # put message to disk, if possible

# Check if $ACCEPT is set to tag message and bypass tests
if [ -n "$ACCEPT" ]; then
 opinion="X-ipsvd: $ACCEPT ($now)"
 pass
fi

if [ -n "$DENY" ]; then
 opinion="$DENY ${TCPREMOTEIP}"
 drop
fi
 
score="X-clamav: $(clamdscan --config-file=/usr/local/etc/clamd.conf --no-summary ${tmp})" ; testexit=$?
case $testexit in
 0) true ;; # no virus
 1) opinion="$(echo $score | sed -e "s;${PWD}/${tmp}: ;;") ($now)" ; fail ;; # virus found
 *) opinion="$(echo $score | sed -e "s;${PWD}/${tmp}: ;;") ($now)" ; warn ;; # clamav error
esac

opinion="X-sbl-xbl:$(dnstxt ${ptrip}sbl-xbl.spamhaus.org \
 | sed 's/http/ http/g' | grep http) ($now)" && failforward

# too many major ISP relays added
#opinion="X-sorbs-spam: $(dnstxt ${ptrip}spam.dnsbl.sorbs.net \
# | grep http)" && fail

# blocked yahoo groups... will restore after ACCEPT peers is fortified
# opinion="X-spamcop: $(dnstxt ${ptrip}bl.spamcop.net \
# | grep http) ($now)" && fail

# score upto 300KB with spamd, 250KB default, but no workie -s 307200 
score=$(spamc -x -c <"$tmp") ; testexit=$?
opinion="X-spamc: ${score} ${TCPREMOTEIP}; ${host} ($now)"
case $testexit in
 0) pass ;; # ham
 1) fail ;; # spam 
 *) warn ;; # spamc error 
esac

exit 81 # Internal bug


-- 
George Georgalis, systems architect, administrator <IXOYE><
http://galis.org/ cell:646-331-2027 mailto:george@galis.org

Re: problems with QMAILQUEUE and reading stdin

[Charlie Brady]

On Mon, 29 May 2006, B S Srinidhi wrote:

> On Mon, 2006-05-29 at 08:11 -0400, Charlie Brady wrote:
>> Why run cat unnecessarily (and depend on /dev/stdin which only sometimes
>> exists)?
>>
>> Try:
>>
>> tmp=$pq/$(/usr/pkg/bin/safecat "$pq/tmp" "$pq")
>
> hmm... Yeah you are right. :) But is the above thing portable? I mean,
> would this work with shells other than BASH?

It's portable, but not portable to every legacy shell. I understand that 
$() is a POSIX standard. This doc (entitled 'The POSIX standard'):

http://www.cems.uwe.ac.uk/~srking/asa/posix/Notes.pdf

says:

  The backtick construction is deprecated in favour of $(). However the.
  Bourne shell only supports backticks. ...