Mail to news (and back again)

Mail to news (and back again)

[Lars Magne Ingebrigtsen]

My, plonking lots of mail into innd (which seems to be the only free
news server out there) is Lots Of Pain.  It's awfully finicky in what
it accepts -- lines longer than 1KB, the Date header has to be valid,
and the Message-ID, and the From, and there has to be a Subject in
there.

I mean, I think that's great and all, but, well.

Anyway, I've hacked away at innd to skip all those test, and now it's
willing to accept anything I throw at it, no matter how badly formed.
As small step for, uhm, ah.

I was thinking of funneling the gnus-bug mails to one group, the ding
mails to another, and set up a non-expiring gnu.emacs.gnus there as
well.  And create to groups for ietf drafts and rfcs, to make it
easier to keep up with what's happening.

Is there some sort of ready-made procmail (or whatever) setup for
having all the mail that's sent to, say "ding@quimby.gnus.org" spooled 
in inn?  I guess one should run, er, rnews on the mail, after
massaging it slightly.  (Like adding the Newsgroups: gnus.ding" header 
and possibly renaming any offending headers (like NNTP-Posting-Host
and the like.))

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mail to news (and back again)

[Lars Balker Rasmussen]

On Sun, Sep 26, 1999 at 07:27:42PM +0200, Lars Magne Ingebrigtsen wrote:
> Is there some sort of ready-made procmail (or whatever) setup for
> having all the mail that's sent to, say "ding@quimby.gnus.org" spooled 
> in inn?  I guess one should run, er, rnews on the mail, after
> massaging it slightly.  (Like adding the Newsgroups: gnus.ding" header 
> and possibly renaming any offending headers (like NNTP-Posting-Host
> and the like.))

Not procmail, but I have some stuff at work I use for bidirectional
mail-gateways for comp.lang.beta and comp.compilers.lcc. 

I'll send a link to it to the list tomorrow.
-- 
Lars Balker Rasmussen, Software Engineer, Mjolner Informatics ApS
lbr@mjolner.dk

Re: Mail to news (and back again)

[Harry Putnam]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Is there some sort of ready-made procmail (or whatever) setup for
> having all the mail that's sent to, say "ding@quimby.gnus.org" spooled 
> in inn?  I guess one should run, er, rnews on the mail, after
> massaging it slightly.  (Like adding the Newsgroups: gnus.ding" header 
> and possibly renaming any offending headers (like NNTP-Posting-Host
> and the like.))

Procmail can handle all the above including adding and subtracting
headers.  Not sure what "spooled in inn" means but if you just mean a
spool file in a certain location then it could be as simple as 

 :0
* To:.* ding@quimby
</path/inn-spool-file-name>

You probably know this stuff already so hope I'm not missing what you
are after.

The .procmailrc file needs to specific $MAILDIR since any files
generated by procmail are relative to it pathwise.  With out $MAILDIR
the root is $HOME Or you must spell out the full path. Some other things
need to be specified too, or else problems may develope.  Something
like this:

********** Begin sample *****
# -*-Shell-script-*-
PATH=/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin
SHELL=/bin/sh   
MAILDIR=$HOME/spool <= all procmail files will go to $HOME/spool <a diretory   
                       unless otherwise directed with full path  
VERBOSE=YES  # <== set to NO once things are working 
LOGFILE=$HOME/.procmail.log  <=will contain lots of useful info to
                               help get setup

 :0
* To:.* ding@quimby
</path/inn-spool-file-name>

******** end ************

You can also have a certain address or other Header set off any
command by having procmail send the message to that command in a pipe
and then telling it to ignore the message and just do the command.

You can do all sorts of legal and semi legal stuff with the formail
part of procmail.  Adding and subtracting headers etc.  I'm not that
good at it but here is a simple example that I use to add a header to
mail being .forwarded from another address, so I can ident it easily.

Below  collects any mail sent to my jtan.com address, applies a header:
X-Delivered-Via: reader@jtan.com 
Then sends the messages on to my newsguy.com address

    :0
     | formail -I "X-Delivered-Via: reader@jtan.com" \
         | $SENDMAIL $SENDMAILFLAGS reader@newsguy.com

Below files a message in a spool and sends a copy  on to start a command or
run a script.

 :0
 * ^Subject:.* <whatever>  <=collects all mail with <whatever>
{
     :0 c:          <=files to this address & sends a copy on
  /lars/inn/spool/filename

     :0 i          <=  pipes to  a script or command,ignoring the message
   |/lars/scipts-todo-lots-of-trick-stuff
}

Re: Mail to news (and back again)

[Michael Cook]

[-- Attachment #1: Type: text/plain, Size: 439 bytes --]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Is there some sort of ready-made procmail (or whatever) setup for
> having all the mail that's sent to, say "ding@quimby.gnus.org" spooled 
> in inn?  I guess one should run, er, rnews on the mail, after
> massaging it slightly.  (Like adding the Newsgroups: gnus.ding" header 
> and possibly renaming any offending headers (like NNTP-Posting-Host
> and the like.))

Here's what I use.


[-- Attachment #2: nntp-post --]
[-- Type: text/plain, Size: 3234 bytes --]

#!/usr/bin/perl -w
#
# Post a news article via an NNTP server.
#
# Example:
#  nntp-post \
#    --insert="Newsgroups: local.pokemon.d" \
#    --insert="Approved: cook" \
#    --delete=received \
#    files...
#
# Michael Cook <cook@sightpath.com>
# Thu Mar  6 22:39:14 EST 1997
#

use strict;
use Net::NNTP;
use Getopt::Long;

my @DELETE;
my @INSERT;
my $DUPLICATE;
my $NEWSGROUP;
my $HOST;
my $DEBUG;

Getopt::Long::config("bundling");
GetOptions
("--delete=s" => \@DELETE, "-d" => \@DELETE,
 #
 # Headers to remove.  Example: -d xref -d nntp-posting-host String is
 # caseless, not a pattern or regex, with no trailing colon.
 #

 "--insert=s" => \@INSERT, "-i" => \@INSERT,
 #
 # Headers to insert.  Example: -i "Approved: moi" String is inserted as-is,
 # with a newline appended.  (Deletes (--delete) are done before inserts.)
 #

 "--duplicate" => \$DUPLICATE, "-u" => \$DUPLICATE,
 #
 # If the server rejects the article claiming that it is a duplicate, remove
 # the Message-Id header and try again.
 #

 "--newsgroup=s" => \$NEWSGROUP, "-g=s" => \$NEWSGROUP,
 #
 # Same as: --delete=xref --delete=received --insert="Newsgroups: $NEWSGROUP"
 #

 "--host=s" => \$HOST, "-h=s" => \$HOST,
 #
 # Which host to connect to.
 # Default is defined in Net::NNTP.
 #

 "--debug" => \$DEBUG,
)
or die "usage: $0 [files...]\n";

if (defined $NEWSGROUP)
{
    push @DELETE, "xref";
    push @DELETE, "received";
    push @INSERT, "Newsgroups: $NEWSGROUP";
}

my %DELETE = map { lc($_) => 1 } @DELETE;
for (@INSERT) { $_ .= "\n" }

my @nntp = ($HOST);
push @nntp, Debug => 1
    if $DEBUG;
my $nntp = Net::NNTP->new(@nntp)
    or die "cannot connect to NNTP server";

@ARGV = ("-")
    unless @ARGV;

my $exit = 0;
for my $arg (@ARGV)
{
    unless (open FH, "< $arg")
    {
	warn "cannot open `$arg': $!";
	$exit = 1;
	next;
    }
    my @article = <FH>;
    close FH;

    print("original article:\n +----\n",
          (map { " | $_" } @article),
          " +----\n")
	if $DEBUG;

    for (@article)
    {
	s/\r$//;
    }

    my $lno = 0;
    if (@article && $article[0] =~ /^From /)
    {
	warn "(warning) eschewing unix `From ' line, first line of `$arg'";
	shift @article;
	$lno++;
    }

    my @header;
    while (@article && $article[0] !~ /^$/)
    {
	$_ = shift @article;
	$lno++;
	while (@article && $article[0] =~ /^[ \t]/)
	{
	    $_ .= shift @article;
	    $lno++;
	}
	unless (/^([^\s:]+):/)
	{
	    warn "(warning) file \"$arg\", line $lno: malformed header:\n--\n$_--";
	    push @header, $_;
	    next;
	}
	push @header, $_
	    unless $DELETE{lc($1)};
    }
    unshift @article, @header, @INSERT;

    print("posting article:\n +----\n",
          (map { " | $_" } @article),
          " +----\n")
	if $DEBUG;

    #next if $DEBUG;
    next if $nntp->post(\@article);
    my $msg = (reverse $nntp->message())[0];
    chomp $msg;
    if ($DUPLICATE && $msg =~ /^435 Duplicate$/)
    {
	warn "server says this message is a duplicate; retrying...";
	for (@article)
	{
	    $_ = "" if /^message-id:/i;
	    last if /^\n/;
	}
	next if $nntp->post(\@article);
	$msg = (reverse $nntp->message())[0];
	chomp $msg;
    }
    warn "failed to post `$arg': (" . $nntp->code() . ") $msg";
    $exit = 1;
}

$nntp->quit;
exit $exit;

[-- Attachment #3: Type: text/plain, Size: 1140 bytes --]


If you use sendmail on quimby, you might add an alias like this:

ding: "|/local/bin/nntp-post local.ding --delete=NNTP-Posting-Host"

Actually, to make the news-to-mail and mail-to-news gatewaying
complete, I'd setup the ding newsgroup as being moderated.  (Use an
`m' instead of a `y' as the 3rd argument to ctlinnd.)  Have
ding-moderator@gnus.org be the moderator (see the moderators file,
often /etc/news/moderators; see also moderators(5)).  Then, I'd
create aliases like this:

ding: ding-list ding-news

ding-list: :include:/whatever/lists/ding

ding-news: "|/local/bin/nntp-post local.ding
 --insert=Approved:auto
 --delete=NNTP-Posting-Host"

ding-moderator: ding

Or some such.

So, mail sent to `ding@gnus.org' goes to everyone on the mailing
list, and also gets fed to the nntp server after having an Approved
header added.

When someone posts an article to the newsgroup, it won't
(ordinarily) have the Approved header, and so inn will mail it to
ding-moderator.  Mail to ding-moderator gets forwarded to everyone
on the ding mailing list, and also gets fed back to the nntp server
after having the Approved header added.

M.

Re: Mail to news (and back again)

[Lars Magne Ingebrigtsen]

Lars Balker Rasmussen <lbr@mjolner.dk> writes:

> Not procmail, but I have some stuff at work I use for bidirectional
> mail-gateways for comp.lang.beta and comp.compilers.lcc. 
> 
> I'll send a link to it to the list tomorrow.

Great.

Does someone happen to have a complete (or completeish, or
kinda-large-ish) archive of the ding list?  I could plonk it down into
the new Quimby nntp server.

Hm.  And I wonder what I should do about gnu.emacs.gnus.  Should I
peer with somebody to get a feed, and be able to propagate articles
posted to that group?  Probably.  Anybody here who runs an nntp server 
that wants to peer?  (Just for that one group.)

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mail to news (and back again)

[Eric Marsden]

>>>>> "mc" == Michael Cook <cook@sightpath.com> writes:

  mc> Here's what I use.

  mc> #!/usr/bin/perl -w

Shame on you! Now you need to repent with a `C-u 1000 hail-elisp'.

-- 
Eric Marsden
It's elephants all the way down

Re: Mail to news (and back again)

[Jason R Mastaler]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Does someone happen to have a complete (or completeish, or
> kinda-large-ish) archive of the ding list?  I could plonk it down
> into the new Quimby nntp server.

Yes, archives of the ding list (in mbox format) from November 1995 to
the present are available at:

        <http://www.gnus.org/list-archives/ding/#browse>

The "Raw Archives" are probably what you want (these are in standard
unix mbox format).

Re: Mail to news (and back again)

[Lars Magne Ingebrigtsen]

Jason R Mastaler <jason@4b.org> writes:

> Yes, archives of the ding list (in mbox format) from November 1995 to
> the present are available at:
> 
>         <http://www.gnus.org/list-archives/ding/#browse>

Thanks; I'll download and funnel, funnel, funnel them into the news
server.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mail to news (and back again)

[Russ Allbery]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Is there some sort of ready-made procmail (or whatever) setup for having
> all the mail that's sent to, say "ding@quimby.gnus.org" spooled in inn?
> I guess one should run, er, rnews on the mail, after massaging it
> slightly.  (Like adding the Newsgroups: gnus.ding" header and possibly
> renaming any offending headers (like NNTP-Posting-Host and the like.))

My News::Gateway package (on CPAN) has a script in it called listgate that
does this that could probably be used with some tweaking.  News::Gateway
needs a bunch of work that I haven't had time to work on recently, though.

-- 
Russ Allbery (rra@stanford.edu)         <URL:http://www.eyrie.org/~eagle/>

Re: Mail to news (and back again)

[Lars Magne Ingebrigtsen]

Michael Cook <cook@sightpath.com> writes:

> Here's what I use.

Yup; works like a charm.

I'm now stuffing the nntp server with old messages, and it should be
possible to use later today.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mail to news (and back again)

[Lars Balker Rasmussen]

Russ Allbery <rra@stanford.edu> writes:
> My News::Gateway package (on CPAN) has a script in it called listgate that
> does this that could probably be used with some tweaking.  News::Gateway
> needs a bunch of work that I haven't had time to work on recently, though.

Yes, my gateways are based on News::Gateway.  Good work, Russ.

Locally, the lcc@mjolner.dk mailing-list has a list-member which is an
alias that expands to "|postnews comp.compilers.lcc".

On the newsserver (inn), the newsfeeds file has the following entries:
----------------------------------------------------------------------
# common funnel
# mail article to the address indicated by the site name
newstomail!\
        :!*\
        :W*,Tp:/news/local/bin/newstomail.perl *

lcc@mjolner.dk\
        :!*,comp.compilers.lcc/!local\
        :Tm:newstomail!
----------------------------------------------------------------------

The scripts are available at:
    http://www.mjolner.dk/~lbr/postnews
    http://www.mjolner.dk/~lbr/newstomail.perl
-- 
Lars Balker Rasmussen, Software Engineer, Mjolner Informatics ApS
lbr@mjolner.dk

Re: Mail to news (and back again)

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Michael Cook <cook@sightpath.com> writes:
>> Here's what I use.

> Yup; works like a charm.

> I'm now stuffing the nntp server with old messages, and it should be
> possible to use later today.

Hm... how about putting up UoW imapd on quimby and export the
newsspool as read-only IMAP folders as well?

(Nope, I don't know how to do it, I just know it is doable)

Re: Mail to news (and back again)

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@metis.no> writes:

> Hm... how about putting up UoW imapd on quimby and export the
> newsspool as read-only IMAP folders as well?

Well, I guess, but is there much point in doing that?  I'm don't know
what extra functionality that would give users...  (But then again, I
know nothing about imap.)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mail to news (and back again)

[Steinar Bang]

>>>>> You wrote:

> Steinar Bang <sb@metis.no> writes:
>> Hm... how about putting up UoW imapd on quimby and export the
>> newsspool as read-only IMAP folders as well?

> Well, I guess, but is there much point in doing that?  I'm don't
> know what extra functionality that would give users...

Searching could be one, but I don't know if the UoW imapd support
searching in folders.

The cyrus imapd supports searching, but I don't know if it can be set
to point to an existing news spool in the same manner as UoW imapd.

Re: Mail to news (and back again)

[Simon Josefsson]

Steinar Bang <sb@metis.no> writes:

> >> Hm... how about putting up UoW imapd on quimby and export the
> >> newsspool as read-only IMAP folders as well?
> 
> > Well, I guess, but is there much point in doing that?  I'm don't
> > know what extra functionality that would give users...
> 
> Searching could be one, but I don't know if the UoW imapd support
> searching in folders.

It does. Searching is in the core IMAP protocol.

> The cyrus imapd supports searching, but I don't know if it can be set
> to point to an existing news spool in the same manner as UoW imapd.

It can, with the "newsspool" command.

Re: Mail to news (and back again)

[Steinar Bang]

>>>>> Simon Josefsson <jas@pdc.kth.se>:

> Steinar Bang <sb@metis.no> writes:
>> >> Hm... how about putting up UoW imapd on quimby and export the
>> >> newsspool as read-only IMAP folders as well?
>> 
>> > Well, I guess, but is there much point in doing that?  I'm don't
>> > know what extra functionality that would give users...
>> 
>> Searching could be one, but I don't know if the UoW imapd support
>> searching in folders.

> It does. Searching is in the core IMAP protocol.

Can the UoW imapd search in a newsspool it is pointed to?

Re: Mail to news (and back again)

[Simon Josefsson]

Steinar Bang <sb@metis.no> writes:

> >> > Well, I guess, but is there much point in doing that?  I'm don't
> >> > know what extra functionality that would give users...
> >> 
> >> Searching could be one, but I don't know if the UoW imapd support
> >> searching in folders.
> 
> > It does. Searching is in the core IMAP protocol.
> 
> Can the UoW imapd search in a newsspool it is pointed to?

I believe it has to if it's supposed to be a IMAP server. AFAIK UoW
imapd doesn't maintain caches or anything, so it would be quite
slow. Cyrus IMAPD maintain caches for normal groups, I'm not sure
about newsspools.