Mailing lists and news archiving

Mailing lists and news archiving

[Lars Magne Ingebrigtsen]

I'm going through old mail, and mailing lists I'm subscribed to, but
haven't read in a while.  And I'm struck by how much of our collective
knowledge is on these mailing lists, and how ephemeral they are.  Some
of them are archived here and there, but the archives go away or move
or don't work.

News, on the other hand, don't really go away if you don't run
expiry.  If more people are interested mirroring, you just start
feeding.  Or mirror using other means.

So I'm wondering whether I should start setting up mail-to-news
gateways for all the mailing lists I'm subscribed to.  And, of course,
any other mailing lists that people would want me to store.

Does this seem like an OK idea?

The mailing lists I would gateway would be, at least:

XEmacs Beta <xemacs-beta@xemacs.org>
Emacs Custom <emacs-custom@sunsite.dk>
XEmacs Mule <xemacs-mule@xemacs.org>
Emacs Development <emacs-devel@gnu.org>
Linux Kernel <linux-kernel@vger.kernel.org>
Linux USB <linux-usb-devel@lists.sourceforge.net>

Is this a good idea?  A bad idea?  And what would the hierarchy be
called? 

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mailing lists and news archiving

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> So I'm wondering whether I should start setting up mail-to-news
> gateways for all the mailing lists I'm subscribed to. 

I've just browsed my ISP's news server, and there are oodles and
oodles of gatewayed mailing lists there.  Makes me wonder why anybody
subscribes to mailing lists.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mailing lists and news archiving

[news]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> ephemeral they are.  Some of them are archived here and there,
> but the archives go away or move or don't work.

Wow, this is a great topic.

http://www.mail-archive.com does a good job of this, and we can
even read these as gnus groups with nnwarchive!  For now,
mail-archive.com is a gift from Jeff Briedenbach and VALinux.  In
the future, who knows?  We've already lost deja-news.com, which
was archiving usenet in a similar way.  Some people lost faith in
VALinux's approach to http://sourceforge.net and are cloning it
at http://savannah.gnu.org, which is another indication.

> News, on the other hand, don't really go away if you don't run
> expiry.  If more people are interested mirroring, you just start
> feeding.  Or mirror using other means.
>
> So I'm wondering whether I should start setting up mail-to-news
> gateways for all the mailing lists I'm subscribed to.  And, of course,
> any other mailing lists that people would want me to store.

I do this now for all of the high volume lists I subscribe to,
because I *want* expiry after a while, and I want to reduce
multiple subscriptions.  I only distribute them locally, for now.

> Does this seem like an OK idea?

As long as you establish them as moderated newsgroups, it should
be OK with the list owners. I don't know about the privacy
concerns, though.  Moderation and privacy are sticking points and
differences between mail and news.

> The mailing lists I would gateway would be, at least:
<-- lists snipped -->
> Is this a good idea?  A bad idea?  And what would the hierarchy be
> called? 

I *already* think this is a great idea for distribution.  Do you
have the disk space for archiving these, though?  In addition,
are newsreaders really good at perusing archived groups?  Even
with gnus, I know only how to select the N oldest articles or M
most recent articles, but I can't get range N to M.  (I'm just
kidding. I'm certain gnus can do this, but what about other
newsreaders :-) )  Downloading the *whole* archive isn't a good use
of resources.  Might it be better to clone mail-archive.com's
approach for archiving?

Your approach, the approach of especially interested people
archiving lists/groups they love, rather than the "archive
everything" approach of deja-news can succeed in the long run.

I'm happy to peer for feeds, but I need to have an estimate of
the rate of archive growth before I commit to /no expiry/ on those
groups.

Maybe Russ or somebody with deja-news.com experience can shed
light on these issues. 

Chris

Re: Mailing lists and news archiving

[news]

(Chris Beggy ) news@kippona.com writes:

> Might it be better to clone mail-archive.com's
> approach for archiving?

Well, Jeff has GPL'ed this:

   http://www.mail-archive.com/contrib/source/README.txt

Chris

Re: Mailing lists and news archiving

[Lars Magne Ingebrigtsen]

(Chris Beggy ) news@kippona.com writes:

> http://www.mail-archive.com does a good job of this, and we can
> even read these as gnus groups with nnwarchive!

Yeah, mail-archive.com is pretty good, but I still prefer reading from
an nntp server.  Even though nnwarchive makes it possible to read the
archives with Gnus, it's so much smoother when reading from an nntp
server... 

> For now, mail-archive.com is a gift from Jeff Briedenbach and
> VALinux.  In the future, who knows?  We've already lost
> deja-news.com, which was archiving usenet in a similar way.  Some
> people lost faith in VALinux's approach to http://sourceforge.net
> and are cloning it at http://savannah.gnu.org, which is another
> indication.

Yeah.  Difficult to say what's going to happen with all that.  People
seem quite jittery about sourceforge these days...

> I *already* think this is a great idea for distribution.  Do you
> have the disk space for archiving these, though?

Sure.  Disks are cheap.  :-)

If I get serious about doing a real big news server, then I'll
probably extend the machine with a nice RAID-ey disk setup, but for
now, it just has one biiig IDE disk.

> In addition, are newsreaders really good at perusing archived
> groups?  Even with gnus, I know only how to select the N oldest
> articles or M most recent articles, but I can't get range N to M.
> (I'm just kidding. I'm certain gnus can do this, but what about
> other newsreaders :-) ) Downloading the *whole* archive isn't a good
> use of resources.  Might it be better to clone mail-archive.com's
> approach for archiving?

I was actually pondering these very things.  How to do a convenient
search from Gnus from such a repository?

Ok, say you go enter the gnus.xemacs.beta group.  (I should come up
with a better top level hierarchy, though.)  You press the "search"
button, and type in a text.  Gnus queries Quimby, and gets a list of
Message-IDs back.  Gnus then requests these headers from the news
server, and displays them in the buffer.

The query can be via HTTP to some search engine or other running
separately on Quimby.  For instance a Glimpse thing, or whatever is
the done thing these days.

I think such a thing could be done without too much work...

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mailing lists and news archiving

[news]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I was actually pondering these very things.  How to do a convenient
> search from Gnus from such a repository?
>
> Ok, say you go enter the gnus.xemacs.beta group.  (I should come up
> with a better top level hierarchy, though.)  You press the "search"
> button, and type in a text.  Gnus queries Quimby, and gets a list of
> Message-IDs back.  Gnus then requests these headers from the news
> server, and displays them in the buffer.
>
> The query can be via HTTP to some search engine or other running
> separately on Quimby.  For instance a Glimpse thing, or whatever is
> the done thing these days.
>
> I think such a thing could be done without too much work...

Sounds like extending nnir.el with wais (Z.90) or glimpse (http)
for searching a published index on the server with the archived
groups.  nntp isn't good for this, I agree.

What about browsing though?

Chris

Re: Mailing lists and news archiving

[Steinar Bang]

>>>>> (Chris Beggy ) news@kippona.com:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

[snip!]
>> The query can be via HTTP to some search engine or other running
>> separately on Quimby.  For instance a Glimpse thing, or whatever is
>> the done thing these days.

>> I think such a thing could be done without too much work...

> Sounds like extending nnir.el with wais (Z.90) or glimpse (http)

The latest free glimpse versions have bugs in them that makes them
crash during indexing of large spools.  This happened for me when I
attempted to index my nnml groups for use with nnir.  Dunno if the
debian glimpse package has been patch to fix this?

The newer glimpse versions are commercial, I think.

The htdig package may be an alternative, but I have never used it
myself. 
	apt-get install htdig

> for searching a published index on the server with the archived
> groups.  nntp isn't good for this, I agree.

How about allowing access to the news spool from a cyrus IMAP server?
Doesn't that open for searching via nnir?  Or is that perhaps just a
flat text search, not an indexed search like glimpse or htdig?

Re: Mailing lists and news archiving

[Lars Magne Ingebrigtsen]

(Chris Beggy ) news@kippona.com writes:

> What about browsing though?

There must be some browse-nntp-like-spool package out there.  If not,
it'll have to be written.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mailing lists and news archiving

[Kai Großjohann]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I was actually pondering these very things.  How to do a convenient
> search from Gnus from such a repository?
>
> Ok, say you go enter the gnus.xemacs.beta group.  (I should come up
> with a better top level hierarchy, though.)  You press the "search"
> button, and type in a text.  Gnus queries Quimby, and gets a list of
> Message-IDs back.  Gnus then requests these headers from the news
> server, and displays them in the buffer.

*cough*

nnir.el has an attempt on this.  You only need to add the function
that does the querying of the search engine:

> The query can be via HTTP to some search engine or other running
> separately on Quimby.  For instance a Glimpse thing, or whatever is
> the done thing these days.

If I do say so myself.  *blush*

> I think such a thing could be done without too much work...

I wish I had more time to devote to nnir.el.  I keep planning to add
useful features (such as interfacing to my pet search engine), but
don't have the time.  Ick.

ftp://ls6-ftp.cs.uni-dortmund.de/pub/src/emacs/
has nnir.el, I believe.

kai
-- 
Simplification good!  Oversimplification bad!  (Larry Wall)

Re: Mailing lists and news archiving

[Lars Magne Ingebrigtsen]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> nnir.el has an attempt on this.  You only need to add the function
> that does the querying of the search engine:
>
>> The query can be via HTTP to some search engine or other running
>> separately on Quimby.  For instance a Glimpse thing, or whatever is
>> the done thing these days.
>
> If I do say so myself.  *blush*

:-)

Is nnir ready for inclusion in Gnus, you think?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mailing lists and news archiving

[Kai Großjohann]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Is nnir ready for inclusion in Gnus, you think?

It seems that the functionality that's offered by nnir.el is stable
and works okay.  Maybe it's not enough functionality.  Things that
come to mind:

* nnir only supports one backend to search in, because it's got a
  global variable for the backend.  I wanted to add server parameters
  so that the nnir-specific things can be specified for each server,
  and nnir would search all servers which are nnir-enabled.  But I
  haven't got around to it :-(

* nnir is kinda restricted to one-file-per-message backends.  This is
  because nnir calls a search engine, and tries to glean group name
  and article number from that output.  And it is easy to tell the
  search engine to print the file name (wherefrom said data can
  easily be gleaned).  If you can convince the search engine to print
  the article number and group name for other backends, then they
  will work with nnir, too.

  For example, nnimap works with nnir, because IMAP has a built-in
  search command.

* nnir is kinda fragile because it parses the output of the
  command-line search tools.  So it's easy to confuse nnir by adding
  a slash at the end of a directory name, or by removing it.

So: after a day of hacking by Lars the Quick Typist, I'm sure it will
be in good shape :-)

kai
-- 
Simplification good!  Oversimplification bad!  (Larry Wall)

Re: Mailing lists and news archiving

[Lars Magne Ingebrigtsen]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> So: after a day of hacking by Lars the Quick Typist, I'm sure it will
> be in good shape :-)

Uhm.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mailing lists and news archiving

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Uhm.  :-)

Hm...  I'm not getting any smileys in XEmacs.  Anybody else see this? 

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mailing lists and news archiving

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>> Uhm.  :-)
>
> Hm...  I'm not getting any smileys in XEmacs.  Anybody else see this? 

I've now made the smiley-ems version work under XEmacs as well.  I
plan on removing smiley.el and replacing it with smiley-ems.el in a
couple of days...

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mailing lists and news archiving

[Björn Torkelsson]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>>> Uhm.  :-)
>>
>> Hm...  I'm not getting any smileys in XEmacs.  Anybody else see this? 
>
> I've now made the smiley-ems version work under XEmacs as well.  I
> plan on removing smiley.el and replacing it with smiley-ems.el in a
> couple of days...

Please remove smiley-ems.el from dgnushack too when compiling for
XEmacs.

/torkel

Index: lisp/dgnushack.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/dgnushack.el,v
retrieving revision 6.22
diff -u -r6.22 dgnushack.el
--- lisp/dgnushack.el   2001/12/29 20:44:10     6.22
+++ lisp/dgnushack.el   2002/01/20 21:36:04
@@ -249,7 +249,7 @@
        (setq files (delete "nnrss.el" files))))
     (dolist (file
             (if (featurep 'xemacs)
-                '("md5.el" "smiley-ems.el")
+                '("md5.el")
               '("gnus-xmas.el" "messagexmas.el" "nnheaderxm.el" "smiley.el")))
       (setq files (delete file files)))
 

Re: Mailing lists and news archiving

[Lars Magne Ingebrigtsen]

torkel@acc.umu.se (Björn Torkelsson) writes:

> Please remove smiley-ems.el from dgnushack too when compiling for
> XEmacs.

Thanks for the patch; I've applied it to Oort Gnus v0.06 (i. e., CVS).

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mailing lists and news archiving

[Steve Youngs]

|--==> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

  LMI> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
  >>>Uhm.  :-)
  >>
  >>Hm...  I'm not getting any smileys in XEmacs.  Anybody else see this? 

  LMI> I've now made the smiley-ems version work under XEmacs as well.  I
  LMI> plan on removing smiley.el and replacing it with smiley-ems.el in a
  LMI> couple of days...

Just wondering, did you look into why you weren't getting smilies?
I'd hate to think you've fixed something that wasn't broken.  I've
never had problems with smilies in XEmacs.

-- 
|---<Steve Youngs>---------------<GnuPG KeyID: 10D5C9C5>---|
|            XEmacs - It's not just an editor.             |
|                    It's a way of life.                   |
|------------------------------------<youngs@xemacs.org>---|

Re: Mailing lists and news archiving

[Lars Magne Ingebrigtsen]

Steve Youngs <youngs@xemacs.org> writes:

> Just wondering, did you look into why you weren't getting smilies?
> I'd hate to think you've fixed something that wasn't broken.  I've
> never had problems with smilies in XEmacs.

Yes, it had something to do with...  with...  Er.  I forget.

But it was a bug.  :-)

smiley-ems seems to work with both XEmacs and Emacs now, so I think
I'll just remove smiley.el.  (And rename smiley-ems.el to smiley.el.) 

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mailing lists and news archiving

[Steve Youngs]

|--==> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

  LMI> Steve Youngs <youngs@xemacs.org> writes:
  >>Just wondering, did you look into why you weren't getting smilies?
  >>I'd hate to think you've fixed something that wasn't broken.  I've
  >>never had problems with smilies in XEmacs.

  LMI> Yes, it had something to do with...  with...  Er.  I forget.

  LMI> But it was a bug.  :-)

  LMI> smiley-ems seems to work with both XEmacs and Emacs now, so I think
  LMI> I'll just remove smiley.el.  (And rename smiley-ems.el to smiley.el.) 

Cool.  I'll let you know if my smilies suddenly stop working. :-)

-- 
|---<Steve Youngs>---------------<GnuPG KeyID: 10D5C9C5>---|
|            XEmacs - It's not just an editor.             |
|                    It's a way of life.                   |
|------------------------------------<youngs@xemacs.org>---|

Re: Mailing lists and news archiving

[Steve Youngs]

|--==> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

  LMI> Hm...  I'm not getting any smileys in XEmacs.  Anybody else see this? 

Nope.  Smileys work fine here.

-- 
|---<Steve Youngs>---------------<GnuPG KeyID: 10D5C9C5>---|
|            XEmacs - It's not just an editor.             |
|                    It's a way of life.                   |
|------------------------------------<youngs@xemacs.org>---|

Re: Mailing lists and news archiving

[Simon Josefsson]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> * nnir only supports one backend to search in, because it's got 
>   a global variable for the backend.  I wanted to add server 
>   parameters so that the nnir-specific things can be specified 
>   for each server, and nnir would search all servers which are 
>   nnir-enabled.  But I haven't got around to it :-(

FWIW, process marking a nnimap group and pressing G G searches it 
without configuration, I think.  It could work the same for nntp 
groups when the server supports a search command.  Nnml, nnfolder 
etc probably needs configuration though, perhaps defaulting to use 
'grep' could work though?

>   For example, nnimap works with nnir, because IMAP has a 
>   built-in search command.

Searching is limited to one mailbox only, with IMAP, though.  It 
is too slow too search the entire server.

Re: Mailing lists and news archiving

[Russ Allbery]

Chris Beggy <news@kippona.com> writes:

> I'm happy to peer for feeds, but I need to have an estimate of the rate
> of archive growth before I commit to /no expiry/ on those groups.

I'd also be happy to peer for feeds, and I can give some pointers on how
to send control messages and the like.

> Maybe Russ or somebody with deja-news.com experience can shed light on
> these issues.

Here are some things to think about:

 * Spammers troll Usenet for addresses aggressively, perhaps more so than
   with the web and definitely more than with mailing lists.  It's quite
   possible that when a mailing list is gated to Usenet, the spam load of
   the participants will increase.  At least, some people will worry about
   that and complain.

 * Most mailing lists actually are gated out there somewhere in some
   poorly distributed hierarchy.  There really needs to be some sort of
   central, organized hierarchy for gating mailing lists, but in the
   interim you can't really get away with using the mailing list message
   IDs since invariably you'll collide with some other gateway and you'll
   only see some of the messages.  That means assigning new message IDs
   to each message as you gate it, and that means rewriting References so
   that threading still works.

-- 
Russ Allbery (rra@stanford.edu)             <http://www.eyrie.org/~eagle/>

Re: Mailing lists and news archiving

[Lars Magne Ingebrigtsen]

Russ Allbery <rra@stanford.edu> writes:

> I'd also be happy to peer for feeds, and I can give some pointers on how
> to send control messages and the like.

Cool.

>> Maybe Russ or somebody with deja-news.com experience can shed light on
>> these issues.
>
> Here are some things to think about:
>
>  * Spammers troll Usenet for addresses aggressively, perhaps more so than
>    with the web and definitely more than with mailing lists.  It's quite
>    possible that when a mailing list is gated to Usenet, the spam load of
>    the participants will increase.  At least, some people will worry about
>    that and complain.

Right.

>  * Most mailing lists actually are gated out there somewhere in some
>    poorly distributed hierarchy.  There really needs to be some sort of
>    central, organized hierarchy for gating mailing lists, but in the
>    interim you can't really get away with using the mailing list message
>    IDs since invariably you'll collide with some other gateway and you'll
>    only see some of the messages.  That means assigning new message IDs
>    to each message as you gate it, and that means rewriting References so
>    that threading still works.

Ick.  <whiny voice>Can't all the other mailing list gateways rewrite
their Message-IDs so that we can keep ours?</whiny voice>

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mailing lists and news archiving

[Per Abrahamsen]

Russ Allbery <rra@stanford.edu> writes:

>  * There really needs to be some sort of
>    central, organized hierarchy for gating mailing lists, 

Isn't this called Usenet?

If we make the mailing list archives too systematic, they might lose
some of the properties that differentiate them from Usenet.  Mailing
lists tend to appear more exclusive, and less prone to spammers and
trolls.  Especially if they are bi-directional, like Lars want.  

A vaguelly heard rumors of the gnu.* hiearchy being revitalized.  If
that is the case, maybe Lars should ask for the ding list being
gatewayed as gnu.emacs.gnus.development.

Re: Mailing lists and news archiving

[Russ Allbery]

Per Abrahamsen <abraham@dina.kvl.dk> writes:

> Isn't this called Usenet?

> If we make the mailing list archives too systematic, they might lose
> some of the properties that differentiate them from Usenet.  Mailing
> lists tend to appear more exclusive, and less prone to spammers and
> trolls.  Especially if they are bi-directional, like Lars want.

Hm.  I'm not sure whether security via obscurity would help here or not.

> A vaguelly heard rumors of the gnu.* hiearchy being revitalized.  If
> that is the case, maybe Lars should ask for the ding list being
> gatewayed as gnu.emacs.gnus.development.

Yes, I'm working on that revitalization, in fact.  Currently, I'm stuck on
trying to figure out how to generate the correct PGP keys that I need; I
should get back to working on that.  The software is pretty much all
written to generate control messages, which is the first major step.  Then
we just have to write a whole bunch of documentation.

-- 
Russ Allbery (rra@stanford.edu)             <http://www.eyrie.org/~eagle/>

Re: Mailing lists and news archiving

[Per Abrahamsen]

Russ Allbery <rra@stanford.edu> writes:

> Per Abrahamsen <abraham@dina.kvl.dk> writes:
>
>> Isn't this called Usenet?
>
>> If we make the mailing list archives too systematic, they might lose
>> some of the properties that differentiate them from Usenet.  Mailing
>> lists tend to appear more exclusive, and less prone to spammers and
>> trolls.  Especially if they are bi-directional, like Lars want.
>
> Hm.  I'm not sure whether security via obscurity would help here or not.

The obscurity is the only significant difference between Usenet groups
and mailing lists as far as I can see.

>> A vaguelly heard rumors of the gnu.* hiearchy being revitalized.  If
>> that is the case, maybe Lars should ask for the ding list being
>> gatewayed as gnu.emacs.gnus.development.
>
> Yes, I'm working on that revitalization, in fact.  Currently, I'm stuck on
> trying to figure out how to generate the correct PGP keys that I need; I
> should get back to working on that.  The software is pretty much all
> written to generate control messages, which is the first major step.  Then
> we just have to write a whole bunch of documentation.

Sounds excellent!

Re: Mailing lists and news archiving

[Russ Allbery]

Per Abrahamsen <abraham@dina.kvl.dk> writes:
> Russ Allbery <rra@stanford.edu> writes:
>> Per Abrahamsen <abraham@dina.kvl.dk> writes:

>>> If we make the mailing list archives too systematic, they might lose
>>> some of the properties that differentiate them from Usenet.  Mailing
>>> lists tend to appear more exclusive, and less prone to spammers and
>>> trolls.  Especially if they are bi-directional, like Lars want.

>> Hm.  I'm not sure whether security via obscurity would help here or not.

> The obscurity is the only significant difference between Usenet groups
> and mailing lists as far as I can see.

I don't think it's the obscurity so much as having to take a positive
action to join and participate in the mailing list.  I think it serves as
a low-grade filter that weeds out people with only a casual or browsing
interest in the topic.

In some ways this is a bad thing, as there is an audience that the mailing
list doesn't reach.  Some of those people may get slowly sucked in and
eventually become maintainers or significant contributors.

In other ways, though, this is a very good thing.  You don't get the hit
and run vandals on mailing lists like you do on Usenet.  (Well, some of
that also has to do with the fact that the mailing list has a central
point of control and people can be easily banned if it's necessary to do
so.)  There tends to be more topical focus, and more willingness to self-
police in a manner that doesn't just degrade into endless arguments.

I'm not sure how the creation of a lot of mail to news gateways would
change that dynamic, if it does at all.  It feels complicated to me.

-- 
Russ Allbery (rra@stanford.edu)             <http://www.eyrie.org/~eagle/>

Re: Mailing lists and news archiving

[Lars Magne Ingebrigtsen]

Russ Allbery <rra@stanford.edu> writes:

> I don't think it's the obscurity so much as having to take a positive
> action to join and participate in the mailing list.  I think it serves as
> a low-grade filter that weeds out people with only a casual or browsing
> interest in the topic.

My main problem with mailing lists is that I can't read old messages
in the same manner I read new messages.  If I get interested in
subject X, then I can just browse the 400 last messages in the
newsgroup, and use the normal Gnus searching functions.  If it's on a
mailing list, I (more often than not) have to use some weird,
exasperating web interface to the old messages.

Pointy-clickety interfaces to mail are evil.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mailing lists and news archiving

[Karl Eichwalder]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> My main problem with mailing lists is that I can't read old messages
> in the same manner I read new messages.  If I get interested in
> subject X, then I can just browse the 400 last messages in the
> newsgroup, and use the normal Gnus searching functions.

"The 400 last messages" don't count :)  Often you want to browse all
the messages from the year 2000 -- and maybe, these are several
thousends.

> If it's on a mailing list, I (more often than not) have to use some
> weird, exasperating web interface to the old messages.
>
> Pointy-clickety interfaces to mail are evil.

Agreed!  If you're a lucky one they have monthly raw mbox files sitting
around somewhere -- unfortunately, this doesn't seem to be the truth
for the PostgreSQL lists.

-- 
ke@suse.de (work) / keichwa@gmx.net (home):              |
http://www.suse.de/~ke/                                  |      ,__o
Free Translation Project:                                |    _-\_<,
http://www.iro.umontreal.ca/contrib/po/HTML/             |   (*)/'(*)

Re: Mailing lists and news archiving

[Harry Putnam]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I'm going through old mail, and mailing lists I'm subscribed to, but
> haven't read in a while.  And I'm struck by how much of our collective
> knowledge is on these mailing lists, and how ephemeral they are.  Some
> of them are archived here and there, but the archives go away or move
> or don't work.
>
> News, on the other hand, don't really go away if you don't run
> expiry.  If more people are interested mirroring, you just start
> feeding.  Or mirror using other means.
>
> So I'm wondering whether I should start setting up mail-to-news
> gateways for all the mailing lists I'm subscribed to.  And, of course,
> any other mailing lists that people would want me to store.
>
> Does this seem like an OK idea?
>
> The mailing lists I would gateway would be, at least:
>
> XEmacs Beta <xemacs-beta@xemacs.org>
> Emacs Custom <emacs-custom@sunsite.dk>
> XEmacs Mule <xemacs-mule@xemacs.org>
> Emacs Development <emacs-devel@gnu.org>
> Linux Kernel <linux-kernel@vger.kernel.org>
> Linux USB <linux-usb-devel@lists.sourceforge.net>
>
> Is this a good idea?  A bad idea?  And what would the hierarchy be
> called? 

[...]

Its a good idea... My server doesn't carry all that many and its a
sort of commercial big time operator sort of. (Newsguy.com).

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> So I'm wondering whether I should start setting up mail-to-news
>> gateways for all the mailing lists I'm subscribed to. 
>
> I've just browsed my ISP's news server, and there are oodles and
> oodles of gatewayed mailing lists there.  Makes me wonder why anybody
> subscribes to mailing lists.  :-)

So are you still considering it, or are there already many of them
available?

You asked about hierarchy:.. The way freebsd does it is like:

mailing.freebsd.questions
mailing.freebsd.stable
etc

So maybe there is a sort of defacto `mailing' hierarchy.

Re: Mailing lists and news archiving

[Lars Magne Ingebrigtsen]

Harry Putnam <reader@newsguy.com> writes:

> So are you still considering it, or are there already many of them
> available?

I've already done it, sorta.  But I'm wondering 1) whether to feed my
mail repository into the news spool (for instance, I've got something
like 5000 xemacs-beta mails I could feed into the server), and 2)
whether to rename the groups.

> You asked about hierarchy:.. The way freebsd does it is like:
>
> mailing.freebsd.questions
> mailing.freebsd.stable
> etc
>
> So maybe there is a sort of defacto `mailing' hierarchy.

I haven't seen that name elsewhere.  I've seen "fa.", though.

How about "gmane", which is short for "Gnu MAil-to-NEws"?  And is
pronounced "main", of course.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mailing lists and news archiving

[Harry Putnam]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Harry Putnam <reader@newsguy.com> writes:
>
>> So are you still considering it, or are there already many of them
>> available?
>
> I've already done it, sorta.  But I'm wondering 1) whether to feed my
> mail repository into the news spool (for instance, I've got something
> like 5000 xemacs-beta mails I could feed into the server), and 2)
> whether to rename the groups.
>
>> You asked about hierarchy:.. The way freebsd does it is like:
>>
>> mailing.freebsd.questions
>> mailing.freebsd.stable
>> etc
>>
>> So maybe there is a sort of defacto `mailing' hierarchy.
>
> I haven't seen that name elsewhere.  I've seen "fa.", though.
>
> How about "gmane", which is short for "Gnu MAil-to-NEws"?  And is
> pronounced "main", of course.

Good as any.. another choice might be g-m2n


Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Steinar Bang <sb@dod.no> writes:
>
>> Would it be possible to provide some sort of emacs-friendly search
>> interface to your newsspool?
>
> Of course.
>
> I'm just wondering whether it's possible to do this whether NNTP or
> whether some extra-protocol channel has to be used.  innd probably
> doesn't come with glimpse or wais pre-installed, but would it be
> possible to have a new NNTP command that would hand off a query to an
> external agent and output the result from that?
>
> Like:
>
> XSEARCH group:gnus.ding search.*term
>
> innd would then do system("/usr/bin/search-program") with the search
> terms from stdin, and just output what the search-program outputs.  It
> could, for instance, return a list of headers with in valid NNTP
> format.
>
> I could write a proof-of-concept version of "search-program"...

Karl K, is something of a `swish++' advocate, in my own experience it is
something of a dog to build and get running... never did get it
running. But Karl claims it is fast.

Re: Mailing lists and news archiving

[Lars Magne Ingebrigtsen]

Harry Putnam <reader@newsguy.com> writes:

> Karl K, is something of a `swish++' advocate, in my own experience it is
> something of a dog to build and get running... never did get it
> running. But Karl claims it is fast.

I haven't tried running it, but installing it was easy enough --
"apt-get install swish++".  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mailing lists and news archiving

[Harry Putnam]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I haven't tried running it, but installing it was easy enough --
> "apt-get install swish++".  :-)

Erk... Maybe I should start running demian.

Re: Mailing lists and news archiving

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>> Karl K, is something of a `swish++' advocate, in my own experience it is
>> something of a dog to build and get running... never did get it
>> running. But Karl claims it is fast.
>
> I haven't tried running it, but installing it was easy enough --
> "apt-get install swish++".  :-)

Arrrgh.  I see what you mean about getting it running.  I just can't
find any way to get it to index files that don't have extensions in
the names.  How annoying.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mailing lists and news archiving

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Arrrgh.  I see what you mean about getting it running.  I just can't
> find any way to get it to index files that don't have extensions in
> the names.  How annoying.

I installed the Debian source package for swish++ and chopped off the
test for extensions, and now it'll index my spool.  However, it knows
nothing about news, so it's indexing headers as well, which isn't too
helpful.  I'm guessing it knows nothing about MIME, so it'll be unable
to index QP- and base64-encoded text.

Somebody must have written an indexer that's specially geared towards
mail and news...

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mailing lists and news archiving

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Somebody must have written an indexer that's specially geared towards
> mail and news...

Somebody has, and it's called swish++.  :-)

The version in Potato Gnus (which is what Quimby is running) is 3.0.3,
while the current, spiffy version of 5.5.3.

I shoulda installed Woody Gnus.  :-/

Oh, well, here goes building 5.5.3...

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mailing lists and news archiving

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Oh, well, here goes building 5.5.3...

And, as promised, it was painful.  But it's built now.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mailing lists and news archiving

[Karl Kleinpaste]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
> And, as promised, it was painful.  But it's built now.

Other than bumping into gcc bugs in handling some of swish++'s
constructs, I didn't find it hard to build.  My swish++.conf is at
http://www.cs.cmu.edu/~karl/gnus/swish++.conf.

It indexes my 560Mb archive in less than 10 minutes every morning.

Re: Mailing lists and news archiving

[Colin Marquardt]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
>> Oh, well, here goes building 5.5.3...
>
> And, as promised, it was painful.  But it's built now.

Didn't 

  apt-get -b source swish++

work? See
  <http://www.debian.org/doc/manuals/apt-howto/ch-sourcehandling.en.html>

You might need a newer apt though, but there should be a package for
Potato.

Cheers,
  Colin

Re: Mailing lists and news archiving

[Lars Magne Ingebrigtsen]

Colin Marquardt <c.marquardt@alcatel.de> writes:

> You might need a newer apt though, but there should be a package for
> Potato.

Sure.  But the version was ancient.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mailing lists and news archiving

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> Colin Marquardt <c.marquardt@alcatel.de> writes:

>> You might need a newer apt though, but there should be a package
>> for Potato.

> Sure.  But the version was ancient.

If 5.3 is new enough for you, you could temporarily change your
/etc/apt/sources.list to "testing" like I suggested earlier, but use
the build from source option instead of installing the binary package.
That would hopefully take care of the library dependencies.

The "unstable" version is 5.5.3, btw.  

Re: Mailing lists and news archiving

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> If 5.3 is new enough for you, you could temporarily change your
> /etc/apt/sources.list to "testing" like I suggested earlier, but use
> the build from source option instead of installing the binary package.
> That would hopefully take care of the library dependencies.

I've built it from source, and all is well.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mailing lists and news archiving

[Colin Marquardt]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Steinar Bang <sb@dod.no> writes:
>
>> If 5.3 is new enough for you, you could temporarily change your
>> /etc/apt/sources.list to "testing" like I suggested earlier, but use
>> the build from source option instead of installing the binary package.
>> That would hopefully take care of the library dependencies.
>
> I've built it from source, and all is well.  :-)

I just wanted to show off Debian's apt-get features anyway :)

Cheers,
  Colin

Re: Mailing lists and news archiving

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> The version in Potato Gnus (which is what Quimby is running) is
> 3.0.3, while the current, spiffy version of 5.5.3.

> I shoulda installed Woody Gnus.  :-/

s/Gnus/debian/, perhaps...? :-)

In any case the version of swish++ in testing (which is currently a
bit ahead of the frozen woody), is only 5.1.

I've been running three machines on testing with regular upgrades,
since early this summer, without any major problems.

And even if you don't wish to upgrade, here's what you could do:
  - change "stable" to "testing" in /etc/apt/sources.list
  - do the following commands:
	apt-get update
	apt-get --dry-run install swish++
  - if this doesn't seem to wish to pull in any new shared libs, which
    may cause you to have to update other packages, just do
	apt-get install swish++
 - change "testing" back to "stable" in /etc/apt/sources.list, and do
   a new
	apt-get update

> Oh, well, here goes building 5.5.3...

This is of course a lot newer, and probably spiffier than 5.1.

Re: Mailing lists and news archiving

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

[snip!]
> So I'm wondering whether I should start setting up mail-to-news
> gateways for all the mailing lists I'm subscribed to.  And, of course,
> any other mailing lists that people would want me to store.

> Does this seem like an OK idea?

Yes, I think it does.

Would it be possible to provide some sort of emacs-friendly search
interface to your newsspool?

Re: Mailing lists and news archiving

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Would it be possible to provide some sort of emacs-friendly search
> interface to your newsspool?

Of course.

I'm just wondering whether it's possible to do this whether NNTP or
whether some extra-protocol channel has to be used.  innd probably
doesn't come with glimpse or wais pre-installed, but would it be
possible to have a new NNTP command that would hand off a query to an
external agent and output the result from that?

Like:

XSEARCH group:gnus.ding search.*term

innd would then do system("/usr/bin/search-program") with the search
terms from stdin, and just output what the search-program outputs.  It
could, for instance, return a list of headers with in valid NNTP
format.

I could write a proof-of-concept version of "search-program"...

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mailing lists and news archiving

[Frank Schmitt]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

>Like:
>
>XSEARCH group:gnus.ding search.*term
>
>innd would then do system("/usr/bin/search-program") with the search
>terms from stdin, and just output what the search-program outputs.  It
>could, for instance, return a list of headers with in valid NNTP
>format.
>
>I could write a proof-of-concept version of "search-program"...

Please don't laugh, but what's wrong with XPAT search? A bit of
increasing it's capabilities and calling it XPAT2...

-- 
One Ring to rule them all, One Ring to find them,
One Ring to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie.
19. Dezember 2001

Re: Mailing lists and news archiving

[Florian Weimer]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Linux Kernel <linux-kernel@vger.kernel.org>

> Is this a good idea?

AFAIK, bidirectional gateways are not welcome for linux-kernel.

Re: Mailing lists and news archiving

[Martin Thornquist]

[ Lars Magne Ingebrigtsen ]

> So I'm wondering whether I should start setting up mail-to-news
> gateways for all the mailing lists I'm subscribed to.  And, of course,
> any other mailing lists that people would want me to store.

nntp.ifi.uio.no (Department of Informatics, University of Oslo,
restricted access to inside of uio.no) gateways a few lists. They call
the hierarchy fa.* (e.g. fa.linux.kernel), and it's actually fed into
Google from there. If you want to peer with them I can talk to the
resposible parties.


Martin
-- 
"An ideal world is left as an exercise to the reader."
                                                 -Paul Graham, On Lisp

Re: Mailing lists and news archiving

[Lars Magne Ingebrigtsen]

Martin Thornquist <martint@ifi.uio.no> writes:

> nntp.ifi.uio.no (Department of Informatics, University of Oslo,
> restricted access to inside of uio.no) gateways a few lists. They call
> the hierarchy fa.* (e.g. fa.linux.kernel), and it's actually fed into
> Google from there. If you want to peer with them I can talk to the
> resposible parties.

The fa.* hierarchy is a mail-to-news gateway thing.  I want a
mail-to-news-and-back-again hierarchy.  :-)

Or, at the very least, for mailing lists that don't want that, I want
people who post to these newsgroups to get an autoresponder message
back suggesting that they send mail to the mailing list instead of
posting to essentially a dead newsgroup.

(Me?  Post to a fa.* group and wondering why I never got any response?
Never!) 

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Mailing lists and news archiving

[Simon Josefsson]

On Mon, 7 Jan 2002, Lars Magne Ingebrigtsen wrote:

> Martin Thornquist <martint@ifi.uio.no> writes:
> 
> > nntp.ifi.uio.no (Department of Informatics, University of Oslo,
> > restricted access to inside of uio.no) gateways a few lists. They call
> > the hierarchy fa.* (e.g. fa.linux.kernel), and it's actually fed into
> > Google from there. If you want to peer with them I can talk to the
> > resposible parties.
> 
> The fa.* hierarchy is a mail-to-news gateway thing.  I want a
> mail-to-news-and-back-again hierarchy.  :-)

Is it bidirectional?  Most lists only accepts posts from subscribers these 
days, how will this work?  Replace From: and add a Reply-To or 
Mail-Reply-To or something?

Re: Mailing lists and news archiving

[Fabien Penso]

[-- Attachment #1: Type: text/plain, Size: 779 bytes --]


Lars on Sat, 05 Jan 2002 20:36:40 +0100 wrote:

 > So I'm wondering whether I should start setting up mail-to-news
 > gateways for all the mailing lists I'm subscribed to.  And, of course,
 > any other mailing lists that people would want me to store.


[...]

Just to say we have a dedicated usenet server for gnu.* fr.* comp.* on
news.linuxfr.org which is open for everyone (read-only). We also offer
personnal accounts with write access to it (see
http://linuxfr.org/usenet/ - english at bottom). We also have some
lists.

Maybe could we also get feeded for this from your usenet server, or host
the groups if it helps.

-- 
Fabien Penso <penso@linuxfr.org> | LinuxFr a toujours besoin de :
http://perso.LinuxFR.org/penso/  | http://linuxfr.org/dons/


[-- Attachment #2: Type: application/pgp-signature, Size: 239 bytes --]