compressed nnml files?

compressed nnml files?

[Colin Rafferty]

Hi-

I love nnml.  I do not want to change back to mbox/babyl.  With that
said, here is my problem and what I would like:

I have some nnml mail groups that should not be auto-expirable, are
active, and are large.  There are hundreds of messages, and some of them
can be huge.

I would like to be able to compress old articles and still be able to
read them normally.  August Gnus does not play well with crypt++.  I
imagine that September Gnus has the same problems.

Is there something that I can do to handle this?  If not, is there
something that someone else can do?

This could be very useful for the mailing list archive, as it would end
up taking up less space wherever it is, and would reduce the time of
sending articles through ange-ftp.

-Colin

compressed nnml files?

[Colin Rafferty]

Hi-

I love nnml.  I do not want to change back to mbox/babyl.  With that
said, here is my problem and what I would like:

I have some nnml mail groups that should not be auto-expirable, are
active, and are large.  There are hundreds of messages, and some of them
can be huge.

I would like to be able to compress old articles and still be able to
read them normally.  August Gnus does not play well with crypt++.  I
imagine that September Gnus has the same problems.

Is there something that I can do to handle this?  If not, is there
something that someone else can do?

This could be very useful for the mailing list archive, as it would end
up taking up less space wherever it is, and would reduce the time of
sending articles through ange-ftp.

-Colin

Re: compressed nnml files?

[Lars Magne Ingebrigtsen]

craffert@sps.ml.com (Colin Rafferty) writes:

> I would like to be able to compress old articles and still be able to
> read them normally.  August Gnus does not play well with crypt++.  I
> imagine that September Gnus has the same problems.

Yes.  It's on the todo list, but the reason I haven't done anything
about it is that I don't use crypt++, so it hasn't really been a
problem for me.  :-)  I'll probably start using it soonish, so this
might also start working sometime in the future...

-- 
Home is where the cat is.

Re: compressed nnml files?

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----

>>>>> "CR" == Colin Rafferty <craffert@sps.ml.com> writes:

CR> I would like to be able to compress old articles and still be able
CR> to read them normally.  August Gnus does not play well with crypt++.
CR> I imagine that September Gnus has the same problems.

crypt++ is not a good choice for transparent, on-the-fly compression;
for that I recomend jka-compr which is also distributed with Emacs now.
But that alone won't work because nnml.el won't operate correctly on the
compressed files.  For that a little bit of hacking in nnml.el is in
order:

- -----
#!/bin/sh

if test -f x-easymenu.el; then
  mv x-easymenu.el easymenu.el
fi

cp nnml.el nnml.el.orig
cat nnml.el.orig | sed 's/\[0-9\]+/\[0-9\]+\\\\(\\\\.gz\\\\)?/g' > nnml.el
- -----

This will at least let nnml work with gzipped mail files for the most
common tasks, allowing jka-compr to do it's thing.  There are a number
of other segments of Gnus' code that operate on directory listings that
this little bit of hackery won't patch, and I wouldn't rely on a brute-
force sed to do it.

The best complete solution, as I see it, is to build a regular
expression from jka-compr-compression-info-list, then modify any code
that references file name in a buffer to include that regexp.  In short,
it's going to be a pain no matter what.

-----BEGIN PGP SIGNATURE-----
Version: 2.6.2
Comment: Processed by Mailcrypt 3.4, an Emacs/PGP interface

iQCVAwUBMMiSeJ6VRH7BJMxHAQGr0gP/TSTBogMe50QDu1smAYc5QOH6oMaPiXPZ
weLiu1u+C/VYsYjce1wztf9U/BRE1E06oxjudEh2xUbhUyzjo8qSLFBbdDBED0qA
nlDLgcARAN3gmDx/mPxosp5Jt6zmYjRe3hL3R4KEdw0PqZEvGCs9DXQQxQ89OHya
bPv1yF5vV48=
=qlji
-----END PGP SIGNATURE-----
-- 
Rat <ratinox@ccs.neu.edu>          \ Do not taunt Happy Fun Ball.
PGP Public Key: Ask for one today!  \ 
http://www.ccs.neu.edu/home/ratinox/ \ 

Re: compressed nnml files?

[Lars Magne Ingebrigtsen]

Stainless Steel Rat <ratinox@ccs.neu.edu> writes:

> The best complete solution, as I see it, is to build a regular
> expression from jka-compr-compression-info-list, then modify any code
> that references file name in a buffer to include that regexp.  In short,
> it's going to be a pain no matter what.

I've played a bit with nnml, and making it jka-comprehensible wasn't
that difficult.  It does feel a bit kludgy, though, so I'm not doing
the same with the other backends right away.

The real pain comes when trying to do
`(insert-file-contents (int-to-string article))', when we really want
article with ".gz" (or whatever) appended...  sigh...

-- 
Home is where the cat is.

Re: compressed nnml files?

[Colin Rafferty]

Stainless Steel Rat writes:
>>>>>> "CR" == Colin Rafferty <craffert@sps.ml.com> writes:

CR> I would like to be able to compress old articles and still be able
CR> to read them normally.  August Gnus does not play well with crypt++.
CR> I imagine that September Gnus has the same problems.

> crypt++ is not a good choice for transparent, on-the-fly compression;
> for that I recomend jka-compr which is also distributed with Emacs now.
> But that alone won't work because nnml.el won't operate correctly on the
> compressed files.  For that a little bit of hacking in nnml.el is in
> order:

Actually, I think that crypt++ is better because it also gives you the
ability to have encrypted mail messages as well.  Since it recognizes
compressed/encrypted files by contents (which works very well with mail
files), you can leave the files named the same (as numbers).

In fact, I could see adding a command gnus-summary-encrypt-article.

Here is my solution for my own problem (in my .gnus).  It handles
reading mail, just editing and then saving a compressed/encrypted file:

(require 'nnmail)
(defun nnmail-find-file (file)
  "Insert FILE in server buffer safely."
  (set-buffer nntp-server-buffer)
  (erase-buffer)
  (condition-case ()
      (progn
	(if (fboundp 'crypt-insert-file-contents)
	    (let ((crypt-auto-decode-insert t))
	      (crypt-insert-file-contents file))
	  (insert-file-contents file))
	     t)
    (file-error nil)))

; -Colin

Re: compressed nnml files?

[Stainless Steel Rat]

-----BEGIN PGP SIGNED MESSAGE-----

>>>>> "CR" == Colin Rafferty <craffert@sps.ml.com> writes:

CR> Actually, I think that crypt++ is better because it also gives you
CR> the ability to have encrypted mail messages as well.

That doesn't change the fact that crypt++ cannot do on-the-fly file
compression like jka-compr can.  You have to specify the compression
extensions with crypt++; jka-compr does it automatically.  Okay, you
need jka-aux to do it, big deal.

There is a way to have them coexist, with compression handled by
jka-compr and encryption handled by crypt++ but as I've never seen the
need (or use!)  of single-key encryption on an open system I've never
bothered trying to make it work.

But I suppose the point is moot; jka-compr and crypt++ are both
distributed with GNU Emacs now, so if support for one is included in
Gnus then support for the other should also be included.

-----BEGIN PGP SIGNATURE-----
Version: 2.6.2
Comment: Processed by Mailcrypt 3.4, an Emacs/PGP interface

iQCVAwUBMMxWDp6VRH7BJMxHAQF2wQP/V27plVwGKREn+7drZpSHQXyFjOhvUkf8
+WE5D8WHn8hBEMhCKRbI6SxMabDM+6JauSRzbue2y4/WbAbqUNRIMzkwMnNX/a8e
kxQvmvNBVcwOnbFG4DtDfGOpKzmbZVCCbBtD64MwhqEBTBzBEX8DnEFjtH8+hZxT
n9UtqC4zykY=
=fi9N
-----END PGP SIGNATURE-----
-- 
Rat <ratinox@ccs.neu.edu>          \ Do not use Happy Fun Ball on concrete.
PGP Public Key: Ask for one today!  \ 
http://www.ccs.neu.edu/home/ratinox/ \ 

Re: compressed nnml files?

[Greg Stark]

Hmm, ideally it would be nice if it was possible to access nnml files from
within a tar file with tar-mode.el ...
Date: Sat, 6 Jan 96 22:12:10 -0500
From: Mail Delivery Subsystem <MAILER-DAEMON@MIT.EDU>
Subject: Returned mail: Unable to deliver mail
To: gsstark@MIT.EDU

   ----- Transcript of session follows -----
554 gsstark... Bad format for qf command: ding (ding

   ----- Unsent message follows -----
Received: by fierce-bad-rabbit.MIT.EDU (5.57/4.7) id AA11416; Sat, 6 Jan 96 22:12:10 -0500
Sender: gsstark@fierce-bad-rabbit.MIT.EDU
To: (ding) GNUS Mailing List <ding@ifi.uio.no>
Subject: feature request: patch file digests
From: gsstark@mit.edu (Greg Stark)
Date: 06 Jan 1996 22:12:09 -0500
In-Reply-To: craffert@sps.ml.com's message of Thu, 7 Dec 95 14:52:03 EST
Message-Id: <ycqn380r94m.fsf@fierce-bad-rabbit.MIT.EDU>
Organization: Massachvsetts Institvte of Technology
Lines: 17


Hmm, There's this group I "read",  de.alt.sources.linux.patches, which posts
patches for new linux kernels.  Sort of handy for watching for when the
drivers I use actually change or new features get added.  But reading over
then thousand line patches that get posted in 7 parts is kinda awkward.

It would be cool if someone wrote a digest format for patch files.  
Each file could show up as a separate article making it easy to jump straight
to the file I want to see.  Hmm, thinking about this it might make more sense
to write a patch-mode.el and shar-mode.el and just have nndoc call them on the
article.  

I could try writing this, and probably will try. 
But I'm not sure I'll be able to do it

--
greg
Date: Sat, 6 Jan 96 23:01:46 -0500
From: Mail Delivery Subsystem <MAILER-DAEMON@MIT.EDU>
Subject: Returned mail: Unable to deliver mail
To: gsstark@MIT.EDU

   ----- Transcript of session follows -----
554 gsstark... Bad format for qf command: ding (ding

   ----- Unsent message follows -----
Received: by fierce-bad-rabbit.MIT.EDU (5.57/4.7) id AA11444; Sat, 6 Jan 96 23:01:46 -0500
Sender: gsstark@fierce-bad-rabbit.MIT.EDU
To: (ding) GNUS Mailing List <ding@ifi.uio.no>
Subject: feature request: handle broken From headers that include signature
From: gsstark@mit.edu (Greg Stark)
Date: 06 Jan 1996 23:01:42 -0500
In-Reply-To: craffert@sps.ml.com's message of Thu, 7 Dec 95 14:52:03 EST
Message-Id: <ycqlonkr6u1.fsf@fierce-bad-rabbit.MIT.EDU>
Organization: Massachvsetts Institvte of Technology
Lines: 19


There's a particular configuration error some people make that causes their
mailer to use their .signature file as their mail signature.  That is, use
their 5+ line signature in the From header of their mail messages.

Gnus doesn't simplify the header in any way when this happens, it does
something like this:

  +[  26: --------------------] Cockatrices, Basilisks and other Gargoyle Manufacturers

From: ------------------------------------------------------------------------------ <name-witheld@MIT.EDU>

Obviously the best it can do is some kind of heuristic, but it seems like Gnus
should be able to tell at least sometimes that it's not looking at the
author's name.  I would need to see a few more examples before I even had a
decent suggestion for what the heuristic should be.

--
greg

Re: compressed nnml files?

[Lars Magne Ingebrigtsen]

Greg Stark <gsstark@MIT.EDU> writes:

> Hmm, ideally it would be nice if it was possible to access nnml files from
> within a tar file with tar-mode.el ...

Well, I see one thing that would make it not-totally-trivial to just
insert something tar-mody between nnml and the files, and that is that
one can't add new files to tar-mode "directories".  But just reading
would be ok.

I don't know what would be cleanest -- add a new backend (nntar?),
or just make nnml tar-enabled?  If I were to add a new backend, it
would probably work as a frontend for all the other backends, so you
could have tar nnmh, nnfolder (etc) groups.  But they would all be
read-only, wouldn't they?

Do we need this?

-- 
   Lars Magne Ingebrigtsen * larsi@ifi.uio.no
      (a red leaf that falls from the purple tree)

Re: compressed nnml files?

[Kai Grossjohann]

>>>>> On 10 Jan 1996 19:25:21 +0100, Lars Magne Ingebrigtsen
>>>>> <larsi@ifi.uio.no> said:

>>>>> Greg Stark <gsstark@MIT.EDU> writes:

  Greg> Hmm, ideally it would be nice if it was possible to access
  Greg> nnml files from within a tar file with tar-mode.el ...

  Lars> Do we need this?

I think we don't.  You can use nnfolder for this, I think, which is
already a single file.

        kai
--
I like both kinds of music.

Re: compressed nnml files?

[Joe Hildebrand]

> Greg Stark <gsstark@MIT.EDU> writes: 

   Greg> Hmm, ideally it would be nice if it was possible to access
   Greg> nnml files from within a tar file with tar-mode.el ...

> "Lars" == Lars Magne Ingebrigtsen <larsi@ifi.uio.no> writes:

   Lars> Do we need this?

No.  Instead of tarring your nnml files together, make them into a
digest.  I think there is a script at http://www.fuentez.com/ding to
do that.  Poof.  Best of all possible worlds.  Less inodes, compressed
file (at least in 0.27...), no new code for Lars, convenient interface
(once digests start working again).

-- 
Joe Hildebrand                  Fuentez Systems Concepts
hildjj@fuentez.com              11781 Lee-Jackson Hwy, Suite 700
Lead Software Engineer          Fairfax, VA 22033
	"Breakfast recapitulates phylogeny" - Spider Robinson

Re: compressed nnml files?

[Aharon (Al) Schkolnik]

>>>>> "Joe" == Joe Hildebrand <hildjj@fuentez.com> writes:

    >> Greg Stark <gsstark@MIT.EDU> writes:

    Greg> Hmm, ideally it would be nice if it was possible to access
    Greg> nnml files from within a tar file with tar-mode.el ...

    >> "Lars" == Lars Magne Ingebrigtsen <larsi@ifi.uio.no> writes:

    Lars> Do we need this?

    Joe> No.  Instead of tarring your nnml files together, make them
    Joe> into a digest.  I think there is a script at
    Joe> http://www.fuentez.com/ding to do that.  Poof.  Best of all
    Joe> possible worlds.  Less inodes, compressed file (at least in
    Joe> 0.27...), no new code for Lars, convenient interface (once
    Joe> digests start working again).


	http://www.fuentez.com/ding  moved to 
	http://www.fuentez.com/public-info/ding/ding.html
	
--
  The day is short, and the work is great,     |   Aharon Schkolnik
  and the laborers are lazy, and the reward    |   Aharon@Matat.Health.Gov.IL
  is great, and the Master of the house is     |
  impatient. - Ethics Of The Fathers Ch. 2     |

Re: compressed nnml files?

[Joe Hildebrand]

> "Aharon" == Aharon (Al) Schkolnik <aharon@osfstats.health.gov.il> writes:

   Aharon> http://www.fuentez.com/ding moved to
   Aharon> http://www.fuentez.com/public-info/ding/ding.html
	
oops.  Sorry about that.  The other webmasters moved things around.
There is now a symlink, so either should work.

-- 
Joe Hildebrand                  Fuentez Systems Concepts
hildjj@fuentez.com              11781 Lee-Jackson Hwy, Suite 700
Lead Software Engineer          Fairfax, VA 22033
	"Breakfast recapitulates phylogeny" - Spider Robinson