gnus-mime-copy-part and jka-compr

gnus-mime-copy-part and jka-compr

[Kevin Ryde]

I wonder if it'd be possible for gnus-mime-copy-part (`c') to
automatically decompress a file like foo.tar.bz2 in a message.

It doesn't seem to cooperate with auto-compression-mode at the moment.
It ends up with a buffer that's still got the compressed data but has
been put into tar-mode, which of course is not right.

It seems jka-compr works by hooking insert-file-contents with
file-name-handler-alist, and leaving a do-nothing ("\\.bz2\\'" nil
jka-compr) in auto-mode-alist, so the normal-mode call in
gnus-mime-copy-part doesn't do the trick.

Re: gnus-mime-copy-part and jka-compr

[Lars Magne Ingebrigtsen]

Kevin Ryde <user42@zip.com.au> writes:

> I wonder if it'd be possible for gnus-mime-copy-part (`c') to
> automatically decompress a file like foo.tar.bz2 in a message.
>
> It doesn't seem to cooperate with auto-compression-mode at the moment.
> It ends up with a buffer that's still got the compressed data but has
> been put into tar-mode, which of course is not right.

That's an interesting problem.

The solution is probably also very interesting, but I don't quite see
how.  `c' just puts the bytes in a buffer and calls `normal-mode',
which will make the right mode appear.  However, for strange modes
like tar mode, that just doesn't seem to work -- it gets triggered
even though the file is called .tar.gz, but doesn't seem to accept
having gzipped contents.

So -- should `c' run stuff through jka to decompress things?  And how
is that done without actually putting things in a file?  Anybody got
any ideas?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: gnus-mime-copy-part and jka-compr

[Lars Magne Ingebrigtsen]

[-- Attachment #1: Type: text/plain, Size: 59 bytes --]

Oh, and here's a test tar file that displays the problem:


[-- Attachment #2: g.tar.gz --]
[-- Type: application/octet-stream, Size: 143 bytes --]

[-- Attachment #3: Type: text/plain, Size: 159 bytes --]


`c' on that MIME boundary up there and get beeped at.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: gnus-mime-copy-part and jka-compr

[Kai Großjohann]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> So -- should `c' run stuff through jka to decompress things?  And how
> is that done without actually putting things in a file?  Anybody got
> any ideas?

I think it's better not to try to avoid the file.  jka-compr calls
call-process and passes the contents as a file.
-- 
Ambibibentists unite!

Re: gnus-mime-copy-part and jka-compr

[Lars Magne Ingebrigtsen]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> I think it's better not to try to avoid the file.  jka-compr calls
> call-process and passes the contents as a file.

Then the thing has to be written out to a temporary file (carefully,
so that jka doesn't compress what's already compressed), and then
read back in again.  And then the file has to be deleted.

What Emacs needs is a RAM-based temporary file system.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: gnus-mime-copy-part and jka-compr

[Kai Großjohann]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> What Emacs needs is a RAM-based temporary file system.  :-)

Actually, just a few minutes ago I suggested to make call-process be
a file operation.  (The handler would be chosen based on
default-directory.)

With this change, you could build a filename handler that does this.

However, one would still have to pass the data to the external
process somehow.  Hmmm...
-- 
Ambibibentists unite!

Re: gnus-mime-copy-part and jka-compr

[Lars Magne Ingebrigtsen]

kai.grossjohann@uni-duisburg.de (Kai Großjohann) writes:

> Actually, just a few minutes ago I suggested to make call-process be
> a file operation.  (The handler would be chosen based on
> default-directory.)
>
> With this change, you could build a filename handler that does this.
>
> However, one would still have to pass the data to the external
> process somehow.  Hmmm...

Well, for gzip you can just put the stuff in a buffer, do "gunzip -c"
in a `shell-command-on-region' and use the output.  So that can be
done, but that's avoiding jka-compr altogether and kinda
re-implementing the entire package on a buffer basis instead of a
file basis.  Which we don't want to do.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: gnus-mime-copy-part and jka-compr

[Kevin Ryde]

[I posted this to ding@gnus the other day, but it didn't seem to make
it through to gmane.  Apologies to anyone getting a duplicate.]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
> `c' just puts the bytes in a buffer and calls `normal-mode',
> which will make the right mode appear.  However, for strange modes
> like tar mode, that just doesn't seem to work -- it gets triggered
> even though the file is called .tar.gz, but doesn't seem to accept
> having gzipped contents.

jka-compr puts a dummy into auto-mode-alist to lose the .gz, assuming
that its uncompress will have already been done via
file-name-handler-alist.  Or I think that's what's going on.

> Well, for gzip you can just put the stuff in a buffer, do "gunzip -c"
> in a `shell-command-on-region' and use the output.

I should have followed up earlier.  I had a go myself and came up with
the stuff below.  It uses call-process-region based on info from
jka-compr's setups.

(The only change to gnus-mime-copy-part is the addition of the call to
gnus-mime-jka-compr-maybe-uncompress.  I've got this stuff in my
.gnus.el, hence copying the code to make that insertion.)

> So that can be
> done, but that's avoiding jka-compr altogether and kinda
> re-implementing the entire package on a buffer basis instead of a
> file basis.  Which we don't want to do.

I don't think jka-compr has anything for acting directly on a buffer
as such, it just seems oriented towards file reading and writing, as a
filename handler.


In any case, what I'm not at all sure about below is coding system
issues.

Does call-process-region do enough on its own, or does it want to be
helped by working the notional filename (from the mail message) into
its figurings?

I'm thinking of say .tar wanting raw 8-bit but .txt wanting the usual
emacs inferences.




;; jka-compr.el uses a "sh -c" to direct stderr to err-file, but these days
;; emacs can do that itself.
;;
(defun gnus-mime-jka-compr-maybe-uncompress ()
  "Uncompress the current buffer if `auto-compression-mode' is enabled.
The uncompress method used is derived from `buffer-file-name'."
  (when (and (fboundp 'jka-compr-installed-p)
             (jka-compr-installed-p))
    (let ((info (jka-compr-get-compression-info buffer-file-name)))
      (when info
        (let ((basename (file-name-nondirectory buffer-file-name))
              (args     (jka-compr-info-uncompress-args    info))
              (prog     (jka-compr-info-uncompress-program info))
              (message  (jka-compr-info-uncompress-message info))
              (err-file (jka-compr-make-temp-name)))
          (if message
              (message "%s %s..." message basename))
          (unwind-protect
              (unless (memq (apply 'call-process-region
                                   (point-min) (point-max) 
                                   prog
                                   t (list t err-file) nil
                                   args)
                            jka-compr-acceptable-retval-list)
                (jka-compr-error prog args basename message err-file))
            (jka-compr-delete-temp-file err-file)))))))

(require 'gnus-art)
(defun gnus-mime-copy-part (&optional handle)
  "Put the MIME part under point into a new buffer."
  (interactive)
  (gnus-article-check-buffer)
  (let* ((handle (or handle (get-text-property (point) 'gnus-data)))
	 (contents (and handle (mm-get-part handle)))
	 (base (and handle
		    (file-name-nondirectory
		     (or
		      (mail-content-type-get (mm-handle-type handle) 'name)
		      (mail-content-type-get (mm-handle-disposition handle)
					     'filename)
		      "*decoded*"))))
	 (buffer (and base (generate-new-buffer base))))
    (when contents
      (switch-to-buffer buffer)
      (insert contents)
      ;; We do it this way to make `normal-mode' set the appropriate mode.
      (unwind-protect
	  (progn
	    (setq buffer-file-name (expand-file-name base))
            (gnus-mime-jka-compr-maybe-uncompress)
	    (normal-mode))
	(setq buffer-file-name nil))
      (goto-char (point-min)))))

Re: gnus-mime-copy-part and jka-compr

[Lars Magne Ingebrigtsen]

Kevin Ryde <user42@zip.com.au> writes:

> I should have followed up earlier.  I had a go myself and came up with
> the stuff below.  It uses call-process-region based on info from
> jka-compr's setups.

Nice.  I'd like to include that in Gnus.  Do you have FSF copyright
assignment papers on file, or if not -- can you sign such papers?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: gnus-mime-copy-part and jka-compr

[Kevin Ryde]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
> Do you have FSF copyright assignment papers on file,

Yes.

I guess the changelog would be not much more than

	* gnus-art.el (gnus-mime-jka-compr-maybe-uncompress): New function.
	(gnus-mime-copy-part): Use it.

Re: gnus-mime-copy-part and jka-compr

[Lars Magne Ingebrigtsen]

Kevin Ryde <user42@zip.com.au> writes:

> I guess the changelog would be not much more than
>
> 	* gnus-art.el (gnus-mime-jka-compr-maybe-uncompress): New function.
> 	(gnus-mime-copy-part): Use it.

Can you send me the previous code changes as a diff?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: gnus-mime-copy-part and jka-compr

[Kevin Ryde]

[-- Attachment #1: Type: text/plain, Size: 107 bytes --]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
> Can you send me the previous code changes as a diff?


[-- Attachment #2: gnus-art.el.uncompress.diff --]
[-- Type: text/plain, Size: 2366 bytes --]

--- gnus-art.el.~6.265.~	2003-01-04 10:22:58.000000000 +1000
+++ gnus-art.el	2003-01-04 10:32:28.000000000 +1000
@@ -1,5 +1,5 @@
 ;;; gnus-art.el --- article mode commands for Gnus
-;; Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002
+;; Copyright (C) 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003
 ;;        Free Software Foundation, Inc.
 
 ;; Author: Lars Magne Ingebrigtsen <larsi@gnus.org>
@@ -3907,6 +3907,36 @@ (defun gnus-mime-view-part-as-type (&opt
 	    (mm-merge-handles gnus-article-mime-handles handle))
       (gnus-mm-display-part handle))))
 
+(eval-when-compile
+  (require 'jka-compr))
+
+;; jka-compr.el uses a "sh -c" to direct stderr to err-file, but these days
+;; emacs can do that itself.
+;;
+(defun gnus-mime-jka-compr-maybe-uncompress ()
+  "Uncompress the current buffer if `auto-compression-mode' is enabled.
+The uncompress method used is derived from `buffer-file-name'."
+  (when (and (fboundp 'jka-compr-installed-p)
+             (jka-compr-installed-p))
+    (let ((info (jka-compr-get-compression-info buffer-file-name)))
+      (when info
+        (let ((basename (file-name-nondirectory buffer-file-name))
+              (args     (jka-compr-info-uncompress-args    info))
+              (prog     (jka-compr-info-uncompress-program info))
+              (message  (jka-compr-info-uncompress-message info))
+              (err-file (jka-compr-make-temp-name)))
+          (if message
+              (message "%s %s..." message basename))
+          (unwind-protect
+              (unless (memq (apply 'call-process-region
+                                   (point-min) (point-max) 
+                                   prog
+                                   t (list t err-file) nil
+                                   args)
+                            jka-compr-acceptable-retval-list)
+                (jka-compr-error prog args basename message err-file))
+            (jka-compr-delete-temp-file err-file)))))))
+
 (defun gnus-mime-copy-part (&optional handle)
   "Put the MIME part under point into a new buffer."
   (interactive)
@@ -3928,6 +3958,7 @@ (defun gnus-mime-copy-part (&optional ha
       (unwind-protect
 	  (progn
 	    (setq buffer-file-name (expand-file-name base))
+	    (gnus-mime-jka-compr-maybe-uncompress)
 	    (normal-mode))
 	(setq buffer-file-name nil))
       (goto-char (point-min)))))

Re: gnus-mime-copy-part and jka-compr

[Lars Magne Ingebrigtsen]

Kevin Ryde <user42@zip.com.au> writes:

> --- gnus-art.el.~6.265.~	2003-01-04 10:22:58.000000000 +1000
> +++ gnus-art.el	2003-01-04 10:32:28.000000000 +1000

Thanks for the patch; I've applied it to Oort Gnus v0.08 (i. e., CVS).

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: gnus-mime-copy-part and jka-compr

[Kevin Ryde]

[-- Attachment #1: Type: text/plain, Size: 300 bytes --]

Some doco for this change, mainly to get a cross reference to
auto-compression-mode so one knows how to enable the decompression.

	* gnus-art.el (gnus-mime-copy-part): Mention auto-compression-mode in
	docstring.

	* gnus.texi (Using MIME): Mention auto-compression-mode with
	gnus-mime-copy-part.


[-- Attachment #2: gnus-art.el.copy-part-doc.diff --]
[-- Type: text/plain, Size: 602 bytes --]

--- gnus-art.el.~6.274.~	2003-01-15 10:17:40.000000000 +1000
+++ gnus-art.el	2003-01-15 10:24:27.000000000 +1000
@@ -3986,7 +3986,9 @@ (defun gnus-mime-jka-compr-maybe-uncompr
             (jka-compr-delete-temp-file err-file)))))))
 
 (defun gnus-mime-copy-part (&optional handle)
-  "Put the MIME part under point into a new buffer."
+  "Put the MIME part under point into a new buffer.
+If `auto-compression-mode' is enabled, compressed files like .gz and .bz2
+are decompressed."
   (interactive)
   (gnus-article-check-buffer)
   (let* ((handle (or handle (get-text-property (point) 'gnus-data)))

[-- Attachment #3: gnus.texi.copy-part.diff --]
[-- Type: text/plain, Size: 565 bytes --]

--- gnus.texi.~6.388.~	2003-01-15 10:14:25.000000000 +1000
+++ gnus.texi	2003-01-15 10:28:30.000000000 +1000
@@ -10563,7 +10563,10 @@
 @item c (Article)
 @kindex c (Article)
 Copy the @sc{mime} object to a fresh buffer and display this buffer
-(@code{gnus-mime-copy-part}).
+(@code{gnus-mime-copy-part}).  Compressed files like @file{.gz} and
+@file{.bz2} are automatically decompressed if
+@code{auto-compression-mode} is enabled (@pxref{Compressed Files,,
+Accessing Compressed Files, emacs, The Emacs Editor}).
 
 @findex gnus-mime-print-part
 @item p (Article)

New IMAP user with some Q's

[dave-mlist]

I have appended my .gnus below.
I want to do IMAP splitting, but I'm running into the following
problems:

1.  it is really really slow!
2.  splitting is working, I've confirmed this on the server, but I
    don't see new groups in *Group* buffer.

More details about #2:

Here's what I see on my courier-IMAP server:

  drwx------    2 courier  mail         4096 Jan 15 02:31 tmp/
  -rw-r--r--    1 courier  courier       287 Jan 15 01:57 courierimapuiddb
  drwx------    2 courier  mail         4096 Jan 15 01:57 cur/
  drwx------    5 courier  courier      4096 Jan 15 01:57 .test.25/
  drwx------    5 courier  courier      4096 Jan 15 01:57 .private/
  drwx------    2 courier  mail         4096 Jan 15 01:56 new/
  drwx------    5 courier  courier      4096 Jan 15 01:56 .Trash/
  -rw-r--r--    1 courier  courier        18 Jan  6 01:03 courierimapsubscribed

That's the splitting I expected: The top level is a Maildir
representing INBOX.  ".test.25" is a Maildir representing
"INBOX.test.25", Et cetera.

But I see a totally blank *Group* buffer.  Even when I type the "^"
key to go into server mode, when I browse to my IMAP server session, I
only see the "INBOX" and the "INBOX.Trash" folders.

Any suggestions?  My .gnus is appended

Thanks,
Dave


(setq user-mail-address "nospam@nospam.com")
(setq user-full-name "No Spam Sam")

(setq gnus-verbose 9)

(setq imap-debug "*imap-debug*")
(setq nnimap-debug "*nnimap-debug*")
(setq nnimap-log "*imap-log*")

(setq gnus-directory "~/mail/Mail/")
(setq gnus-kill-files-directory "~/mail/Mail/")

(setq 
 gnus-use-cache t
 gnus-cacheable-groups ".*" 
 gnus-uncacheable-groups "^nnmbox" 
 gnus-cache-enter-articles '(ticked dormant unread)
)

(setq gnus-select-method '(nnml "private"))

(setq nnimap-split-crosspost t)
(setq nnimap-split-inbox '("INBOX" "INBOX.Trash"))

(setq nnimap-split-rule
                '(("INBOX.test" "^Sender: spammer@spamhouse.com")
                  ("INBOX.test.\\1"   "^Subject: test \\([0-9]+\\)")
                  ("INBOX.junk"       "^Subject: junk")
                  ("INBOX.private" "")))

(setq gnus-secondary-select-methods '((nnimap "test@example.com"
					      (nnimap-authinfo-file "~/path/to/some/file/it/doesnt/seem/to/use")
					      (nnimap-address "mymx.example.com")
					      (nnimap-server-port 993)
					      (nnimap-stream ssl)
					      (nnimap-list-pattern ("*"))
					      (nnimap-nov-is-evil t)
					      (nnmap-expunge-on-close 'ask))))

(setq nnml-directory "~/mail/Mail")

(setq gnus-default-subscribed-newsgroups t)
(setq gnus-check-new-newsgroups 'always) ; queries back ends for new groups even when you do the g command
(setq gnus-subscribe-newsgroup-method 'gnus-subscribe-interactively)
(setq gnus-save-newsrc-file nil)
(setq gnus-startup-file "~/mail/.newsrc") ; dribble files and .eld files go in the same directory as .newsrc

Re: New IMAP user with some Q's

[Kai Großjohann]

dave-mlist@bfnet.com writes:

> That's the splitting I expected: The top level is a Maildir
> representing INBOX.  ".test.25" is a Maildir representing
> "INBOX.test.25", Et cetera.

Maybe Gnus gets confused by group names that are numbers.  (Or where
components are numbers.)
-- 
Ambibibentists unite!

Re: New IMAP user with some Q's

[Simon Josefsson]

dave-mlist@bfnet.com writes:

> I have appended my .gnus below.
> I want to do IMAP splitting, but I'm running into the following
> problems:
>
> 1.  it is really really slow!

Could you profile gnus or (probably more useful) look at tcpdump what
is taking so long?  Maybe it is the server that is slow.  My guess is
that it is the subscribed-groups operation that takes time.  (setq
gnus-check-new-newsgroups nil) to disable it.

> 2.  splitting is working, I've confirmed this on the server, but I
> don't see new groups in *Group* buffer.
...
> (setq gnus-check-new-newsgroups 'always) ; queries back ends for new groups even when you do the g command

The nnimap splitter doesn't imap-subscribe auto-created groups, you
need to do that manually.  g-c-n-n only gnus-subscribe imap-subscribed
mailboxes, which is slow and only useful if you create groups using
other IMAP clients often, and want Gnus to notice them.

Re: gnus-mime-copy-part and jka-compr

[Lars Magne Ingebrigtsen]

Kevin Ryde <user42@zip.com.au> writes:

> 	* gnus-art.el (gnus-mime-copy-part): Mention auto-compression-mode in
> 	docstring.
>
> 	* gnus.texi (Using MIME): Mention auto-compression-mode with
> 	gnus-mime-copy-part.

Thanks for the patch; I've applied it to Oort Gnus v0.13 (i. e., CVS).

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen