Using `links' to process HTML

Using `links' to process HTML

[Hrvoje Niksic]

[ Note: I'm not on the list.  If you want me to see your replies, add Cc. ]

I've been using this for some time now and want to share it with
others.  It hooks into the processing of "text/html" and renders it by
noninteractively calling the textual browser `links'.  Although this
doesn't support images, colors, forms and whatnot, those are the
things I turn off anyway.  It does support tables, however.

Overall it does a much better job of rendering HTML than w3 does.  My
spams finally come out right before I ignore them.

Here is what I put in `.gnus'.  It relies on the program `links' being
somewhere on the PATH.  If you try it, please let me know if it works
for you.

(eval-after-load "mm-decode"
  '(push '("text/html"
	   my-html-handler
	   (lambda (handle) t))
	 mm-inline-media-tests))

;; If you're not running under XEmacs...
(unless (fboundp 'temp-directory)
  (defun temp-directory ()
    (or (getenv "TMPDIR") "/tmp")))

(defun my-html-handler (handle)
  (let ((source (mm-get-part handle))
	(tmpfile (make-temp-name (expand-file-name
				  "www" (temp-directory))))
	text)
    (with-temp-buffer
      (insert source)
      (write-region (point-min) (point-max) tmpfile nil 'silent)
      (delete-region (point-min) (point-max))
      (unwind-protect
	  ;; It would be nice if we could use `call-process-region' to
	  ;; feed the HTML to links's stdin thus avoiding the tmpfile.
	  ;; But `links -dump /dev/stdin' doesn't work when stdin is a
	  ;; pipe.
	  (call-process "links" nil t nil "-dump" tmpfile)
	(delete-file tmpfile))

      ;; Delete the annoying three spaces preceding each line of links
      ;; output.
      (goto-char (point-min))
      (while (re-search-forward "^   " nil t)
	(delete-region (match-beginning 0) (match-end 0)))

      (setq text (buffer-string)))

    (mm-insert-inline handle text)))

Re: Using `links' to process HTML

[Nevin Kapur]

Hrvoje Niksic <hniksic@arsdigita.com> writes:

> [ Note: I'm not on the list.  If you want me to see your replies, add Cc. ]
>
> I've been using this for some time now and want to share it with
> others.  It hooks into the processing of "text/html" and renders it by
> noninteractively calling the textual browser `links'.  Although this
> doesn't support images, colors, forms and whatnot, those are the
> things I turn off anyway.  It does support tables, however.
>
> Overall it does a much better job of rendering HTML than w3 does.  My
> spams finally come out right before I ignore them.
>
> Here is what I put in `.gnus'.  It relies on the program `links' being
> somewhere on the PATH.  If you try it, please let me know if it works
> for you.

I've been using this since you posted it yesterday.  Seems to work
really well, doing exactly what I want for HTML only mails.

Thanks!

-- 
Nevin

Re: Using `links' to process HTML

[Matthieu Moy]

Hrvoje Niksic <hniksic@arsdigita.com> writes:

> others.  It hooks into the processing of "text/html" and renders it by
> noninteractively calling the textual browser `links'.  Although this
> doesn't support images, colors, forms and whatnot, those are the
> things I turn off anyway.  It does support tables, however.

I have  something similar, which I  find much simpler  : The following
line in my .mailcap : 

text/html; lynx.sh %s; copiousoutput

Where lynx.sh is :

#!/bin/tcsh
set term=rxvt
lynx -dump -force_html "$1" -term=rxvt

What is the difference (except the use of lynx instead of links) ? 

-- 
Matthieu

Re: Using `links' to process HTML

[Hrvoje Niksic]

Matthieu Moy <Matthieu.Moy@imag.fr> writes:

> I have  something similar, which I  find much simpler  : The following
> line in my .mailcap : 
>
> text/html; lynx.sh %s; copiousoutput
>
> Where lynx.sh is :
>
> #!/bin/tcsh
> set term=rxvt
> lynx -dump -force_html "$1" -term=rxvt
>
> What is the difference (except the use of lynx instead of links) ?

Two differences:

* I never bothered to read up on what you can do with `.mailcap'.  I
  preferred hacking Lisp.  :-)

* Links is so much cooler than lynx.  For one, it handles HTML tables
  correctly.

Re: Using `links' to process HTML

[Steinar Bang]

>>>>> Hrvoje Niksic <hniksic@arsdigita.com>:

> Matthieu Moy <Matthieu.Moy@imag.fr> writes:

>> What is the difference (except the use of lynx instead of links) ?

> * Links is so much cooler than lynx.  For one, it handles HTML
> tables correctly.

Links (0.96.20011222 on debian testing) doesn't seem to do HTTP Basic
auth, though.

Re: Using `links' to process HTML

[Nevin Kapur]

Matthieu Moy <Matthieu.Moy@imag.fr> writes:

> I have  something similar, which I  find much simpler  : The following
> line in my .mailcap : 
>
> text/html; lynx.sh %s; copiousoutput

Doesn't that make it a global setting?  I, for one, am only interested
in l[iy]nks handle text/html when I'm reading mail from within Gnus.

-- 
Nevin

Re: Using `links' to process HTML

[Kai Großjohann]

Nevin Kapur <nevin@jhu.edu> writes:

> Doesn't that make it a global setting?  I, for one, am only interested
> in l[iy]nks handle text/html when I'm reading mail from within Gnus.

I think that .mailcap supports tests.  But I'm not sure if there is a
test which evals to true for Gnus and false otherwise.

To be sure, you can add the entry in mailcap-mime-data directly.

IIRC, there is something similar for w3m but with that solution it is
even possible to follow links!

kai
-- 
~/.signature is: umop 3p!sdn    (Frank Nobis)

Re: Using `links' to process HTML

[Nuutti Kotivuori]

Kai Großjohann wrote:
> Nevin Kapur <nevin@jhu.edu> writes:
> 
>> Doesn't that make it a global setting?  I, for one, am only
>> interested in l[iy]nks handle text/html when I'm reading mail from
>> within Gnus.
> 
> I think that .mailcap supports tests.  But I'm not sure if there is
> a test which evals to true for Gnus and false otherwise.
> 
> To be sure, you can add the entry in mailcap-mime-data directly.
> 
> IIRC, there is something similar for w3m but with that solution it
> is even possible to follow links!

I'm using w3m-el with some hacks copied somewhere to integrate it with
gnus. All HTML mails are rendered inline, with links and tables and
the ability to browse them. Renders pictures inline as well, if I ask
it to (I don't, usually).

-- Naked

Re: Using `links' to process HTML

[Paul Jarc]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) wrote:
> I think that .mailcap supports tests.  But I'm not sure if there is a
> test which evals to true for Gnus and false otherwise.

You could add (setenv "CALLER" "Gnus") in .gnus, and the shell command
in .mailcap could do different things for different $CALLERs.


paul

Re: Using `links' to process HTML

[Paul Jarc]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) wrote:
> I think that .mailcap supports tests.  But I'm not sure if there is a
> test which evals to true for Gnus and false otherwise.

You could add (setenv "CALLER" "Gnus") in .gnus, and the shell command
in .mailcap could do different things for different $CALLERs.


paul

Re: Using `links' to process HTML

[ShengHuo ZHU]

Hrvoje Niksic <hniksic@arsdigita.com> writes:

> [ Note: I'm not on the list.  If you want me to see your replies, add Cc. ]
>
> I've been using this for some time now and want to share it with
> others.  It hooks into the processing of "text/html" and renders it by
> noninteractively calling the textual browser `links'.  Although this
> doesn't support images, colors, forms and whatnot, those are the
> things I turn off anyway.  It does support tables, however.
>
> Overall it does a much better job of rendering HTML than w3 does.  My
> spams finally come out right before I ignore them.
>
> Here is what I put in `.gnus'.  It relies on the program `links' being
> somewhere on the PATH.  If you try it, please let me know if it works
> for you.

Inspired by the code, I added a new feature which enables users to
choose w3, w3m, links or lynx to render HTML parts. For example,

     (setq mm-text-html-renderer 'links)

(I don't have links installed. Please test.)

This variable can be overridden if mm-inline-text-html-renderer is
set. Also, gnus-article-wash-html uses this variable too.

If anyone knows some external commands to render html or other MIME
type, please post the solutions. Something like html2txt or word2txt.

ShengHuo

Re: Using `links' to process HTML

[Matthieu Moy]

ShengHuo ZHU <zsh@cs.rochester.edu> writes:

> If anyone knows some external commands to render html or other MIME
> type, please post the solutions. Something like html2txt or word2txt.

There is antiword or catdoc for  M$ word .doc files. Very usefull when
people  use word  for typing  plain  text, and  then send  it by  mail
(Aarrgh !).  Its  behaviour is similar to lynx -dump.  See if you want
to add something  about this. But I think  .mailcap and its equivalent
in  elisp  are sufficient.  Maybe  add some  tips  about  this in  the
documentation.

,----[ ~/.mailcap ]
| application/msword; catdoc %s ; copiousoutput ; nametemplate=%s.doc
`----

-- 
Matthieu

Re: Using `links' to process HTML

[ShengHuo ZHU]

Matthieu Moy <Matthieu.Moy@imag.fr> writes:

> ShengHuo ZHU <zsh@cs.rochester.edu> writes:
>
>> If anyone knows some external commands to render html or other MIME
>> type, please post the solutions. Something like html2txt or word2txt.
>
> There is antiword or catdoc for  M$ word .doc files. Very usefull when
> people  use word  for typing  plain  text, and  then send  it by  mail
> (Aarrgh !).  Its  behaviour is similar to lynx -dump.  See if you want
> to add something  about this. But I think  .mailcap and its equivalent
> in  elisp  are sufficient.  Maybe  add some  tips  about  this in  the
> documentation.
>
> ,----[ ~/.mailcap ]
> | application/msword; catdoc %s ; copiousoutput ; nametemplate=%s.doc
> `----

You are right. I've added copiousoutput in the documentation.

ShengHuo

Re: Using `links' to process HTML

[Steve Youngs]

|--==> "ZSH" == ShengHuo ZHU <zsh@cs.rochester.edu> writes:

  ZSH>      (setq mm-text-html-renderer 'links)

With (setq mm-text-html-renderer 'lynx) I'm getting an error back from
lynx:  "Unknown option '-stdin'".



-- 
|---<Steve Youngs>---------------<GnuPG KeyID: 10D5C9C5>---|
|            XEmacs - It's not just an editor.             |
|                    It's a way of life.                   |
|------------------------------------<youngs@xemacs.org>---|

Re: Using `links' to process HTML

[ShengHuo ZHU]

Steve Youngs <youngs@xemacs.org> writes:

> |--==> "ZSH" == ShengHuo ZHU <zsh@cs.rochester.edu> writes:
>
>   ZSH>      (setq mm-text-html-renderer 'links)
>
> With (setq mm-text-html-renderer 'lynx) I'm getting an error back from
> lynx:  "Unknown option '-stdin'".

I got the option. Maybe it is because of the different versions of
lynx.

$ lynx -version
Lynx Version 2.8.4rel.1 (17 Jul 2001)
libwww-FM 2.14, SSL-MM 1.4.1, OpenSSL 0.9.6b
Built on linux-gnu Jul 31 2001 09:03:10


ShengHuo

Re: Using `links' to process HTML

[Steve Youngs]

|--==> "ZSH" == ShengHuo ZHU <zsh@cs.rochester.edu> writes:

  ZSH> Steve Youngs <youngs@xemacs.org> writes:
  >>|--==> "ZSH" == ShengHuo ZHU <zsh@cs.rochester.edu> writes:
  >>
  ZSH> (setq mm-text-html-renderer 'links)
  >>
  >>With (setq mm-text-html-renderer 'lynx) I'm getting an error back from
  >>lynx:  "Unknown option '-stdin'".

  ZSH> I got the option. Maybe it is because of the different versions of
  ZSH> lynx.

That was it.  Updating lynx fixed it.

-- 
|---<Steve Youngs>---------------<GnuPG KeyID: 10D5C9C5>---|
|            XEmacs - It's not just an editor.             |
|                    It's a way of life.                   |
|------------------------------------<youngs@xemacs.org>---|