gnus-article-unsplit-urls

gnus-article-unsplit-urls

[Michael Cook]

here's an idea for a `W' command maybe someone would be inclined to
implement.

for certain mail user agents (like ms outlook), if the user types a
long url into a message they are composing, as the agent sends the
message, the agent will graciously insert a newline into the url
rendering the url unclickable by the recipients.  i'm sure you've
all seen messages containing text like this:

bla bla bla at
http://foo.bar.com/baz/some/really/long/path/la/dee/da/dum/dee/ind
ex.html and bla bla bla.

(noting the newline between "ind" and "ex" in this case).

i think this url splitting would be reasonably easy to detect
automatically reliably because the agents seem to leave the
beginning of the url alone on a line (with no leading or trailing
whitespace), and the end of the url is at the beginning of the next
line (with no leading whitespace).  in other words, the W command
might be basically this:

 s%^((https?|ftp)://\S+)\n(\S+)%$1$3%g;

any takers?

m.

Re: gnus-article-unsplit-urls

[Simon Josefsson]

On Fri, 4 Jan 2002, Michael Cook wrote:

> bla bla bla at
> http://foo.bar.com/baz/some/really/long/path/la/dee/da/dum/dee/ind
> ex.html and bla bla bla.

Just insert a \n at the proper place in the URL regexp.  Unfortunately,
you need to re-evaluate gnus-button-alist because it inlines
gnu-button-url-regexp.  The patch below should fix both things.  (Why are
these variables internal?)

--- gnus-art.el.~6.149.~	Wed Jan  2 10:05:38 2002
+++ gnus-art.el	Fri Jan  4 16:30:36 2002
@@ -4996,13 +4996,13 @@
 
 ;;; Internal Variables:
 
-(defcustom gnus-button-url-regexp 
"\\b\\(\\(www\\.\\|\\(s?https?\\|ftp\\|file\\|gopher\\|news\\|telnet\\|wais\\|mailto\\|info\\):\\)\\(//[-a-zA-Z0-9_.]+:[0-9]*\\)?\\([-a-zA-Z0-9_=!?#$@~`%&*+|\\/:;.,]\\|\\w\\)+\\([-a-zA-Z0-9_=#$@~`%&*+|\\/]\\|\\w\\)\\)"
+(defcustom gnus-button-url-regexp 
"\\b\\(\\(www\\.\\|\\(s?https?\\|ftp\\|file\\|gopher\\|news\\|telnet\\|wais\\|mailto\\|info\\):\\)\\(//[-a-zA-Z0-9_.]+:[0-9]*\\)?\\([-a-zA-Z\n0-9_=!?#$@~`%&*+|\\/:;.,]\\|\\w\\)+\\([-a-zA-Z0-9_=#$@~`%&*+|\\/]\\|\\w\\)\\)"
   "Regular expression that matches URLs."
   :group 'gnus-article-buttons
   :type 'regexp)
 
 (defcustom gnus-button-alist
-  `(("<\\(url:[>\n\t ]*?\\)?\\(nntp\\|news\\):[>\n\t ]*\\([^>\n\t 
]*@[^>\n\t ]*\\)>"
+  '(("<\\(url:[>\n\t ]*?\\)?\\(nntp\\|news\\):[>\n\t ]*\\([^>\n\t 
]*@[^>\n\t ]*\\)>"
      0 t gnus-button-handle-news 3)
     ("\\b\\(nntp\\|news\\):\\([^>\n\t ]*@[^>)!;:,\n\t ]*\\)" 0 t
      gnus-button-handle-news 2)
@@ -5021,7 +5021,7 @@
     ;; This is how URLs _should_ be embedded in text...
     ("<URL: *\\([^<>]*\\)>" 0 t gnus-button-embedded-url 1)
     ;; Raw URLs.
-    (,gnus-button-url-regexp 0 t browse-url 0))
+    (gnus-button-url-regexp 0 t browse-url 0))
   "*Alist of regexps matching buttons in article bodies.
 
 Each entry has the form (REGEXP BUTTON FORM CALLBACK PAR...), where
@@ -5035,7 +5035,7 @@
 CALLBACK can also be a variable, in that case the value of that
 variable it the real callback function."
   :group 'gnus-article-buttons
-  :type '(repeat (list regexp
+  :type '(repeat (list (choice regexp variable)
 		       (integer :tag "Button")
 		       (sexp :tag "Form")
 		       (function :tag "Callback")
@@ -5221,7 +5221,7 @@
       (article-goto-body)
       (setq beg (point))
       (while (setq entry (pop alist))
-	(setq regexp (car entry))
+	(setq regexp (eval (car entry)))
 	(goto-char beg)
 	(while (re-search-forward regexp nil t)
 	  (let* ((start (and entry (match-beginning (nth 1 entry))))
@@ -5326,7 +5326,7 @@
 	(entry nil))
     (while alist
       (setq entry (pop alist))
-      (if (looking-at (car entry))
+      (if (looking-at (eval (car entry)))
 	  (setq alist nil)
 	(setq entry nil)))
     entry))

Re: gnus-article-unsplit-urls

[Michael Cook]

Simon Josefsson <jas@extundo.com> writes:

> Just insert a \n at the proper place in the URL regexp.

the newlines would need to be discard, not just matched, right?

m.

Re: gnus-article-unsplit-urls

[Simon Josefsson]

On Fri, 4 Jan 2002, Michael Cook wrote:

> Simon Josefsson <jas@extundo.com> writes:
> 
> > Just insert a \n at the proper place in the URL regexp.
> 
> the newlines would need to be discard, not just matched, right?

Ah, you are right.  Hmm.  Is it possible to write a regexp to match the
complete URL but not include the \n in the \\(...\\) part?  I don't think
so.  Maybe the code should simply strip \n's from the matched data?

Re: gnus-article-unsplit-urls

[Lars Magne Ingebrigtsen]

Simon Josefsson <jas@extundo.com> writes:

> Ah, you are right.  Hmm.  Is it possible to write a regexp to match the
> complete URL but not include the \n in the \\(...\\) part?  I don't think
> so.  Maybe the code should simply strip \n's from the matched data?

Sure.  `gnus-button-embedded-url' already does that, so the other URL
buttons could do so, too.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: gnus-article-unsplit-urls

[Simon Josefsson]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Ah, you are right.  Hmm.  Is it possible to write a regexp to match the
>> complete URL but not include the \n in the \\(...\\) part?  I don't think
>> so.  Maybe the code should simply strip \n's from the matched data?
>
> Sure.  `gnus-button-embedded-url' already does that, so the other URL
> buttons could do so, too.

So <URL:http://josefsson.org/foo bar.html> doesn't work?  Hm.

Re: gnus-article-unsplit-urls

[Lars Magne Ingebrigtsen]

Simon Josefsson <jas@extundo.com> writes:

>> Sure.  `gnus-button-embedded-url' already does that, so the other URL
>> buttons could do so, too.
>
> So <URL:http://josefsson.org/foo bar.html> doesn't work?  Hm.

No, it does work.  It'll fetch "http://josefsson.org/foobar.html" just
fine.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: gnus-article-unsplit-urls

[Simon Josefsson]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>>> Sure.  `gnus-button-embedded-url' already does that, so the other URL
>>> buttons could do so, too.
>>
>> So <URL:http://josefsson.org/foo bar.html> doesn't work?  Hm.
>
> No, it does work.  It'll fetch "http://josefsson.org/foobar.html" just
> fine.  :-)

Ah, I know see that is how RFC 1738 intended things to work.  Oki.

Re: gnus-article-unsplit-urls

[Simon Josefsson]

On Fri, 4 Jan 2002, Simon Josefsson wrote:

> -(defcustom gnus-button-url-regexp 
> "\\b\\(\\(www\\.\\|\\(s?https?\\|ftp\\|file\\|gopher\\|news\\|telnet\\|wais\\|mailto\\|info\\):\\)\\(//[-a-zA-Z0-9_.]+:[0-9]*\\)?\\([-a-zA-Z0-9_=!?#$@~`%&*+|\\/:;.,]\\|\\w\\)+\\([-a-zA-Z0-9_=#$@~`%&*+|\\/]\\|\\w\\)\\)"
> +(defcustom gnus-button-url-regexp 
> "\\b\\(\\(www\\.\\|\\(s?https?\\|ftp\\|file\\|gopher\\|news\\|telnet\\|wais\\|mailto\\|info\\):\\)\\(//[-a-zA-Z0-9_.]+:[0-9]*\\)?\\([-a-zA-Z\n0-9_=!?#$@~`%&*+|\\/:;.,]\\|\\w\\)+\\([-a-zA-Z0-9_=#$@~`%&*+|\\/]\\|\\w\\)\\)"

Hm, that regexp wasn't very good, maybe this one is better:

(defcustom gnus-button-url-regexp 
"\\b\\(\\(www\\.\\|\\(s?https?\\|ftp\\|file\\|gopher\\|news\\|telnet\\|wais\\|mailto\\|info\\):\\)\\(//[-a-zA-Z0-9_.]+:[0-9]*\\)?\\(\\([-a-zA-Z0-9_=!?#$@~`%&*+|\\/:;.,]\\|\\w\\)\n?\\([-a-zA-Z0-9_=!?#$@~`%&*+|\\/:;.,]\\|\\w\\)?\\)+\\([-a-zA-Z0-9_=#$@~`%&*+|\\/]\\|\\w\\)\\)"
  "Regular expression that matches URLs."
  :group 'gnus-article-buttons
  :type 'regexp)

Re: gnus-article-unsplit-urls

[Lars Magne Ingebrigtsen]

Simon Josefsson <jas@extundo.com> writes:

> Hm, that regexp wasn't very good, maybe this one is better:

But won't that match stuff like

http://google.com/thing
and then there's something else on the next line?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: gnus-article-unsplit-urls

[Michael Cook]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> But won't that match stuff like
>
> http://google.com/thing
> and then there's something else on the next line?

that's why i thought a washing function would be better.  the
detection of the problem is not perfect.  so you'd probably not want
it to happen implicitly.  i figure more likely you'd see a broken
url, you'd mumble something under your breath about microsoft, you'd
type `W u', and then go about your business.

m.

Re: gnus-article-unsplit-urls

[Lars Magne Ingebrigtsen]

Michael Cook <michael.cook@cisco.com> writes:

> that's why i thought a washing function would be better.  the
> detection of the problem is not perfect.  so you'd probably not want
> it to happen implicitly.  i figure more likely you'd see a broken
> url, you'd mumble something under your breath about microsoft, you'd
> type `W u', and then go about your business.

Yes, I agree.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: gnus-article-unsplit-urls

[Simon Josefsson]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Simon Josefsson <jas@extundo.com> writes:
>
>> Hm, that regexp wasn't very good, maybe this one is better:
>
> But won't that match stuff like
>
> http://google.com/thing
> and then there's something else on the next line?

Yes.  Hm.  A good heuristic might be to match

http://foo
bar

but not

http://foo
bar baz

That is, do not assume the URL is continued if there is more than one
word on the second line.  Perhaps?

Updating the regexp to match this is left as an exercise for the
wickedly perverted...

Re: gnus-article-unsplit-urls

[Michael Cook]

Simon Josefsson <jas@extundo.com> writes:

> http://foo
> bar baz
>
> That is, do not assume the URL is continued if there is more than one
> word on the second line.  Perhaps?

i think it's common for long urls to be within a paragraph (that was
the case in the message that got me thinking about this issue).  and
so it's common for baz to follow bar on the same line like that.

m.

Re: gnus-article-unsplit-urls

[Michael Cook]

[-- Attachment #1: Type: text/plain, Size: 327 bytes --]

Michael Cook <michael.cook@cisco.com> writes:

> for certain mail user agents (like ms outlook), if the user types a
> long url into a message they are composing, as the agent sends the
> message, the agent will graciously insert a newline into the url
> rendering the url unclickable by the recipients.

ok, how's this patch?

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: unsplit-urls.patch --]
[-- Type: text/x-patch, Size: 4263 bytes --]

--- lisp/gnus-art.el.DIST	Tue Feb  5 10:58:24 2002
+++ lisp/gnus-art.el	Fri Feb  8 09:06:45 2002
@@ -858,6 +858,13 @@
   :group 'gnus-article-treat
   :type gnus-article-treat-custom)
 
+(defcustom gnus-treat-unsplit-urls nil
+  "Remove newlines from within URLs.
+Valid values are nil, t, `head', `last', an integer or a predicate.
+See Info node `(gnus)Customizing Articles' for details."
+  :group 'gnus-article-treat
+  :type gnus-article-treat-custom)
+
 (defcustom gnus-treat-leading-whitespace nil
   "Remove leading whitespace in headers.
 Valid values are nil, t, `head', `last', an integer or a predicate.
@@ -1223,6 +1230,7 @@
     (gnus-treat-fill-article gnus-article-fill-cited-article)
     (gnus-treat-fill-long-lines gnus-article-fill-long-lines)
     (gnus-treat-strip-cr gnus-article-remove-cr)
+    (gnus-treat-unsplit-urls gnus-article-unsplit-urls)
     (gnus-treat-date-ut gnus-article-date-ut)
     (gnus-treat-date-local gnus-article-date-local)
     (gnus-treat-date-english gnus-article-date-english)
@@ -2048,6 +2056,16 @@
     (let ((buffer-read-only nil))
       (rfc1843-decode-region (point-min) (point-max)))))
 
+(defun article-unsplit-urls ()
+  "Remove the newlines that some other mailers insert into URLs."
+  (interactive)
+  (save-excursion
+    (let ((buffer-read-only nil))
+      (goto-char (point-min))
+      (while (re-search-forward
+	      "^\\(\\(https?\\|ftp\\)://\\S-+\\)\n\\(\\S-+\\)" nil t)
+	(replace-match "\\1\\3" t)))))
+
 (defun article-wash-html (&optional read-charset)
   "Format an html article.
 If READ-CHARSET, ask for a coding system."
@@ -3172,6 +3190,7 @@
      article-de-base64-unreadable
      article-decode-HZ
      article-wash-html
+     article-unsplit-urls
      article-hide-list-identifiers
      article-hide-pgp
      article-strip-banner
@@ -3268,6 +3287,7 @@
        ["Remove quoted-unreadable" gnus-article-de-quoted-unreadable t]
        ["Remove base64" gnus-article-de-base64-unreadable t]
        ["Treat html" gnus-article-wash-html t]
+       ["Remove newlines from within URLs" gnus-article-unsplit-urls t]
        ["Decode HZ" gnus-article-decode-HZ t]))
 
     ;; Note "Commands" menu is defined in gnus-sum.el for consistency
--- lisp/gnus.el.DIST	Sat Feb  2 08:54:16 2002
+++ lisp/gnus.el	Fri Feb  8 09:06:45 2002
@@ -2110,6 +2110,7 @@
       gnus-article-de-base64-unreadable
       gnus-article-decode-HZ
       gnus-article-wash-html
+      gnus-article-unsplit-urls
       gnus-article-hide-pgp
       gnus-article-hide-pem gnus-article-hide-signature
       gnus-article-strip-leading-blank-lines gnus-article-date-local
--- lisp/gnus-sum.el.DIST	Tue Feb  5 10:58:24 2002
+++ lisp/gnus-sum.el	Fri Feb  8 09:06:45 2002
@@ -1729,6 +1729,7 @@
     "6" gnus-article-de-base64-unreadable
     "Z" gnus-article-decode-HZ
     "h" gnus-article-wash-html
+    "u" gnus-article-unsplit-urls
     "s" gnus-summary-force-verify-and-decrypt
     "f" gnus-article-display-x-face
     "l" gnus-summary-stop-page-breaking
@@ -1953,6 +1954,7 @@
 	      ["Unfold headers" gnus-article-treat-unfold-headers t]
 	      ["Fold newsgroups" gnus-article-treat-fold-newsgroups t]
 	      ["Html" gnus-article-wash-html t]
+	      ["URLs" gnus-article-unsplit-urls t]
 	      ["Verify X-PGP-Sig" gnus-article-verify-x-pgp-sig t]
 	      ["HZ" gnus-article-decode-HZ t])
 	     ("Output"
--- texi/gnus.texi.DIST	Tue Feb  5 10:58:24 2002
+++ texi/gnus.texi	Fri Feb  8 09:14:47 2002
@@ -8125,6 +8125,14 @@
 common encoding employed when sending Chinese articles.  It typically
 makes strings look like @samp{~@{<:Ky2;S@{#,NpJ)l6HK!#~@}}.
 
+@item W u
+@kindex W u (Summary)
+@findex gnus-article-unsplit-urls
+Remove newlines from within URLs.  Some mailers insert newlines into
+outgoing email messages to keep lines short.  This reformatting can
+split long URLs onto multiple lines.  Repair those URLs by removing
+the newlines (@code{gnus-article-unsplit-urls}).
+
 @item W h
 @kindex W h (Summary)
 @findex gnus-article-wash-html
@@ -10348,6 +10356,7 @@
 @item gnus-treat-strip-pem (t, last, integer)
 @item gnus-treat-strip-pgp (t, last, integer)
 @item gnus-treat-strip-trailing-blank-lines (t, last, integer)
+@item gnus-treat-unsplit-urls (t, integer)
 
 @xref{Article Washing}.
 

Re: gnus-article-unsplit-urls

[ShengHuo ZHU]

Michael Cook <michael.cook@cisco.com> writes:

> Michael Cook <michael.cook@cisco.com> writes:
>
>> for certain mail user agents (like ms outlook), if the user types a
>> long url into a message they are composing, as the agent sends the
>> message, the agent will graciously insert a newline into the url
>> rendering the url unclickable by the recipients.
>
> ok, how's this patch?

Looks good.  Have you signed the FSF paper? I'll send the request
form.

ShengHuo