Fixing Gmane's mangled MIDs (was: Stop messing with Message-IDs, you're ruining threading)

Fixing Gmane's mangled MIDs (was: Stop messing with Message-IDs, you're ruining threading)

[Reiner Steib]

[-- Attachment #1: Type: text/plain, Size: 1579 bytes --]

On Sun, Dec 07 2008, Sven Joachim wrote:

> On 2008-12-07 06:50 +0100, Lars Magne Ingebrigtsen wrote:
>> When the same message comes from two mailing lists, and there's not
>> enough information in them to identify them as such (to crosspost the
>> message between the list), the Message-ID has to be renamed.  Remove the
>> bit after __ to find the original ID.
>
> Here is some Emacs lisp code that does this automatically when replying
> in Gnus:
>
> (defun sj-unmangle-gmane-message-id ()
>   "Undo the Message-ID mangling that Gmane does on some postings."
>   (save-excursion
>     (save-restriction
>       (message-narrow-to-headers)
>       (goto-char (point-min))
>       (while
> 	  (re-search-forward "\\(<.*\\)" (point-max) t)
> 	(replace-match "\\1")))))

I would have expected a search for an expression like
"__[0-9.]+\\$gmane\\$org@" or similar.  What am I missing?

> ;; Fix up References header
> (add-hook 'message-header-setup-hook
> 	  'sj-unmangle-gmane-message-id)
>
> ;; Fix up In-Reply-To header
> (add-hook 'message-send-mail-hook
> 	  'sj-unmangle-gmane-message-id)
>
> Note that your own Gcc'ed copy will still have the mangled Message-ID in
> the In-Reply-To header, because message.el inserts that very late.  I
> can live with that, but if somebody has a better idea, please let me
> know.

I'd guess the unmangling should be done when `message-id' (fixing
References should not be necessary, I think) is put into
`message-reply-headers' (in `message-cite-original-1',
`message-reply', `message-followup'?).

Here's a very rough, untested patch:


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: rs-message-unmangle-Gmane.patch --]
[-- Type: text/x-patch, Size: 2632 bytes --]

Index: message.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/message.el,v
retrieving revision 7.277
diff -u -u -r7.277 message.el
--- message.el	6 Dec 2008 17:30:03 -0000	7.277
+++ message.el	8 Dec 2008 19:12:15 -0000
@@ -3665,6 +3667,23 @@
 	  (push (buffer-name buffer) buffers))))
     (nreverse buffers)))
 
+(defcustom message-gmane-mangled-mid-regexp
+  ;; FIXME: Don't use gnus-...
+  (concat "\\(<" gnus-button-valid-localpart-regexp "\\)"
+	  "__[0-9a-z.$]+\\$gmane\\$org" ;; bogus part
+	  ;; message-valid-fqdn-regexp probably would be too strict here
+	  "\\(@[a-z0-9][-.a-z0-9]+>\\)")
+  "FIXME"
+  :version "23.1" ;; No Gnus
+  :group 'message-headers
+  :type '(choice (const :tag "Don't unmangle MIDs" nil)
+		 regexp))
+
+(defun message-fix-gmane-mids (mids)
+  ;; FIXME: Don't use gnus-...
+  (gnus-replace-in-string mids message-gmane-mangled-mid-regexp
+			  "\\1\\2"))
+
 (defun message-cite-original-1 (strip-signature)
   "Cite an original message.
 If STRIP-SIGNATURE is non-nil, strips off the signature from the
@@ -3694,7 +3713,11 @@
 		      (or (message-fetch-field "subject") "none")
 		      (or (message-fetch-field "from") "nobody")
 		      (message-fetch-field "date")
-		      (message-fetch-field "message-id" t)
+		      (funcall
+		       (if (stringp message-gmane-mangled-mid-regexp)
+			   'message-fix-gmane-mids
+			 'identity)
+		       (message-fetch-field "message-id" t))
 		      (message-fetch-field "references")
 		      0 0 ""))))
       (mml-quote-region start end)
@@ -6622,7 +6658,11 @@
 	  (save-excursion
 	    (setq follow-to
 		  (funcall message-wide-reply-to-function)))))
-      (setq message-id (message-fetch-field "message-id" t)
+      (setq message-id (funcall
+			(if (stringp message-gmane-mangled-mid-regexp)
+			    'message-fix-gmane-mids
+			  'identity)
+			(message-fetch-field "message-id" t))
 	    references (message-fetch-field "references")
 	    date (message-fetch-field "date")
 	    from (or (message-fetch-field "from") "nobody")
@@ -6683,7 +6723,11 @@
 	    date (message-fetch-field "date")
 	    subject (or (message-fetch-field "subject") "none")
 	    references (message-fetch-field "references")
-	    message-id (message-fetch-field "message-id" t)
+	    message-id (funcall
+			(if (stringp message-gmane-mangled-mid-regexp)
+			    'message-fix-gmane-mids
+			  'identity)
+			(message-fetch-field "message-id" t))
 	    followup-to (message-fetch-field "followup-to")
 	    newsgroups (message-fetch-field "newsgroups")
 	    posted-to (message-fetch-field "posted-to")

[-- Attachment #3: Type: text/plain, Size: 199 bytes --]


Bye, Reiner.

[ Omitted gmane.dicuss from Mail-Followup-To as this is only relevant
  for Gnus. ]
-- 
       ,,,
      (o o)
---ooO-(_)-Ooo---  |  PGP key available  |  http://rsteib.home.pages.de/

[-- Attachment #4: Type: text/plain, Size: 169 bytes --]

_______________________________________________
Gmane-discuss mailing list
Gmane-discuss@hawk.netfonds.no
http://hawk.netfonds.no/cgi-bin/mailman/listinfo/gmane-discuss

Re: Fixing Gmane's mangled MIDs

[Sven Joachim]

On 2008-12-08 20:17 +0100, Reiner Steib wrote:

> On Sun, Dec 07 2008, Sven Joachim wrote:
>
>> On 2008-12-07 06:50 +0100, Lars Magne Ingebrigtsen wrote:
>>> When the same message comes from two mailing lists, and there's not
>>> enough information in them to identify them as such (to crosspost the
>>> message between the list), the Message-ID has to be renamed.  Remove the
>>> bit after __ to find the original ID.
>>
>> Here is some Emacs lisp code that does this automatically when replying
>> in Gnus:
>>
>> (defun sj-unmangle-gmane-message-id ()
>>   "Undo the Message-ID mangling that Gmane does on some postings."
>>   (save-excursion
>>     (save-restriction
>>       (message-narrow-to-headers)
>>       (goto-char (point-min))
>>       (while
>> 	  (re-search-forward "\\(<.*\\)" (point-max) t)

Aaargh, did I really write that?!  Emacs must have swallowed characters
in copy-and-paste, I swear that the line in my file looks

	  (re-search-forward "\\(<.*\\)" (point-max) t)

which makes a bit more sense and may sometimes even work (it did for me).

>> 	(replace-match "\\1")))))
>
> I would have expected a search for an expression like
> "__[0-9.]+\\$gmane\\$org@" or similar.  What am I missing?

At least the '.' and '$' characters that also occur in the mangled
Message-IDs, although this is fixed in your patch below.

> I'd guess the unmangling should be done when `message-id' (fixing
> References should not be necessary, I think) is put into
> `message-reply-headers' (in `message-cite-original-1',
> `message-reply', `message-followup'?).
>
> Here's a very rough, untested patch:
>
> Index: message.el
> ===================================================================
> RCS file: /usr/local/cvsroot/gnus/lisp/message.el,v
> retrieving revision 7.277
> diff -u -u -r7.277 message.el
> --- message.el	6 Dec 2008 17:30:03 -0000	7.277
> +++ message.el	8 Dec 2008 19:12:15 -0000
> @@ -3665,6 +3667,23 @@
>  	  (push (buffer-name buffer) buffers))))
>      (nreverse buffers)))
>  
> +(defcustom message-gmane-mangled-mid-regexp
> +  ;; FIXME: Don't use gnus-...
> +  (concat "\\(<" gnus-button-valid-localpart-regexp "\\)"
> +	  "__[0-9a-z.$]+\\$gmane\\$org" ;; bogus part
> +	  ;; message-valid-fqdn-regexp probably would be too strict here
> +	  "\\(@[a-z0-9][-.a-z0-9]+>\\)")
> +  "FIXME"
> +  :version "23.1" ;; No Gnus
> +  :group 'message-headers
> +  :type '(choice (const :tag "Don't unmangle MIDs" nil)
> +		 regexp))
> +
> +(defun message-fix-gmane-mids (mids)
> +  ;; FIXME: Don't use gnus-...
> +  (gnus-replace-in-string mids message-gmane-mangled-mid-regexp
> +			  "\\1\\2"))
> +
>  (defun message-cite-original-1 (strip-signature)
>    "Cite an original message.
>  If STRIP-SIGNATURE is non-nil, strips off the signature from the
> @@ -3694,7 +3713,11 @@
>  		      (or (message-fetch-field "subject") "none")
>  		      (or (message-fetch-field "from") "nobody")
>  		      (message-fetch-field "date")
> -		      (message-fetch-field "message-id" t)
> +		      (funcall
> +		       (if (stringp message-gmane-mangled-mid-regexp)
> +			   'message-fix-gmane-mids
> +			 'identity)
> +		       (message-fetch-field "message-id" t))
>  		      (message-fetch-field "references")
>  		      0 0 ""))))
>        (mml-quote-region start end)
> @@ -6622,7 +6658,11 @@
>  	  (save-excursion
>  	    (setq follow-to
>  		  (funcall message-wide-reply-to-function)))))
> -      (setq message-id (message-fetch-field "message-id" t)
> +      (setq message-id (funcall
> +			(if (stringp message-gmane-mangled-mid-regexp)
> +			    'message-fix-gmane-mids
> +			  'identity)
> +			(message-fetch-field "message-id" t))
>  	    references (message-fetch-field "references")
>  	    date (message-fetch-field "date")
>  	    from (or (message-fetch-field "from") "nobody")
> @@ -6683,7 +6723,11 @@
>  	    date (message-fetch-field "date")
>  	    subject (or (message-fetch-field "subject") "none")
>  	    references (message-fetch-field "references")
> -	    message-id (message-fetch-field "message-id" t)
> +	    message-id (funcall
> +			(if (stringp message-gmane-mangled-mid-regexp)
> +			    'message-fix-gmane-mids
> +			  'identity)
> +			(message-fetch-field "message-id" t))
>  	    followup-to (message-fetch-field "followup-to")
>  	    newsgroups (message-fetch-field "newsgroups")
>  	    posted-to (message-fetch-field "posted-to")
>
> Bye, Reiner.

I'll try this in the next days.

> [ Omitted gmane.dicuss from Mail-Followup-To as this is only relevant
>   for Gnus. ]

Cc'ed it anyway as people may read the thread there and even try the
nonsense I posted; at least I wanted to correct the regexp.

Sven