UTF-8 Support

UTF-8 Support

[Simon Josefsson]

I committed the following to support UTF-8 group names better (which
supposedly the USEFOR group is trying to migrate to).  It should be
possible to subscribe to dk.test.utf8-æøå from sunsite.dk and things
should work.  There may be problems when Gcc'ing UTF-8 names though,
patches welcome.

You need to frob `gnus-group-name-charset-group-alist' to something
like `((".*" . utf-8))' though.  Per, do you want to commit the change
to what you suggested?

>From GNUS-NEWS:

** Group names are treated as UTF-8 by default.

This is supposedly what USEFOR wants to migrate to.  See
`gnus-group-name-charset-group-alist' and
`gnus-group-name-charset-method-alist' for customization.

ChangeLog entry:

2001-10-06  Simon Josefsson  <jas@extundo.com>

	Support UTF-8 group names better.
	
	* message.el (message-check-news-header-syntax): Encode group
	names before comparison.

	* gnus-msg.el (gnus-copy-article-buffer): Run all
	`gnus-article-decode-hook's except `article-decode-charset'
	instead of hardcoding call to one of them.

	* gnus-art.el (gnus-article-decode-hook): Add
	`article-decode-group-name'.
	(article-decode-group-name): New function, use `g-d-n'.

	* gnus-group.el (gnus-group-insert-group-line): Decode
	gnus-tmp-group using `g-d-n'.

	* gnus-util.el (gnus-decode-newsgroups): New function.

The patch:

Index: gnus-art.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/gnus-art.el,v
retrieving revision 6.109
diff -u -r6.109 gnus-art.el
--- gnus-art.el	2001/09/28 11:22:41	6.109
+++ gnus-art.el	2001/10/06 21:04:01
@@ -638,7 +638,8 @@
 			       (face :value default)))))
 
 (defcustom gnus-article-decode-hook
-  '(article-decode-charset article-decode-encoded-words)
+  '(article-decode-charset article-decode-encoded-words
+			   article-decode-group-name)
   "*Hook run to decode charsets in articles."
   :group 'gnus-article-headers
   :type 'hook)
@@ -1753,6 +1754,22 @@
     (save-restriction
       (article-narrow-to-head)
       (funcall gnus-decode-header-function (point-min) (point-max)))))
+
+(defun article-decode-group-name ()
+  "Decode group names in `Newsgroups:'."
+  (let ((inhibit-point-motion-hooks t)
+	buffer-read-only
+	(method (gnus-find-method-for-group gnus-newsgroup-name)))
+    (when (and (or gnus-group-name-charset-method-alist
+		   gnus-group-name-charset-group-alist)
+	       (gnus-buffer-live-p gnus-original-article-buffer)
+	       (mail-fetch-field "Newsgroups"))
+      (nnheader-replace-header "Newsgroups"
+			       (gnus-decode-newsgroups
+				(with-current-buffer
+				    gnus-original-article-buffer
+				  (mail-fetch-field "Newsgroups"))
+				gnus-newsgroup-name method)))))
 
 (defun article-de-quoted-unreadable (&optional force read-charset)
   "Translate a quoted-printable-encoded article.
Index: gnus-group.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/gnus-group.el,v
retrieving revision 6.40
diff -u -r6.40 gnus-group.el
--- gnus-group.el	2001/09/29 19:25:31	6.40
+++ gnus-group.el	2001/10/06 21:04:02
@@ -1339,7 +1339,9 @@
      (point)
      (prog1 (1+ (point))
        ;; Insert the text.
-       (eval gnus-group-line-format-spec))
+       (let ((gnus-tmp-group (gnus-group-name-decode
+			      gnus-tmp-group group-name-charset)))
+	 (eval gnus-group-line-format-spec)))
      `(gnus-group ,(gnus-intern-safe gnus-tmp-group gnus-active-hashtb)
 		  gnus-unread ,(if (numberp number)
 				   (string-to-int gnus-tmp-number-of-unread)
Index: gnus-msg.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/gnus-msg.el,v
retrieving revision 6.47
diff -u -r6.47 gnus-msg.el
--- gnus-msg.el	2001/09/23 06:17:41	6.47
+++ gnus-msg.el	2001/10/06 21:04:03
@@ -579,7 +579,10 @@
 			   (or (message-goto-body) (point-max)))
 	    ;; Insert the original article headers.
 	    (insert-buffer-substring gnus-original-article-buffer beg end)
-	    (article-decode-encoded-words))))
+	    ;; Decode charsets.
+	    (let ((gnus-article-decode-hook
+		   (delq 'article-decode-charset gnus-article-decode-hook)))
+	      (run-hooks 'gnus-article-decode-hook)))))
       gnus-article-copy)))
 
 (defun gnus-post-news (post &optional group header article-buffer yank subject
Index: gnus-util.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/gnus-util.el,v
retrieving revision 6.19
diff -u -r6.19 gnus-util.el
--- gnus-util.el	2001/08/24 04:43:11	6.19
+++ gnus-util.el	2001/10/06 21:04:03
@@ -187,6 +187,14 @@
 		   (search-forward ":" eol t)
 		   (point)))))
 
+(defun gnus-decode-newsgroups (newsgroups group &optional method)
+  (let ((method (or method (gnus-find-method-for-group group))))
+    (mapconcat (lambda (group)
+		 (gnus-group-name-decode group (gnus-group-name-charset
+						method group)))
+	       (message-tokenize-header newsgroups ", ")
+	       ", ")))
+
 (defun gnus-remove-text-with-property (prop)
   "Delete all text in the current buffer with text property PROP."
   (save-excursion
Index: message.el
===================================================================
RCS file: /usr/local/cvsroot/gnus/lisp/message.el,v
retrieving revision 6.119
diff -u -r6.119 message.el
--- message.el	2001/09/29 19:25:32	6.119
+++ message.el	2001/10/06 21:04:04
@@ -2915,12 +2915,15 @@
 		     (if followup-to
 			 (concat newsgroups "," followup-to)
 		       newsgroups)))
+	    (method (if (message-functionp message-post-method)
+			(funcall message-post-method)
+		      message-post-method))
 	    (known-groups
-	     (mapcar (lambda (n) (gnus-group-real-name n))
-		     (gnus-groups-from-server
-		      (if (message-functionp message-post-method)
-			  (funcall message-post-method)
-			message-post-method))))
+	     (mapcar (lambda (n)
+		       (gnus-group-name-decode 
+			(gnus-group-real-name n)
+			(gnus-group-name-charset method n)))
+		     (gnus-groups-from-server method)))
 	    errors)
        (while groups
 	 (unless (or (equal (car groups) "poster")

Re: UTF-8 Support

[Per Abrahamsen]

Simon Josefsson <jas@extundo.com> writes:

> You need to frob `gnus-group-name-charset-group-alist' to something
> like `((".*" . utf-8))' though.  Per, do you want to commit the change
> to what you suggested?

I have done that, and hacked a bit more on the code.  UTF8 groupnames
should "just work" for users of Emacs 21 now.

Re: UTF-8 Support

[Katsumi Yamaoka]

[-- Attachment #1: Type: text/plain, Size: 349 bytes --]

Hi,

I am using XEmacs with Mule-UCS v0.84.  It works for encoding
and decoding UTF-8 newsgroup names.  Could you please apply the
following patch?

2001-10-10  Katsumi Yamaoka  <yamaoka@jpl.org>

	* gnus-group.el (gnus-group-name-charset-group-alist): Use
	`find-coding-system' for XEmacs to check whether the coding-system
	`utf-8' is available.


[-- Attachment #2: gnus-group.el.diff --]
[-- Type: application/x-patch, Size: 599 bytes --]

[-- Attachment #3: Type: text/plain, Size: 595 bytes --]


FYI:

You can use Mule-UCS package for both XEmacs with MULE and FSF
Emacs 20.7/21.  It is available from:

:::The primary distribution:::
  ftp://ftp.m17n.org/pub/mule/Mule-UCS/Mule-UCS-0.84.tar.gz

:::Unofficial XEmacs package (made by me):::
  ftp://ftp.jpl.org/xemacs/packages/Mule-UCS-0.84-pkg.tar.gz

I am using the following codes in my XEmacs startup file.

(require 'un-define)
(coding-system-put 'utf-8 'category 'utf-8)
(set-coding-category-system 'utf-8 'utf-8)
(set-coding-priority-list (cons 'utf-8
				(delq 'utf-8 (coding-priority-list))))
-- 
Katsumi Yamaoka <yamaoka@jpl.org>

Re: UTF-8 Support

[Simon Josefsson]

On Wed, 10 Oct 2001, Katsumi Yamaoka wrote:

> Hi,
>
> I am using XEmacs with Mule-UCS v0.84.  It works for encoding
> and decoding UTF-8 newsgroup names.  Could you please apply the
> following patch?

Done.  Good to know that this works in XEmacs with UTF-8 support, thanks
for the report.