nnmail split multi-byte bug or feature?

nnmail split multi-byte bug or feature?

[stormwatch]

Note: It seems that the original message sent thorugh
news.individual.net was not delivered.

I did set this nnmail split method
("mail.utn.campusvirtual.\\1" "^X-Course-Name: \\(.*\\)$")

The problem is that the matched text is multi-byte encoded. Setting
(setq nnmail-mail-splitting-decodes t) did the trick. Well, almost,
because the group name shown in the *Group* buffer is:

28: mail.utn.campusvirtual.ingenier\355a y sociedad

instead of:

28: mail.utn.campusvirtual.ingeniería y sociedad

The dir '/home/stormwatch/Mail/mail/utn/campusvirtual/ingeniería y
sociedad' was created by gnus but I could not view the summary of the
group; not until I created a symlink with a latin-1 encoded dirname by
means of convmv:

david :: mail/utn/campusvirtual » ls -l
total 4
lrwxrwxrwx 1 stormwatch stormwatch   22 jul 22 05:09 ingenier?a y
sociedad -> ingeniería y sociedad
drwxrwxr-x 2 stormwatch stormwatch 4096 jul 22 05:19 ingeniería y
sociedad

My locale is utf-8 (LANG=es_AR.UTF-8)

--
Ezequiel Birman

Re: nnmail split multi-byte bug or feature?

[Katsumi Yamaoka]

Newsgroups: gnu.emacs.gnus (posted to news.eternal-september.org)
Cc: info-gnus-english@gnu.org

stormwatch wrote:
> Note: It seems that the original message sent thorugh
> news.individual.net was not delivered.

individual.net may forward articles posted to gnu.emacs.gnus to
a wrong moderator, or /dev/null, or the moderator may neglect it,
sigh.

> I did set this nnmail split method
> ("mail.utn.campusvirtual.\\1" "^X-Course-Name: \\(.*\\)$")

> The problem is that the matched text is multi-byte encoded. Setting
> (setq nnmail-mail-splitting-decodes t) did the trick. Well, almost,
> because the group name shown in the *Group* buffer is:

> 28: mail.utn.campusvirtual.ingenier\355a y sociedad

> instead of:

> 28: mail.utn.campusvirtual.ingeniería y sociedad

> The dir '/home/stormwatch/Mail/mail/utn/campusvirtual/ingeniería y
> sociedad' was created by gnus but I could not view the summary of the
> group; not until I created a symlink with a latin-1 encoded dirname by
> means of convmv:

> david :: mail/utn/campusvirtual » ls -l
> total 4
> lrwxrwxrwx 1 stormwatch stormwatch   22 jul 22 05:09 ingenier?a y
> sociedad -> ingeniería y sociedad
> drwxrwxr-x 2 stormwatch stormwatch 4096 jul 22 05:19 ingeniería y
> sociedad

> My locale is utf-8 (LANG=es_AR.UTF-8)

What is the mail back end you use?  As for the most recent Gnus:

,---- (info "(gnus)Non-ASCII Group Names") ----
| Currently Gnus supports non-ASCII group names not only with the `nntp'
| back end but also with the `nnml' back end and the `nnrss' back end.
`----

Besides them, nnimap supports non-ASCII group names since Mar.
2010 (I haven't tried it though).

I tried this for the nnml back end in the utf-8 locale and got
no problem:

(setq nnmail-mail-splitting-decodes t
      nnmail-split-methods
      '(("mail.utn.campusvirtual.\\1" "^X-Course-Name: \\(.*\\)$")
	("mail.misc" "")))

Re: nnmail split multi-byte bug or feature?

[Katsumi Yamaoka]

Newsgroups: gnu.emacs.gnus
Cc: info-gnus-english@gnu.org

stormwatch wrote:
> On Jul 27, 9:04 pm, Katsumi Yamaoka <yama...@jpl.org> wrote:
[...]
>> What is the mail back end you use?  As for the most recent Gnus:
[...]
> I use nnml and gnus included in an up-to-date emacs built from bzr.

Hmm, for the most recent Gnus there should be nothing you have
to do as you are in the utf-8 locale.

> Variables that might have to do with this problem:

> gnus-group-name-charset-method-alist is a variable defined in `gnus-
> group.el'.
> Its value is nil

ok.

> gnus-group-name-charset-group-alist is a variable defined in `gnus-
> group.el'.
> Its value is ((".*" . utf-8))

ok.

> nnmail-pathname-coding-system is a variable defined in `nnmail.el'.
> Its value is nil

It should be ok, but I had set it to utf-8.  I don't recall why
I did it.  But I verified now that nil is ok for non-ASCII group
names.

> file-name-coding-system is a variable defined in `C source code'.
> Its value is nil

ok.

> default-file-name-coding-system is a variable defined in `C source
> code'.
> Its value is utf-8-unix

ok.

> and also, this info node seems relevant:
> 2.17 Accessing groups of non-English names
> ==========================================
> ...
>    Note that when you copy or move articles from a non-ASCII group to
> another group, the charset used to encode and decode group names
> should
> be the same in both groups.  Otherwise the Newsgroups header will be
> displayed incorrectly in the article buffer.

[...]

> Perhaps the problem arised because I respooled the messages. ¿Would
> you be so kind as to try the same and see if you can reproduce the bug/
> feature?

No problem here.  What I did was:

1. Move articles in
   nnml:mail.utn.campusvirtual.ingeniería y sociedad
   to nnml:tmp .
2. `C-u G DEL' to delete
   nnml:mail.utn.campusvirtual.ingeniería y sociedad .
3. `B r' in nnml:tmp to respool the mails that have the header:
   X-Course-Name: ingeniería y sociedad
4. Confirm "nnml:mail.utn.campusvirtual.ingeniería y sociedad" is
   created and mails are spooled there.

> I, otoh, will try to remove everything and create a shiny
> brand-new fake-with-the-fingers-crossed incoming message.

BTW, I noticed `B nnml RET RET' in the group buffer shows
non-ASCII group names without properly decoding.  It should be
fixed!  How do you find possibly unknown mail groups that
nnmail-split creates?  (Text that X-Course-Name header contains
is unknown, isn't it?)

Re: nnmail split multi-byte bug or feature?

[Katsumi Yamaoka]

Newsgroups: gnu.emacs.gnus
Cc: info-gnus-english@gnu.org

Katsumi Yamaoka wrote:
> BTW, I noticed `B nnml RET RET' in the group buffer shows
> non-ASCII group names without properly decoding.  It should be
> fixed!

Fixed in the Emacs bzr trunk and the Gnus git trunk.

Stormwatch, maybe you have entries of which the group names are
corrupted in the ~/Mail/active file, and may want to fix them
manually.  To do that, one of ways will be to do find-file it as

C-x RET c utf-8-unix RET C-x C-f ~/Mail/active

, edit it, and save it.

Re: nnmail split multi-byte bug or feature?

[stormwatch]

On 29 jul, 23:09, Katsumi Yamaoka <yama...@jpl.org> wrote:
> Newsgroups: gnu.emacs.gnus
> Cc: info-gnus-engl...@gnu.org
>
> Katsumi Yamaoka wrote:
    > How do you find possibly unknown mail groups that
    > nnmail-split creates?  (Text that X-Course-Name header contains
is
    > unknown, isn't it?)

That's a very good question. I think gnus-auto-subscribed-groups whose
default
value is "^nnml\\|^nnfolder\\|^nnmbox\\|^nnmh\\|^nnbabyl\\|^nnmaildir"
should be
enough.

BTW, I'd like to change gnus-subscribe-options-newsgroup-method to
'gnus-subscribe-topics. Some time ago I tried to add a topic parameter
in my .gnus with no success, forcing me to edit the topic parameters
interactively each time I reinstalled or moved my os. Is it possible
to add something like '(("utn" (subscribe . "\\.utn"))) to gnus-
parameters? ("utn" being the topic name).


> Stormwatch, maybe you have entries of which the group names are
> corrupted in the ~/Mail/active file, and may want to fix them
> manually.  To do that, one of ways will be to do find-file it as
>
> C-x RET c utf-8-unix RET C-x C-f ~/Mail/active
>
> , edit it, and save it.

YES! That fixed the issue. I had to modify .newsrc.eld too. Thanks for
your help.

Re: nnmail split multi-byte bug or feature?

[Katsumi Yamaoka]

Newsgroups: gnu.emacs.gnus
Cc: info-gnus-english@gnu.org

stormwatch wrote:
>> Katsumi Yamaoka wrote:
>> How do you find possibly unknown mail groups that
>> nnmail-split creates?  (Text that X-Course-Name header contains
>> is unknown, isn't it?)

> That's a very good question. I think gnus-auto-subscribed-groups whose
> default
> value is "^nnml\\|^nnfolder\\|^nnmbox\\|^nnmh\\|^nnbabyl\\|^nnmaildir"
> should be enough.

I see.  I didn't recall it since the group level of the newly
created groups is 3 by default (I use the level 1 for the mail
groups and usually do `1 g' for checking new mails, but `3 g' or
`g' takes a long time because there are many foreign groups).

> BTW, I'd like to change gnus-subscribe-options-newsgroup-method to
> 'gnus-subscribe-topics. Some time ago I tried to add a topic parameter
> in my .gnus with no success, forcing me to edit the topic parameters
> interactively each time I reinstalled or moved my os. Is it possible
> to add something like '(("utn" (subscribe . "\\.utn"))) to gnus-
> parameters? ("utn" being the topic name).

I'll try it.

>> Stormwatch, maybe you have entries of which the group names are
>> corrupted in the ~/Mail/active file, and may want to fix them
>> manually.  To do that, one of ways will be to do find-file it as
>>
>> C-x RET c utf-8-unix RET C-x C-f ~/Mail/active
>>
>> , edit it, and save it.

> YES! That fixed the issue. I had to modify .newsrc.eld too. Thanks for
> your help.

I've made further change.  Maybe it doesn't affect you and Emacs
users, though.  The last change I made caused XEmacs to encode
non-ASCII nnml group names doubly in the ~/Mail/active file.  So,
XEmacs users who keep track of the latest, Gnus please take a look
at the thread in the ding list:

http://news.gmane.org/group/gmane.emacs.gnus.general/thread=69853

I'm sorry to XEmacs users for inconvenience.

Re: nnmail split multi-byte bug or feature?

[Katsumi Yamaoka]

Newsgroups: gnu.emacs.gnus
Cc: info-gnus-english@gnu.org

Katsumi Yamaoka wrote:
> stormwatch wrote:
>> BTW, I'd like to change gnus-subscribe-options-newsgroup-method to
>> 'gnus-subscribe-topics. Some time ago I tried to add a topic parameter
>> in my .gnus with no success, forcing me to edit the topic parameters
>> interactively each time I reinstalled or moved my os. Is it possible
>> to add something like '(("utn" (subscribe . "\\.utn"))) to gnus-
>> parameters? ("utn" being the topic name).

> I'll try it.

There seems to be no other way than to modify `gnus-topic-topology'
(~/.newsrc.eld specifies its value).  How about this?

--8<---------------cut here---------------start------------->8---
(require 'gnus-topic)

(add-hook
 'gnus-setup-news-hook
 (lambda ()
   ;; Create the "utn" topic if needed.
   (unless (member "utn" (gnus-topic-list))
     (gnus-topic-create-topic "utn" (caar gnus-topic-topology)))
   ;; Add `(subscribe . "\\.utn")' parameter to the "utn" topic if needed.
   (let* ((subscribe '((subscribe . "\\.utn")))
	  (topology (cadr (gnus-topic-find-topology "utn")))
	  (len (length topology))
	  (params (car (nthcdr 3 topology))))
     (cond ((= len 1)
	    (setcdr topology `(visible nil ,subscribe)))
	   ((= len 2)
	    (setcdr (cdr topology) `(nil ,subscribe)))
	   ((or (= len 3) (equal params '(nil)) (not (consp params)))
	    (setcdr (cddr topology) `(,subscribe)))
	   ((not (assq 'subscribe params))
	    (setcdr params (copy-sequence params))
	    (setcar params subscribe))))))
--8<---------------cut here---------------end--------------->8---

If you want to set the group level, try adding this one:

--8<---------------cut here---------------start------------->8---
(add-hook
 'gnus-setup-news-hook
 (lambda ()
   ;; Add `(subscribe-level . 1)' parameter to the "utn" topic if needed.
   (let ((params (car (nthcdr 3 (cadr (gnus-topic-find-topology "utn"))))))
     (unless (assq 'subscribe-level params)
       (setcdr params (copy-sequence params))
       (setcar params '(subscribe-level . 1)))
     params))
 'append)
--8<---------------cut here---------------end--------------->8---