Korean chracters with nnmail-split-methods

Korean chracters with nnmail-split-methods

[Jinhyok Heo]

----
(setq nnmail-split-methods
      `(
	("SPAM"                  "^Subject:.*광고")
        ("SPAM"                  "^Subject:.*홍보")))
----

In short, above split rule doesn't work.

광고 and 홍보 is Korean words, which mean advertisement, spam. :(

I tried to filter out those spams with above split rule, but failed.
Of cource, the spams' Subject headers were not encoded with character
set.

-- 
| Jinhyok Heo (novembre @ ournature.org || http://ournature.org/~novembre/)
|--------------------------------------------------------------------------
| "We are still reaching for the sky. In the developed countries people
|  are coming back down, saying, `It's empty up there.'" --- a Ladakhi monk

Re: Korean chracters with nnmail-split-methods

[Kai Großjohann]

Jinhyok Heo <novembreN0$PAM@ournature.org> writes:

> I tried to filter out those spams with above split rule, but failed.
> Of cource, the spams' Subject headers were not encoded with character
> set.

Gnus doesn't decode before splitting for performance reasons.  Can
you put in variants for the various encodings?

kai
-- 
Simplification good!  Oversimplification bad!  (Larry Wall)

Re: Korean chracters with nnmail-split-methods

[Oystein Viggen]

* [Jinhyok Heo] 

> ----
> (setq nnmail-split-methods
>       `(
> 	("SPAM"                  "^Subject:.*±¤°í")
>         ("SPAM"                  "^Subject:.*È«º¸")))
> ----

I've had great success with the following procmail recipe:

:0 fhw
^Subject:.*=\?ks_c_5601-1987\?
| formail -A"X-Dustemikler: spam"

Then I can split on "X-Dustemikler: spam" in nnmail-split-methods.

Oystein
-- 
When in doubt: Flaunt.

Re: Korean chracters with nnmail-split-methods

[Jinhyok Heo]

>>>>> "KG" == Kai Gro^[-Aß^[$)Cjohann <Kai.Grossjohann@CS.Uni-Dortmund.DE> writes:

    KG> Gnus doesn't decode before splitting for performance reasons.
    KG> Can you put in variants for the various encodings?

Some misunderstanding.

Subjects of spams are not usually encoded, so Gnus doesn't have to
decode them before splitting, I think.

-- 
| Jinhyok Heo (novembre @ ournature.org || http://ournature.org/~novembre/)
|--------------------------------------------------------------------------
| "We are still reaching for the sky. In the developed countries people
|  are coming back down, saying, `It's empty up there.'" --- a Ladakhi monk

Re: Korean chracters with nnmail-split-methods

[Henrik Enberg]

Jinhyok Heo <novembreN0$PAM@ournature.org> writes:

> Subjects of spams are not usually encoded, so Gnus doesn't have to
> decode them before splitting, I think.

I use the following nnmail-split-fancy rule with success.

        ("from\\|subject" ".*\\?ks_c_5601-1987" "mail.spam")

Henrik
-- 
I know the human being and fish can coexist peacefully.
		-- George W. Bush

Re: Korean chracters with nnmail-split-methods

[Niklas Morberg]

I'm also having problems with this, see:

<http://www.gnus.org/list-archives/ding/200111/msg00526.html>
and
<http://www.gnus.org/list-archives/ding/200112/msg00420.html>

I've sent Lars the output from the lisp debugger (I had to
use breakpoints since the debugger was never entered --
maybe `Search failed' is not considered an error?).

Niklas

Re: Korean chracters with nnmail-split-methods

[Kai Großjohann]

Oystein Viggen <oysteivi@tihlde.org> writes:

> I've had great success with the following procmail recipe:
>
> :0 fhw
> ^Subject:.*=\?ks_c_5601-1987\?
> | formail -A"X-Dustemikler: spam"

I have this vague feeling that a Korean like Jinhyok might get
legitimate mails with Korean subjects.  Therefore, it is probably not
a good idea for him (?) to delete all Korean mails.

kai
-- 
Simplification good!  Oversimplification bad!  (Larry Wall)

Re: Korean chracters with nnmail-split-methods

[Henrik Enberg]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

> Oystein Viggen <oysteivi@tihlde.org> writes:
>
>> I've had great success with the following procmail recipe:
>>
>> :0 fhw
>> ^Subject:.*=\?ks_c_5601-1987\?
>> | formail -A"X-Dustemikler: spam"
>
> I have this vague feeling that a Korean like Jinhyok might get
> legitimate mails with Korean subjects.  Therefore, it is probably not
> a good idea for him (?) to delete all Korean mails.

But doesn't legitimate senders normally encode their mail properly?  It
seems that most spammers fail to do so.

Henrik
-- 
If a person doesn't have the capacity that we all want that person to
have, I suspect hope is in the far distant future, if at all.
		-- George W. Bush

Re: Korean chracters with nnmail-split-methods

[Jinhyok Heo]

>>>>> "HE" == Henrik Enberg <henrik@enberg.org> writes:

    HE> But doesn't legitimate senders normally encode their mail
    HE> properly?  It seems that most spammers fail to do so.

You're right in case of me. Most spamers who bug me don't encode their
"Subject"s.

-- 
| Jinhyok Heo (novembre @ ournature.org || http://ournature.org/~novembre/)
|--------------------------------------------------------------------------
| "We are still reaching for the sky. In the developed countries people
|  are coming back down, saying, `It's empty up there.'" --- a Ladakhi monk

Re: Korean chracters with nnmail-split-methods

[Lars Magne Ingebrigtsen]

Niklas Morberg <niklas.morberg@axis.com> writes:

> I've sent Lars the output from the lisp debugger (I had to
> use breakpoints since the debugger was never entered --
> maybe `Search failed' is not considered an error?).

You probably have to set `debug-ignored-errors' to nil for it to give
you a backtrace on all errors.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Korean chracters with nnmail-split-methods

[Lars Magne Ingebrigtsen]

Jinhyok Heo <novembreN0$PAM@ournature.org> writes:

> ----
> (setq nnmail-split-methods
>       `(
> 	("SPAM"                  "^Subject:.*광고")
>         ("SPAM"                  "^Subject:.*홍보")))
> ----
>
> In short, above split rule doesn't work.

If you put one of these spams into a buffer and try to
`(re-search-forward "^Subject:.*광고")', do you get any matches?

My guess is that Gnus has inhibited all the external-to-internal
parsing things, so "광고" won't match anything.  For it to match, Gnus
would have to (in addition to doing the MIME stuff, which (in the case
of spammers isn't usually necessary)), Gnus has to do the MULE thing.

I'm surprised that nobody has seen this bug before.  Or perhaps they
have, and are as non-plussed as to what to do about it as I am.  :-)

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Korean chracters with nnmail-split-methods

[Jinhyok Heo]

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

    LMI> If you put one of these spams into a buffer and try to
    LMI> `(re-search-forward "^Subject:.*광고")', do you get any matches?

Yes, that finds "광고".

Any idea?

-- 
| Jinhyok Heo (novembre @ ournature.org || http://ournature.org/~novembre/)
|--------------------------------------------------------------------------
| "We are still reaching for the sky. In the developed countries people
|  are coming back down, saying, `It's empty up there.'" --- a Ladakhi monk

Re: Korean chracters with nnmail-split-methods

[Lars Magne Ingebrigtsen]

Jinhyok Heo <novembreN0$PAM@ournature.org> writes:

>     LMI> If you put one of these spams into a buffer and try to
>     LMI> `(re-search-forward "^Subject:.*광고")', do you get any matches?
>
> Yes, that finds "광고".
>
> Any idea?

You could try setting `nnmail-incoming-coding-system' to a coding
system that will correctly read/convert Korean external encoding into
internal encoding.  That would be `euc-kr', I guess...

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Korean chracters with nnmail-split-methods

[Jinhyok Heo]

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

    LMI> You could try setting `nnmail-incoming-coding-system' to a
    LMI> coding system that will correctly read/convert Korean
    LMI> external encoding into internal encoding.  That would be
    LMI> `euc-kr', I guess...

When I set n-i-c-s to euc-kr as you have suggested, my Korean mails
are saved in a wrong way. Characters are currupted.

All Korean letters are saved all with '~'.

-- 
| Jinhyok Heo (novembre @ ournature.org || http://ournature.org/~novembre/)
|--------------------------------------------------------------------------
| "We are still reaching for the sky. In the developed countries people
|  are coming back down, saying, `It's empty up there.'" --- a Ladakhi monk

Re: Korean chracters with nnmail-split-methods

[Lars Magne Ingebrigtsen]

Jinhyok Heo <novembreN0$PAM@ournature.org> writes:

> When I set n-i-c-s to euc-kr as you have suggested, my Korean mails
> are saved in a wrong way. Characters are currupted.

So that's definitely not the way to go.  Hm.  How about if you used
the external encoding in the split rules instead?

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Korean chracters with nnmail-split-methods

[Jinhyok Heo]

>>>>> "LMI" == Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

    LMI> Jinhyok Heo <novembreN0$PAM@ournature.org> writes:
    >  When I set n-i-c-s to euc-kr as you have suggested, my Korean
    >  mails are saved in a wrong way. Characters are currupted.

    LMI> So that's definitely not the way to go.  Hm.  How about if
    LMI> you used the external encoding in the split rules instead?

Could you tell me more how I can do that?

-- 
| Jinhyok Heo (novembre @ ournature.org || http://ournature.org/~novembre/)
|--------------------------------------------------------------------------
| "We are still reaching for the sky. In the developed countries people
|  are coming back down, saying, `It's empty up there.'" --- a Ladakhi monk

Re: Korean chracters with nnmail-split-methods

[Lars Magne Ingebrigtsen]

Jinhyok Heo <novembreN0$PAM@ournature.org> writes:

>     LMI> So that's definitely not the way to go.  Hm.  How about if
>     LMI> you used the external encoding in the split rules instead?
>
> Could you tell me more how I can do that?

If you open a file containing that word without doing any decoding of
the file, you'll get something pretty binary-looking that you can put
in the split rules.  I think.

But that's probably the wrong approach, anyway.

nnmail does this when doing the splitting:

	;; Copy the headers into the work buffer.
	(insert-buffer-substring obuf beg end)
	;; Fold continuation lines.
	(goto-char (point-min))
	(while (re-search-forward "\\(\r?\n[ \t]+\\)+" nil t)
	  (replace-match " " t t))
       ...

So if we did optional parsing/decoding of the headers after they were
put into the work buffer, that should probably do the trick?  The
headers should be rfc2047-decoded and default charset stuff should be
allowed to be established.

Anybody got any comments on this?  Is this a reasonable approach?
          
-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: Korean chracters with nnmail-split-methods

[Kai Großjohann]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> So if we did optional parsing/decoding of the headers after they were
> put into the work buffer, that should probably do the trick?

I think that would DTRT.  Not that I'm an expert on this...

kai
-- 
Simplification good!  Oversimplification bad!  (Larry Wall)

Re: Korean chracters with nnmail-split-methods

[Lars Magne Ingebrigtsen]

Kai.Grossjohann@CS.Uni-Dortmund.DE (Kai Großjohann) writes:

>> So if we did optional parsing/decoding of the headers after they were
>> put into the work buffer, that should probably do the trick?
>
> I think that would DTRT.  Not that I'm an expert on this...

Ok; I've now added this.  Beware!  If your mail ends up in the wrong
place, give a shout.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen