regexp filter to match non-english characters

regexp filter to match non-english characters

[Robert D. Crawford]

Hello all,

I am subscribed to an rss feed from del.icio.us that gives me links to
pages tagged emacs.  There are a lot of titles that come through with
Asian characters of various kinds.  Since I don't read these languages,
I would like to filter them out.  

Is there a regexp class that will match?  Has anyone else done this?
How might one accomplish this?

Thanks for any help,
rdc
-- 
Robert D. Crawford                                      rdc1x@comcast.net

The descent to Hades is the same from every place.
		-- Anaxagoras

Re: regexp filter to match non-english characters

[Robert D. Crawford]

Hello Ted.  Thanks for all the work.  See below for comments.

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Thu, 06 Nov 2008 10:43:38 -0600 "Robert D. Crawford" <rdc1x@comcast.net> wrote: 
>
> RDC> Score files are great.  Truth be told, I'm just looking for what works.
> RDC> I like your solution but it will exclude posts with unicode characters,
> RDC> which is something I would like to avoid if possible.
>
> OK, so the question now is "how to tell if a character is in the Asian
> Unicode character ranges."  Unfortunately I recall Emacs' own character
> database will misrepresent some Latin characters, so I wouldn't depend
> on character properties.
>
> I looked at ftp://ftp.unicode.org/Public/UNIDATA/Blocks.txt and picked
> the blocks that looked useful.

[snip long code]

> Evaluating this (you have to load the 'cl library too) gives
>
> "[^\\u0D00-\\u0D7F\\u0D80-\\u0DFF\\u0E00-\\u0E7F\\u0E80-\\u0EFF\\u0F00-\\u0FFF\\u1000-\\u109F\\u1780-\\u17FF\\u1800-\\u18AF\\u1900-\\u194F\\u1950-\\u197F\\u1980-\\u19DF\\u19E0-\\u19FF\\u1A00-\\u1A1F\\u1B00-\\u1B7F\\u2E80-\\u2EFF\\u2F00-\\u2FDF\\u2FF0-\\u2FFF\\u3000-\\u303F\\u3040-\\u309F\\u30A0-\\u30FF\\u3100-\\u312F\\u3130-\\u318F\\u3190-\\u319F\\u31A0-\\u31BF\\u31C0-\\u31EF\\u31F0-\\u31FF\\u3200-\\u32FF\\u3300-\\u33FF\\u3400-\\u4DBF\\u4DC0-\\u4DFF\\u4E00-\\u9FFF\\uA000-\\uA48F\\uA490-\\uA4CF\\uAC00-\\uD7AF\\uF900-\\uFAFF]"
>
> I don't know if this is good enough for you, but the ranges are correct
> at least and you see how you can add more.  I tested with a few
> characters like this:
>
> (string-match (zme) "helloഀ")
>
> and it seems to work OK.  In a score file you'll have only one backslash
> but otherwise it should work.

I tested with this:

(string-match
"[^\\u0D00-\\u0D7F\\u0D80-\\u0DFF\\u0E00-\\u0E7F\\u0E80-\\u0EFF\\u0F00-\\u0FFF\\u1000-\\u109F\\u1780-\\u17FF\\u1800-\\u18AF\\u1900-\\u194F\\u1950-\\u197F\\u1980-\\u19DF\\u19E0-\\u19FF\\u1A00-\\u1A1F\\u1B00-\\u1B7F\\u2E80-\\u2EFF\\u2F00-\\u2FDF\\u2FF0-\\u2FFF\\u3000-\\u303F\\u3040-\\u309F\\u30A0-\\u30FF\\u3100-\\u312F\\u3130-\\u318F\\u3190-\\u319F\\u31A0-\\u31BF\\u31C0-\\u31EF\\u31F0-\\u31FF\\u3200-\\u32FF\\u3300-\\u33FF\\u3400-\\u4DBF\\u4DC0-\\u4DFF\\u4E00-\\u9FFF\\uA000-\\uA48F\\uA490-\\uA4CF\\uAC00-\\uD7AF\\uF900-\\uFAFF]"
">>") 

and it returns nil.  Great!

Testing with the unicode character » (C-q 273 RET) returns 0.  Curses.

Thank you for all your help.  This has been way more difficult than I
thought it would be.  Considering that, if you don't feel the need to
continue, I agree.  Some would take this as a personal challenge to make
it work.  I however have other things to do than to track down why this
regexp doesn't work.  I can't spend time on it and I surely don't expect
you to do so either.

Thanks again, but I think I'll be hitting C-k and saving my time for
other things.

rdc
-- 
Robert D. Crawford                                      rdc1x@comcast.net

QOTD:
	"I only touch base with reality on an as-needed basis!"

Re: regexp filter to match non-english characters

[Ted Zlatanov]

On Thu, 06 Nov 2008 10:43:38 -0600 "Robert D. Crawford" <rdc1x@comcast.net> wrote: 

RDC> Score files are great.  Truth be told, I'm just looking for what works.
RDC> I like your solution but it will exclude posts with unicode characters,
RDC> which is something I would like to avoid if possible.

OK, so the question now is "how to tell if a character is in the Asian
Unicode character ranges."  Unfortunately I recall Emacs' own character
database will misrepresent some Latin characters, so I wouldn't depend
on character properties.

I looked at ftp://ftp.unicode.org/Public/UNIDATA/Blocks.txt and picked
the blocks that looked useful.

(defun zme ()
  (let ((data "
0D00..0D7F; Malayalam
0D80..0DFF; Sinhala
0E00..0E7F; Thai
0E80..0EFF; Lao
0F00..0FFF; Tibetan
1000..109F; Myanmar
1780..17FF; Khmer
1800..18AF; Mongolian
1900..194F; Limbu
1950..197F; Tai Le
1980..19DF; New Tai Lue
19E0..19FF; Khmer Symbols
1A00..1A1F; Buginese
1B00..1B7F; Balinese
2E80..2EFF; CJK Radicals Supplement
2F00..2FDF; Kangxi Radicals
2FF0..2FFF; Ideographic Description Characters
3000..303F; CJK Symbols and Punctuation
3040..309F; Hiragana
30A0..30FF; Katakana
3100..312F; Bopomofo
3130..318F; Hangul Compatibility Jamo
3190..319F; Kanbun
31A0..31BF; Bopomofo Extended
31C0..31EF; CJK Strokes
31F0..31FF; Katakana Phonetic Extensions
3200..32FF; Enclosed CJK Letters and Months
3300..33FF; CJK Compatibility
3400..4DBF; CJK Unified Ideographs Extension A
4DC0..4DFF; Yijing Hexagram Symbols
4E00..9FFF; CJK Unified Ideographs
A000..A48F; Yi Syllables
A490..A4CF; Yi Radicals
AC00..D7AF; Hangul Syllables
F900..FAFF; CJK Compatibility Ideographs")
      (out ""))
  (dolist (line (split-string data "\n"))
    (dolist (item (split-string line ";"))
      (when (string-match "\\([0-9A-F]+\\)\\.\\.\\([0-9A-F]+\\)" item)
	(setq out 
	      (concat out (format 
			   "\\u%s-\\u%s"
			   (match-string 1 item)
			   (match-string 2 item) ))))))
  (concat "[^" out "]")))

Evaluating this (you have to load the 'cl library too) gives

"[^\\u0D00-\\u0D7F\\u0D80-\\u0DFF\\u0E00-\\u0E7F\\u0E80-\\u0EFF\\u0F00-\\u0FFF\\u1000-\\u109F\\u1780-\\u17FF\\u1800-\\u18AF\\u1900-\\u194F\\u1950-\\u197F\\u1980-\\u19DF\\u19E0-\\u19FF\\u1A00-\\u1A1F\\u1B00-\\u1B7F\\u2E80-\\u2EFF\\u2F00-\\u2FDF\\u2FF0-\\u2FFF\\u3000-\\u303F\\u3040-\\u309F\\u30A0-\\u30FF\\u3100-\\u312F\\u3130-\\u318F\\u3190-\\u319F\\u31A0-\\u31BF\\u31C0-\\u31EF\\u31F0-\\u31FF\\u3200-\\u32FF\\u3300-\\u33FF\\u3400-\\u4DBF\\u4DC0-\\u4DFF\\u4E00-\\u9FFF\\uA000-\\uA48F\\uA490-\\uA4CF\\uAC00-\\uD7AF\\uF900-\\uFAFF]"

I don't know if this is good enough for you, but the ranges are correct
at least and you see how you can add more.  I tested with a few
characters like this:

(string-match (zme) "helloഀ")

and it seems to work OK.  In a score file you'll have only one backslash
but otherwise it should work.

Ted

Re: regexp filter to match non-english characters

[Robert D. Crawford]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Thu, 06 Nov 2008 10:34:25 +0100 Michal Nazarewicz <mina86@tlen.pl> wrote: 
>
> MN> "<<" and ">>" have codes U+00AB and U+00BB so that's why they match but
> MN> there are plenty of other characters which may show up in an English
> MN> text, like (I'll use a (sequence of) ASCII characters which resembles
> MN> the proper unicode character) "`" (U+2018), "'" (U+2019), "``" (U+201C)
> MN> , "''" (U+201D) or "..." (U+2026) which will cause the entry to be
> MN> filtered out.
>
> Agreed.  It's not an easy problem without Unicode properties, but for
> the *subject* of the message it's a passable heuristic.
>
> MN> Besides, I think what you really meant was:
>
> MN> (string-match "[^\\0-\\177]" "string") 
>
> MN> since "1ff" is not a valid octal number.
>
> Yes.  Sorry.
>
> MN> I think that taking the title of the entry and checking if at least 90%
> MN> are ASCII characters would be sufficient to filter out Asian texts.  You
> MN> can also try taking first 100 (or so) characters of the body.  I think
> MN> you could use replace-regexp-in-string for that purpose:
>
> MN> (defun mn-non-english-p (string) 
> MN>   (>
> MN>    (* (length (replace-regexp-in-string "[^\\0-\\77]" "" string)) 10)
> MN>    (* (length string) 9)))
>
> That might work, but for a score file a simple regular expression is
> better, and I understood the OP to need a score file.

Score files are great.  Truth be told, I'm just looking for what works.
I like your solution but it will exclude posts with unicode characters,
which is something I would like to avoid if possible.

Thanks,
rdc
-- 
Robert D. Crawford                                      rdc1x@comcast.net

semper en excretus

Re: regexp filter to match non-english characters

[Robert D. Crawford]

Michal Nazarewicz <mina86@tlen.pl> writes:

> "<<" and ">>" have codes U+00AB and U+00BB so that's why they match but
> there are plenty of other characters which may show up in an English
> text, like (I'll use a (sequence of) ASCII characters which resembles
> the proper unicode character) "`" (U+2018), "'" (U+2019), "``" (U+201C)
> , "''" (U+201D) or "..." (U+2026) which will cause the entry to be
> filtered out.
>
> Besides, I think what you really meant was:
>
> (string-match "[^\\0-\\177]" "string") 
>
> since "1ff" is not a valid octal number.
>
> I think that taking the title of the entry and checking if at least 90%
> are ASCII characters would be sufficient to filter out Asian texts.  You
> can also try taking first 100 (or so) characters of the body.  I think
> you could use replace-regexp-in-string for that purpose:
>
> (defun mn-non-english-p (string) 
>   (>
>    (* (length (replace-regexp-in-string "[^\\0-\\77]" "" string)) 10)
>    (* (length string) 9)))

I like the way this looks.  Seems that it will allow the characters I
would like to keep but remove posts which I cannot read.  Here is my
problem, and forgive what I can only assume is my lack of understanding
in doing complex scoring/filtering, but I don't know how to implement
this.  I have read through the gnus info manual section on scoring and
don't see anywhere that I can plug in a function to perform this action
on the subject.  I will readily admit that it is probable that I just
missed it.  If someone could point me to the place where this is
explained in the manual I would be very appreciative.  

I must add that the body of the posts from this nnrss group consist of
only the following lines:

Tables Linearized                  
 About This Style 

link

comments

About This Style

Table contents are turned into a sequence of paragraphs, one per cell.

The part about "Tables Linearized" is added by something I use.  The
explanation is on the last line.  I mention this because I don't think
scoring on the body of the post will work in this case.

Thanks for all the help from both you and Ted,
rdc
-- 
Robert D. Crawford                                      rdc1x@comcast.net

Your temporary financial embarrassment will be relieved in a surprising manner.

Re: regexp filter to match non-english characters

[Ted Zlatanov]

On Thu, 06 Nov 2008 10:34:25 +0100 Michal Nazarewicz <mina86@tlen.pl> wrote: 

MN> "<<" and ">>" have codes U+00AB and U+00BB so that's why they match but
MN> there are plenty of other characters which may show up in an English
MN> text, like (I'll use a (sequence of) ASCII characters which resembles
MN> the proper unicode character) "`" (U+2018), "'" (U+2019), "``" (U+201C)
MN> , "''" (U+201D) or "..." (U+2026) which will cause the entry to be
MN> filtered out.

Agreed.  It's not an easy problem without Unicode properties, but for
the *subject* of the message it's a passable heuristic.

MN> Besides, I think what you really meant was:

MN> (string-match "[^\\0-\\177]" "string") 

MN> since "1ff" is not a valid octal number.

Yes.  Sorry.

MN> I think that taking the title of the entry and checking if at least 90%
MN> are ASCII characters would be sufficient to filter out Asian texts.  You
MN> can also try taking first 100 (or so) characters of the body.  I think
MN> you could use replace-regexp-in-string for that purpose:

MN> (defun mn-non-english-p (string) 
MN>   (>
MN>    (* (length (replace-regexp-in-string "[^\\0-\\77]" "" string)) 10)
MN>    (* (length string) 9)))

That might work, but for a score file a simple regular expression is
better, and I understood the OP to need a score file.

Ted

Re: regexp filter to match non-english characters

[Michal Nazarewicz]

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Wed, 05 Nov 2008 14:14:31 -0600 "Robert D. Crawford" <rdc1x@comcast.net> wrote:
>
> RDC> Ted Zlatanov <tzz@lifelogs.com> writes:
>>> (string-match "[^\\000-\\1ff]" "hello")   ;; OK
>>> 
>>> This will match character values over 0x1FF, which is the limit of
>>> extended ASCII.  Does that work for you?
>
> RDC> Will this match the unicode double ">" and the like?  Some people
> RDC> feel the need to use these in their breadcrumbs and such.  If
> RDC> there is no way to just filter out the foreign characters, I will
> RDC> use it.
>
> You can just try it!
>
> (string-match "[^\\000-\\1ff]" "»") ;; returns 0, meaning it's a match
> (string-match "[^\\000-\\1ff]" ">>") ;; returns nil, meaning it's not a match
>
> RDC> The other possibility is to lower permanently on each character that is
> RDC> read to me, but this seems tedious and time consuming on my part and
> RDC> likely slow for gnus to score.
>
> Nah, the above should work.  You will need a single backslash instead of
> two, though (the doubling is needed to tell Emacs Lisp that's a real
> backslash inside the string when it reads it in).

"<<" and ">>" have codes U+00AB and U+00BB so that's why they match but
there are plenty of other characters which may show up in an English
text, like (I'll use a (sequence of) ASCII characters which resembles
the proper unicode character) "`" (U+2018), "'" (U+2019), "``" (U+201C)
, "''" (U+201D) or "..." (U+2026) which will cause the entry to be
filtered out.

Besides, I think what you really meant was:

(string-match "[^\\0-\\177]" "string") 

since "1ff" is not a valid octal number.


I think that taking the title of the entry and checking if at least 90%
are ASCII characters would be sufficient to filter out Asian texts.  You
can also try taking first 100 (or so) characters of the body.  I think
you could use replace-regexp-in-string for that purpose:

(defun mn-non-english-p (string) 
  (>
   (* (length (replace-regexp-in-string "[^\\0-\\77]" "" string)) 10)
   (* (length string) 9)))

-- 
Best regards,                                         _     _
 .o. | Liege of Serenly Enlightened Majesty of      o' \,=./ `o
 ..o | Computer Science,  Michal "mina86" Nazarewicz   (o o)
 ooo +--<mina86*tlen.pl>--<jid:mina86*jabber.org>--ooO--(_)--Ooo--

Re: regexp filter to match non-english characters

[Ted Zlatanov]

On Wed, 05 Nov 2008 14:14:31 -0600 "Robert D. Crawford" <rdc1x@comcast.net> wrote: 

RDC> Ted Zlatanov <tzz@lifelogs.com> writes:
>> (string-match "[^\\000-\\1ff]" "hello")   ;; OK
>> (string-match "[^\\000-\\1ff]" "здрасти") ;; not OK (Unicode characters)
>> 
>> This will match character values over 0x1FF, which is the limit of
>> extended ASCII.  Does that work for you?

RDC> Will this match the unicode double ">" and the like?  Some people feel
RDC> the need to use these in their breadcrumbs and such.  If there is no way
RDC> to just filter out the foreign characters, I will use it.  

You can just try it!

(string-match "[^\\000-\\1ff]" "»") ;; returns 0, meaning it's a match
(string-match "[^\\000-\\1ff]" ">>") ;; returns nil, meaning it's not a match

Put the cursor after the closing parenthesis and hit C-x C-e in Emacs to
see the result.

RDC> The other possibility is to lower permanently on each character that is
RDC> read to me, but this seems tedious and time consuming on my part and
RDC> likely slow for gnus to score.

Nah, the above should work.  You will need a single backslash instead of
two, though (the doubling is needed to tell Emacs Lisp that's a real
backslash inside the string when it reads it in).

Ted

Re: regexp filter to match non-english characters

[Robert D. Crawford]

Hello Ted,

Ted Zlatanov <tzz@lifelogs.com> writes:

> On Wed, 05 Nov 2008 11:26:14 -0600 "Robert D. Crawford" 
> <rdc1x@comcast.net> wrote:
>
>> I am subscribed to an rss feed from del.icio.us that gives me
>> links to pages tagged emacs.  There are a lot of titles that come
>> through with Asian characters of various kinds.  Since I don't
>> read these languages, I would like to filter them out.
>
>> Is there a regexp class that will match?  Has anyone else done
>> this?  How might one accomplish this?
>
> One idea is:
>
> (string-match "[^\\000-\\1ff]" "hello")   ;; OK
> (string-match "[^\\000-\\1ff]" "здрасти") ;; not OK (Unicode characters)
>
> This will match character values over 0x1FF, which is the limit of
> extended ASCII.  Does that work for you?

Will this match the unicode double ">" and the like?  Some people feel
the need to use these in their breadcrumbs and such.  If there is no way
to just filter out the foreign characters, I will use it.  

The other possibility is to lower permanently on each character that is
read to me, but this seems tedious and time consuming on my part and
likely slow for gnus to score.

Thanks for your help,
rdc
-- 
Robert D. Crawford                                      rdc1x@comcast.net

To add insult to injury.
		-- Phaedrus

Re: regexp filter to match non-english characters

[Ted Zlatanov]

On Wed, 05 Nov 2008 11:26:14 -0600 "Robert D. Crawford" <rdc1x@comcast.net> wrote: 

RDC> I am subscribed to an rss feed from del.icio.us that gives me links to
RDC> pages tagged emacs.  There are a lot of titles that come through with
RDC> Asian characters of various kinds.  Since I don't read these languages,
RDC> I would like to filter them out.  

RDC> Is there a regexp class that will match?  Has anyone else done this?
RDC> How might one accomplish this?

One idea is:

(string-match "[^\\000-\\1ff]" "hello")   ;; OK
(string-match "[^\\000-\\1ff]" "здрасти") ;; not OK (Unicode characters)

This will match character values over 0x1FF, which is the limit of
extended ASCII.  Does that work for you?

Ted