A washing function to turn Unicode punctuation into ASCII

A washing function to turn Unicode punctuation into ASCII

[Lars Magne Ingebrigtsen]

I was reading Hacker News on my E72 yesterday, and many of the articles
were kinda, er, not very readable, because characters like “ were
rendered as 0x+201c in the article buffer, or something as awful.
Probably because the font wasn't available?  Or something?

So my question is: Does Emacs have a function to, er, translate these
characters into their nearest ASCII equivalents?  Or should I just write
a Gnus washing function for that?  I'm guessing that there aren't more
than 20 commonly used characters that it would make sense to wash this
way... 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: A washing function to turn Unicode punctuation into ASCII

[Steinar Bang]

>>>>> Lars Magne Ingebrigtsen <larsi@gnus.org>:

> So my question is: Does Emacs have a function to, er, translate these
> characters into their nearest ASCII equivalents?  Or should I just
> write a Gnus washing function for that?  I'm guessing that there
> aren't more than 20 commonly used characters that it would make sense
> to wash this way...

Doesn't that dumbquotes thing already do something like this?

Ie. deuglify.el...?

Re: A washing function to turn Unicode punctuation into ASCII

[Lars Magne Ingebrigtsen]

Steinar Bang <sb@dod.no> writes:

> Doesn't that dumbquotes thing already do something like this?

Yes, but only for dumbquotes.  :-)  The unicode->ascii equivalence thing
doesn't seem to exist in Gnus.  Or Emacs.  At least, I can't find
anything... 

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: A washing function to turn Unicode punctuation into ASCII

[Lawrence Mitchell]

Lars Magne Ingebrigtsen wrote:
> Steinar Bang <sb@dod.no> writes:

>> Doesn't that dumbquotes thing already do something like this?

> Yes, but only for dumbquotes.  :-)  The unicode->ascii equivalence thing
> doesn't seem to exist in Gnus.  Or Emacs.  At least, I can't find
> anything...


org-entities.el maybe?

Lawrence
-- 
Lawrence Mitchell <wence@gmx.li>

Re: A washing function to turn Unicode punctuation into ASCII

[Lars Magne Ingebrigtsen]

Lawrence Mitchell <wence@gmx.li> writes:

> org-entities.el maybe?

Yes, that looks exactly like what I need; thanks.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: A washing function to turn Unicode punctuation into ASCII

[Lars Magne Ingebrigtsen]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Yes, that looks exactly like what I need; thanks.

This is now implemented.  New command `W U'.

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: A washing function to turn Unicode punctuation into ASCII

[Kevin Ryde]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
> wash this way...

A bit of display table can do a similar thing, and helps all modes.
I made my own few lines for hypens and quotes which seem to be the most
common nonsense.  Is there more or better code doing the same?

http://user42.tuxfamily.org/unicode-disp/index.html

Re: A washing function to turn Unicode punctuation into ASCII

[Lars Magne Ingebrigtsen]

Kevin Ryde <user42@zip.com.au> writes:

> A bit of display table can do a similar thing, and helps all modes.
> I made my own few lines for hypens and quotes which seem to be the most
> common nonsense.  Is there more or better code doing the same?
>
> http://user42.tuxfamily.org/unicode-disp/index.html

This makes a lot of sense.  But I'm beginning to wonder whether Emacs
has any built-in functionality for doing this stuff?  It would seem like
a very natural thing to do -- declare that your Emacs is running in an
ASCII environment, and then Emacs sets up display tables to
automatically do this transformation...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: A washing function to turn Unicode punctuation into ASCII

[Andreas Schwab]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> This makes a lot of sense.  But I'm beginning to wonder whether Emacs
> has any built-in functionality for doing this stuff?  It would seem like
> a very natural thing to do -- declare that your Emacs is running in an
> ASCII environment, and then Emacs sets up display tables to
> automatically do this transformation...

See international/iso-ascii.el (only for latin1) or
international/latin1-disp.el.

Andreas.

-- 
Andreas Schwab, schwab@linux-m68k.org
GPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5
"And now for something completely different."

Re: A washing function to turn Unicode punctuation into ASCII

[Lars Magne Ingebrigtsen]

(We're discussing how to read utf-8 messages on a terminal that doesn't
support utf-8, e.g. Nokia E72.)

Andreas Schwab <schwab@linux-m68k.org> writes:

> See international/iso-ascii.el (only for latin1) or
> international/latin1-disp.el.

Hm...  so the function to call would be `latin1-display-ucs-per-lynx',
perhaps?  That's a huge mapping in there.  I'll give it a try.

Yes, that did the trick.  Perhaps that function should be
;;;###autoloaded and given a more understandable name?  It seems really
useful. 

Or are there other methods to do the same thing?

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen

Re: A washing function to turn Unicode punctuation into ASCII

[Kevin Ryde]

Andreas Schwab <schwab@linux-m68k.org> writes:
>
> international/latin1-disp.el.

Oh, yes, of course that was the inspiration for the name unicode-disp.el
:-)  I ended up mangling any window-display-table too, but much less
chars.

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:
>
> latin1-display-ucs-per-lynx

I had a lot of doubts about multi-char replacements, as they can make a
mess of lined-up text tables, or make lines very long.  As long as it
can be turned on and off I suppose you can look at it both ways when
needed.  (If it's coming out of html then of course asking the renderer
to do the replacements can flow etc at that level.)

I toyed with face colours as a visual clue, but didn't much like it,
even something modest like `escape-face'.  With many replacements the
screen breaks out in freckles when in all honesty you don't care if some
drongo thought variant hyphens or quotes were smart.

Re: A washing function to turn Unicode punctuation into ASCII

[Lars Magne Ingebrigtsen]

Kevin Ryde <user42@zip.com.au> writes:

>> latin1-display-ucs-per-lynx
>
> I had a lot of doubts about multi-char replacements, as they can make a
> mess of lined-up text tables, or make lines very long.  As long as it
> can be turned on and off I suppose you can look at it both ways when
> needed.

It's quite useful when reading RSS feeds on my E72...

-- 
(domestic pets only, the antidote for overdose, milk.)
  larsi@gnus.org * Lars Magne Ingebrigtsen