Re: [9fans] Woes of New Language Support

[erik quanstrom <quanstro@quanstro.net>]

On Sun Jul 26 14:40:56 EDT 2009, knapjack@gmail.com wrote:
> If I'm reading you right, you're saying it might be easier if
> everything were encoded as combining (or maybe more aptly
> non-combining) codes, regardless of language?
>
> So, we might encode 'Waffles' as w+upper a f f l e s and let the
> renderer (if there is one) handle the presentation of the case shift
> and the potential ligature, but things like grep get noticeably easier
> with no overlap of ő and o+umlaut.
>
> Again, oversimplified, with no real understanding on my part of the
> depth or breadth of the problem space.

you understand.  except, i was taking the opposite position.

if you did for english what is done for indic languages,
if you typed 'this is a sentence.' the 't' would be capitalized
as soon as you typed the '.'.  there's no hint that this rule
need to be applied, the rendered would just have to know
it.  in ak's example a certain combination of codepoints yields
a specific 'letter'.  (i hope i have that right.)  the renderer is
just supposed to know this.  so for consistency and reducing
the need for complicated language-specific (how do we know
that the text represented is actually from the language we think
it is?), i would force the producer to declare the combinations.

btw, the search problem is not at all solved by standardizing
(or is that standardising?) the combiners problem.  consider
the following bits of unicode fun:

; grep 'zero width' /lib/unicode
200b	zero width space
200c	zero width non-joiner
200d	zero width joiner
feff	zero width no-break space

i'm sure that someone more conversant in unicode could
point out other points of real difficulty.

how do you tell unicode from uni\ufeffcode?  not only
is that an annoyance, but it could be a pretty interesting
security problem.  and what a gift for spammers!

- erik