9fans - fans of the OS Plan 9 from Bell Labs
 help / color / mirror / Atom feed
From: "Jack Johnson" <knapjack@gmail.com>
To: "Fans of the OS Plan 9 from Bell Labs" <9fans@cse.psu.edu>
Subject: Re: [9fans] awk, not utf aware...
Date: Mon,  3 Mar 2008 14:48:01 -0900	[thread overview]
Message-ID: <6e35c0620803031548v583c051flb4d9ace76e220998@mail.gmail.com> (raw)
In-Reply-To: <6e3f8d391c4219c9b0acb323d74603d6@quanstro.net>

On Thu, Feb 28, 2008 at 6:10 AM, erik quanstrom <quanstro@quanstro.net> wrote:
>  perhaps it would be more effective to break down the concept
>  a bit.  instead of a general locale hammer, why not expose some
>  operations that could go into a locale?  for example, have a base-
>  character folding switch that allows regexps to fold codpoints into
>  base codepoints so that íïìîi -> i.  this information is in the unicode
>  tables.  perhaps the language-dependent character mapping should
>  be specified explictly. &c.

Loosely-related tangent:

http://www.mail-archive.com/rsync@lists.samba.org/msg20395.html

> On the LINUX machines running utf-8 the ä is coded as $C3A4 which is
> in utf-8 equal to the character E4. The ä occupies in that way 2 bytes.
>
> I was very astonished, when I copied a mac-filename, pasted into a
> texteditor and looked at the file:
>
> In the mac-filename the letter ä is coded as: $61CC88, which in utf-8
> means the letter "a" followed by a $0308. (Combining diacritical marks)
> So the Mac combines the letter a with the two points above it instead
> using the E4 letter
> Now the things are clear: The filenames are different, in spite of
> looking equally.

So, if folding codepoints is a reasonable tactic, how many
representations do you need to fold?  How many binary representations
are needed to fold íïìîi -> i?

-Jack


  reply	other threads:[~2008-03-03 23:48 UTC|newest]

Thread overview: 25+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2008-02-26 12:18 Gorka Guardiola
2008-02-26 13:16 ` Martin Neubauer
2008-02-26 14:54   ` Gorka Guardiola
2008-02-26 20:24 ` erik quanstrom
2008-02-26 21:08   ` geoff
2008-02-26 21:21     ` Pietro Gagliardi
2008-02-26 21:24       ` erik quanstrom
2008-02-26 21:32       ` Steven Vormwald
2008-02-26 21:40         ` Pietro Gagliardi
2008-02-26 21:42           ` Pietro Gagliardi
2008-02-26 23:59           ` Steven Vormwald
2008-02-27  2:38       ` Joel C. Salomon
2008-02-29 17:00         ` Douglas A. Gwyn
2008-02-26 21:34     ` erik quanstrom
2008-02-27  7:36   ` Gorka Guardiola
2008-02-27 15:54     ` Sape Mullender
2008-02-27 20:01       ` Uriel
2008-02-28 19:06         ` [9fans] localization, unicode, regexps (was: awk, not utf aware...) Tristan Plumb
2008-02-28 15:10       ` [9fans] awk, not utf aware erik quanstrom
2008-03-03 23:48         ` Jack Johnson [this message]
2008-03-04  0:13           ` erik quanstrom
2008-02-27  9:57 erik quanstrom
2008-02-28 18:54 Aharon Robbins
2008-02-28 21:48 ` Uriel
2008-02-28 22:08   ` erik quanstrom

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=6e35c0620803031548v583c051flb4d9ace76e220998@mail.gmail.com \
    --to=knapjack@gmail.com \
    --cc=9fans@cse.psu.edu \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).