Re: [Caml-list] ocaml-pcre and UTF-8

[Mauricio Fernandez <mfp@acm.org>]

On Thu, Feb 16, 2012 at 10:29:30AM +0100, Philippe Strauss wrote:
> Hello caml'ers,
> 
> How do I convince PCRE to be UTF-8 friendly? example:
> 
> --
> open Pcre
> 
> external show : 'a -> string = "%show"

As an aside: where did you get this external from? I had to write a proper
show function on 3.12.0 in order to compile your example.

> let recomp = regexp ~flags:[`UTF8; `CASELESS]
> 
> let res_w = "(*UTF8)^(\w+)$"
                       =====
It would be \\w if anything, but the pcre manual warns that

       6. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly
       test characters  of any  code  value, but, by default, the characters
       that PCRE recognizes as digits, spaces, or word characters remain the
       same set as before, all with values  less  than  256.  This remains
       true  even when PCRE is built to include Unicode property support,
       because to do otherwise would slow down PCRE in many common cases. Note
       in particular that this applies to  \b and \B, because they are defined
       in terms of \w and \W. If you really want to test for a wider sense of,
       say, "digit", you can use explicit Unicode property tests  such  as
       \p{Nd}. Alternatively, if you set the PCRE_UCP option, the way that the
       character escapes work is changed so that Unicode properties are used
       to determine which characters  match.  There are more details in the
       section on generic character types in the pcrepattern documentation.

and pcrepattern lists this Unicode property:

  Xwd   Any Perl "word" character
  
so, given a suitable show function, both 

  let res_w = "^(\\p{Xwd}+)$"
and
  let res_w = "(*UCP)^(\\w+)$"

yield

./pcre_utf 
config_utf8=true
[|Some blurb|]
[|Some toxicité|]
[|Some velléités|]
[|Some à|]
[|Some où|]
[|Some über|]
Not_found was raised on "marie-jeanne" :-(

('-' not being a "word character" in my locale)

-- 
Mauricio Fernandez  -   http://eigenclass.org