RE: validation of utf-8 strings passed as system call arguments

[<writeonce@midipix.org>]

On 12/13/2013 02:46 PM, Rich Felker wrote:
> On Fri, Dec 13, 2013 at 11:57:54AM -0700, writeonce@midipix.org
> wrote:
>> There's no way to convert between UTF-8 and UTF-16 without 
>> parsing/decoding the UTF-8, which includes validating it for free
>> if your parser is written properly. Failure to validate would lead
>> to all sorts of bugs, many of them dangerous, including things like
>> treating strings not containing '/', '\', ':', '.', etc. as if they
>> contained those characters, resulting in directory escape
>> vulnerabilities.
>> 
>> Absolutely, and this is something that I am checking anyway.  But
>> there is also the special case where an ill-formed utf-8 byte
>> sequence can still result in a valid code point, which can then be
>> safely converted to utf-16.  These cases, which are generally known
>> as the problem of the "non shortest form," pertain to byte
>> sequences that used to be valid before Unicode version 3.1, but are
>> now forbidden, hence table 3-7 of the current (6.2) standard.
> 
> What I was saying is that you don't have this problem if you're 
> parsing/decoding UTF-8 correctly. And parsing it correctly is not 
> harder/slower than doing it the way that results in misinterpreting 
> illegal sequences as "non shortest form" for other characters. A
> good treatment of the subject (and near-optimal implementation) is
> here:
> 
> http://bjoern.hoehrmann.de/utf-8/decoder/dfa/
> 
> My implementation in musl is based on the same ideas (UTF-8 decoding 
> as a state machine rather than complex conditionals) but I reduced
> the size of the state from two ints to just one and reduced the size
> of the state table significantly by essentially encoding the
> transitions and partial character values into the state values.

Thanks for the tips and reference.  Once everything else is working I'll
certainly switch to a method that follows either your, or Hoehrmann's
optimization (which I'll admittedly need more than a few minutes to
understand...)  For the time being I am leaving the set of conditionals
that follows the standard and table 3-7, as that is very easy to
implement.  And with the target strings being relative shortness,
hopefully this won't even bear any real performance consequences.

> 
> If you're making UTF-8 to UTF-16 conversions to feed to the Windows 
> kernel filesystem code, I'd do them at the last possible opportunity 
> before passing the strings to the kernel, and just generate a fake 
> error equivalent to "file does not exist" or "invalid filename" if
> the conversion encounters any illegal sequences.

Indeed, that is exactly how I am doing this.
zg

> 
> Rich