On Thu, Dec 12, 2013 at 09:30:06PM -0700, writeonce@midipix.org wrote:

   Hello,

   While working on code that converts arguments from utf-16 to utf-8, I
   found myself wondering about the "responsibility" for checking
   well-formedness of utf-8 strings that are passed to the kernel.  As I
   suspected, validation of these strings takes place neither in the kernel,
   nor in the C library.  The attached program demonstrates this by creating
   a file named <0xE0 0x9F 0x80>, which according to the Unicode Standard
   (6.2, p. 95) is an ill-formed byte sequence.

   I am not sure whether this can officially be considered a bug, and it is
   quite clear that fixing this is going to entail some performance penalty. 
   That being said, after deleting this file from my Ubuntu desktop most (but
   not all) attempts to open the Trash folder made Nautilus crash, and it was
   only after deleting the file permanently from the shell that order had
   been restored...

There's nothing in POSIX that says that filenames have to be valid
strings in the current locale's encoding -- in fact, this is highly
problematic to enforce on implementations other than musl, such as
glibc, where the encoding might vary by locale and where different
users might be using locales with different encodings.

But there's also nothing that says arbitrary byte sequences (excluding
of course those containing '/' and NUL) have to be accepted as
filenames either. The historical _expectation_ and practice has been
that filenames can contain arbitrary byte sequences. And Linus in
particular is opposed to changing this, though there's been some
indicastion (I don't have references right off) that he might be open
to optional restrictions at the kernel level.

What's clear to me is that restrictions at the libc level are not
useful. If your concern is that creating files with illegal sequences
in their names can confuse/break/crash some software, adding a
restriction on file creation in libc won't help. A malicious user can
just make the syscalls directly to make malicious filenames. On the
other hand, having the restriction in libc would be annoying because
it would _prevent_ you from renaming or deleting these bad filenames
using standard tools; you'd have to use special tools that make the
syscalls directly.

As always, you are absolutely right:-)  but my situation is slightly different, though; the input I receive is expected to be in utf-8, but the nt kernel only accepts utf-16.  This means that I need to choose between conversion that is based on bit distribution only, which might  produce ill-formed utf-16 byte sequences, or do all the validation on my end despite the minor performance penalty.  Since path strings are normally only a few hundred bytes long, and given that the nt kernel cannot be (easily) debugged from my end, I'm leaning towards the latter option.

So if you want protection against illegal sequences in filenames (personally, I want this too) the right place to lobby for it (and propose an optional feature) is in the kernel, not in libc. Rich