Re: UTF-8 input [was Re: PATCH: zle_params.c]

[Bart Schaefer <schaefer@brasslantern.com>]

On Jan 30,  1:07am, Peter Stephenson wrote:
} Subject: Re: UTF-8 input [was Re: PATCH: zle_params.c]
}
} I thought of the following: self-insert could take a single character,
} as at present, and then test if it was the initial part of a multibyte
} character.  If it was, it could read the rest; we might need a timeout to
} avoid an infinite hang on systems that didn't do multibyte input
} properly

This would mean what, in terms of binding other functions to wide chars?
That they'd behave like escape sequences do now?  I would think you'd
want to decide whether the input is a wide char at a lower level than
that.  Otherwise don't you have issues if what the user really means to
bind to self-insert is a single-byte character that happens to have the
high bit set?

It seems to me that some stage of the input process has to be "told"
that the input stream is UTF-8 rather than e.g. iso-8859-something.  If
it's the widget level that's going to handle that [*], I think it'd be
most useful to create a self-insert-multibyte which does in fact wait
indefinitely (or at least, longer than the normal escape-sequence key
timeout) for the "rest" of a multibyte character after the first byte is
seen, and feep if it doesn't get something recognizable as the rest.

Then, probably, create a shortcut along the lines of bindkey -m that sets
up self-insert-multibyte on the appropriate prefixes.

[*] Is there a plan yet for UTF-8 shell scripts, by the way?  That
can't be handled at the ZLE level.  What about zcompile?

} In addition to getkey() and friends, there is the related matter of the
} variable lastchar.  Currently this is a single character; I'm not yet
} 100% sure whether we can keep this, or promote it to a wchar_t, or
} whether we might need both types.  I fear it may be the last.

Not just lastchar, but also the KEYS parameter.  If wide chars are dealt
with as sequences at the widget binding level, but BUFFER contains the
corresponding wchars instead, then various currently-working tricks that
involve inserting all or part of KEYS into BUFFER will fail.  At least,
it becomes harder to emulate self-insert(-multibyte) in widget funcs.