Re: PATCH: parse from even deeper in hell

[Peter Stephenson <p.stephenson@samsung.com>]

On Fri, 20 Feb 2015 04:43:49 +0100
Mikael Magnusson <mikachu@gmail.com> wrote:
> Oh I see, you renumbered a bunch of stuff in zsh.h, so text would be
> metafied differently now. But only metafy uses the table, unmetafy
> doesn't. That explains why the strings are different, but not why the
> old string causes an error. Unless we are parsing it before
> unmetafying it, which means any random bytes that weren't special
> before but are now would be interpreted specially.

I haven't confirmed the behaviour in detail, but I think what's
happening is the old string read in from history now looks like
an already metafied byte and when we scan the line gets unmetafied into
an invalid 8-bit character which is where things are falling over.

> Can we
> unmetafy+metafy the string before lexing? I guess that might be slower
> though. (Sorry for the 500 mails).

I believe any character read in from the history file that appears as
"needs metafication" is illegal, because we don't output tokenised
text.  So we simply metafy it at that point.  We can pre-scan for this
so we don't treat lines we don't need to it.

The question is where to put this in on history read.  I think it's
going to affect non-lexical history, too, but the error on reading won't
be flagged up.

pws

-- 
Peter Stephenson | Principal Engineer Samsung Cambridge Solution Centre
Email: p.stephenson@samsung.com | Phone: +44 1223 434724 |
www.samsung.com