From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 26542 invoked by alias); 23 Feb 2015 11:36:06 -0000 Mailing-List: contact zsh-workers-help@zsh.org; run by ezmlm Precedence: bulk X-No-Archive: yes List-Id: Zsh Workers List List-Post: List-Help: X-Seq: 34610 Received: (qmail 28005 invoked from network); 23 Feb 2015 11:35:54 -0000 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on f.primenet.com.au X-Spam-Level: X-Spam-Status: No, score=-2.7 required=5.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,RCVD_IN_DNSWL_LOW autolearn=ham version=3.3.2 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=Cw/3y0nYgRY859kOo5rNwEKCSDXv+VHWD02MUSYia9U=; b=JzH4wEO/0LgV9cSDHxthxuiVfjTUfGe+Ss60px0SsBRY2LDoTepkIwU2oaEbesn7U3 sWMwy8lMRTFem0MGzACxhXk68ZBKZl6MDO0D3marqJJ7omiBWIjZTf51ryyiWNHEQLGG kMZISiOVEa9OobzQ4uMS3X7RSMQLZKxFpZeAP4WYQEF03iO3w5fwZIZuLW2eeGLggEbn w8PbqGukmrFEEXBr41+ID5QyYOxhmCz9T0AJTKyMo/bQvhAH7aP19Gagqi6j4ks+9QKM GLpRP/QxUUD3b7nzIpaYSr3b6U6ht0W1z2Lsw89FIG/hKjOZ2OVvED102iVNoZWID8mp OQzQ== MIME-Version: 1.0 X-Received: by 10.42.249.2 with SMTP id mi2mr10883890icb.36.1424691351652; Mon, 23 Feb 2015 03:35:51 -0800 (PST) In-Reply-To: <20150223101157.08acf00c@pwslap01u.europe.root.pri> References: <20150219101315.477f7f95@pwslap01u.europe.root.pri> <20150219220311.7dfdc4ec@ntlworld.com> <20150220100006.24224469@pwslap01u.europe.root.pri> <20150222182619.1851e983@ntlworld.com> <20150223095443.028f0905@pwslap01u.europe.root.pri> <20150223101157.08acf00c@pwslap01u.europe.root.pri> Date: Mon, 23 Feb 2015 12:35:51 +0100 Message-ID: Subject: Re: PATCH: parse from even deeper in hell From: Mikael Magnusson To: Peter Stephenson Cc: "Zsh Hackers' List" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable > Apparently this message from Mikael didn't go to the list... I made a draft last night before going to bed, and decided I was too tired to send, apparently when I woke up I was still too tired to check if I had set reply-to-all. :) On Mon, Feb 23, 2015 at 11:11 AM, Peter Stephenson wrote: >> > Mikael Magnusson wrote: >> I actually thought the patch in 34587 had fixed it, turns out I just >> lost my --enable-zsh-debug flag. This patch doesn't fix it either >> though, and I get a different set of errors every time I open a >> terminal now, so that's fun. Which is to say it's all the same >> wordsplit error, but it's printed for a different set of lines. Some >> of them are the same though. > > Obviously, if you want to go down that route, you'll have to look at see > what's up with those lines. It seemed to work on the line you originall > gave. > >> I noticed another thing from these errors too, they're printed also >> when I exit zsh. There's not much point in lexing the history at >> write-out time, is there? > > I can't remember what that is, but I've noticed it before and I have a > vague feeling I looked and decided I wasn't going to look any more. Okay, that sounds ominous enough for me to drop the idea. >> I tried playing around with the code a bit. The thing that looks >> suspicious to me is >> if (*ptr !=3D Meta && imeta(*ptr)) { >> Shouldn't they check ptr[0] and ptr[1] or so? I tried this, >> if ((ptr =3D=3D lineptr || ptr[-1] !=3D Meta) && imeta(ptr[0])) { >> (in both places) but it didn't improve matters. > > No, the problem here isn't a standard one with Tok -> Meta + NonTok. > It's that there's something that looks like Tok but isn't. So we > need to turn it into Meta + NonTok. That's the second part of the > test. However imeta(*ptr) triggers if *ptr is Meta, which isn't what we > want because in that case it means we've hit a correct Meta + NonTok > sequence. > > One thing I didn't think I needed to do, but may have got wrong, > is skip the byte after the Meta, i.e. > > if (*ptr =3D=3D Meta) > ptr++; > > (which acts together with the existing increment). > > You could try that. > >> I tried the following instead of the for+if: >> unmetafy(lineptr, NULL); >> lineptr =3D metafy(lineptr, -1, META_USEHEAP); >> and it gets rid of the errors, but it also causes insert-last-word to >> do nothing. So maybe my whole theory is wrong. > > I'd be worried about doing that on every single line -- we already know > HIST_LEX_WORDS can be very slow, which is the only reason it's an option > (it's logically the right thing to do, modulo lex failures). > > However, it should be possible to combine that with the "remeta" check, > i.e. see if the line needs it first, but not use the value of remeta, > just whether it's non-zero, and then it becomes easy and not too > intrusive --- and also safe about false positives. > > I don't see why this would cause failures later on, particularly ones > apparently unrelated to the meta state of the string. I figured out that when we assign lineptr after fiddling with it, we also need to update start, it records the location of *lineptr on entry to the function, and is used to calculate things later on. With that addition, the unmetafy + metafy mostly works. insert-last-word still gets "stuck" on words that came from the old metafication and starts over from the end of history, leaving the old word in place. Indeed fc -l lists this line differently still, even though it appears correctly in zle. 10673* echo mp3info =E5=A5=BD=E3=81=8D=E3=81=AB=E3=81=AA=E3=82=8A\M-c\M-^A= =E3=81=84.mp3 10675* echo mp3info =E5=A5=BD=E3=81=8D=E3=81=AB=E3=81=AA=E3=82=8A=E3=81=9F= =E3=81=84.mp3 This is with the following code, which fixes(?) all the errors and seems to make things work pretty well. However, even doing the remeta unconditionally, I still get the above result with fc -l and insert-last-word. It also happens, unsurprisingly, if I don't use hist_lex_words. If I accept a line recalled in this way, the new history entry is saved correctly and works fine in new shells. for (ptr =3D lineptr; *ptr; ptr++) if (*ptr !=3D Meta && imeta(*ptr)) { remeta =3D 1; break; } if (remeta) { unmetafy(lineptr, &remeta); start =3D lineptr =3D metafy(lineptr, remeta, META_USEHEAP); } --=20 Mikael Magnusson