From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 20887 invoked by alias); 17 Sep 2012 19:04:40 -0000 Mailing-List: contact zsh-workers-help@zsh.org; run by ezmlm Precedence: bulk X-No-Archive: yes List-Id: Zsh Workers List List-Post: List-Help: X-Seq: 30673 Received: (qmail 27213 invoked from network); 17 Sep 2012 19:04:29 -0000 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on f.primenet.com.au X-Spam-Level: X-Spam-Status: No, score=-3.3 required=5.0 tests=BAYES_00,DKIM_ADSP_ALL, DKIM_SIGNED,RCVD_IN_DNSWL_MED,T_DKIM_INVALID,UNPARSEABLE_RELAY autolearn=ham version=3.3.2 Received-SPF: none (ns1.primenet.com.au: domain at spodhuis.org does not designate permitted sender hosts) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=spodhuis.org; s=d201107; h=In-Reply-To:Content-Type:MIME-Version:References:Message-ID:Subject:Cc:To:From:Date; bh=Is15KULo0ZdyvtvpXm9/m6lSXdFoVBfvi9CiXjnZJZU=; b=CJIBOgepw56FIMtmrVp+nxJLpzUsQvZJwKMqf7IKUSQKSbK92kuaE92zDAQd7xkkdnHpwMN4beP9twHqOj3PQl2X9rvubkIJ9TCWVqreiw9xK2LcSpbtLuPWwnG+dgl8tG07fVh30iGjhF110Z7kzgqt3vnpKqwPWYv6JNedZc0=; Date: Mon, 17 Sep 2012 15:04:23 -0400 From: Phil Pennock To: Peter Stephenson Cc: zsh-workers@zsh.org Subject: Re: PATCH: PCRE support for embedded NUL characters Message-ID: <20120917190422.GA41017@redoubt.spodhuis.org> Mail-Followup-To: Peter Stephenson , zsh-workers@zsh.org References: <20120916125015.GA87764@redoubt.spodhuis.org> <20120917095727.23896b8b@pwslap01u.europe.root.pri> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20120917095727.23896b8b@pwslap01u.europe.root.pri> On 2012-09-17 at 09:57 +0100, Peter Stephenson wrote: > You can just do a pre-scan of the whole string for backslashes. If > there's a backslash followed by a non-NULL, skip checking that next > character (which may itself be a backslash that's escaped); if there's a > backslash followed by a NULL the backslash can go. It's such an unusual > case it hardly seems worth it, though. I'm inclined to document it as a limitation, rather than add complexity, wherein will lurk bugs. > They are characters. If the string is unmetafied you can skip the > MB_METACHARLEN() stuff and use the mbrtowc()/WCWIDTH() library calls > directly (WCWIDTH() is only defined in order to be able to replace an > unusable wcwidth()), but a null probably needs to be a special case > since I think the libraries assume it's a terminator. It looks like the > existing pattern code uses metafied strings. Yeah, but correlating offsets in unmetafied strings to the metafied strings for then counting is non-trivial (or so it seems to me). And wcwidth() tells how many display cells are needed for a given character, assuming a monospace layout. For this, instead, mblen() is needed, on a character-by-character basis. Given that mblen() is C99, I opted to avoid it, and implement this just for UTF-8 with bit-pattern examination to quickly count past characters. We only initialise PCRE for wide characters with UTF-8. I've no idea how much effort we want to put into supporting non-UTF-8 wide-character PCRE across multiple OSes. Thoughts? (Code not finished yesterday, will be a little bit before I get back to it). -Phil