From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 7821 invoked from network); 27 Sep 2006 16:52:01 -0000 X-Spam-Checker-Version: SpamAssassin 3.1.5 (2006-08-29) on f.primenet.com.au X-Spam-Level: X-Spam-Status: No, score=-2.4 required=5.0 tests=AWL,BAYES_00, FORGED_RCVD_HELO autolearn=ham version=3.1.5 Received: from news.dotsrc.org (HELO a.mx.sunsite.dk) (130.225.247.88) by ns1.primenet.com.au with SMTP; 27 Sep 2006 16:52:01 -0000 Received-SPF: none (ns1.primenet.com.au: domain at sunsite.dk does not designate permitted sender hosts) Received: (qmail 95332 invoked from network); 27 Sep 2006 16:51:55 -0000 Received: from sunsite.dk (130.225.247.90) by a.mx.sunsite.dk with SMTP; 27 Sep 2006 16:51:55 -0000 Received: (qmail 10995 invoked by alias); 27 Sep 2006 16:51:53 -0000 Mailing-List: contact zsh-workers-help@sunsite.dk; run by ezmlm Precedence: bulk X-No-Archive: yes X-Seq: 22781 Received: (qmail 10986 invoked from network); 27 Sep 2006 16:51:52 -0000 Received: from news.dotsrc.org (HELO a.mx.sunsite.dk) (130.225.247.88) by sunsite.dk with SMTP; 27 Sep 2006 16:51:52 -0000 Received: (qmail 94998 invoked from network); 27 Sep 2006 16:51:52 -0000 Received: from cluster-c.mailcontrol.com (168.143.177.190) by a.mx.sunsite.dk with SMTP; 27 Sep 2006 16:51:50 -0000 Received: from cameurexb01.EUROPE.ROOT.PRI ([62.189.241.200]) by rly25c.srv.mailcontrol.com (MailControl) with ESMTP id k8RGpObk028997 for ; Wed, 27 Sep 2006 17:51:25 +0100 Received: from news01.csr.com ([10.103.143.38]) by cameurexb01.EUROPE.ROOT.PRI with Microsoft SMTPSVC(6.0.3790.1830); Wed, 27 Sep 2006 17:51:23 +0100 Received: from news01.csr.com (localhost.localdomain [127.0.0.1]) by news01.csr.com (8.13.7/8.13.4) with ESMTP id k8RGpNiJ031440 for ; Wed, 27 Sep 2006 17:51:23 +0100 Received: from csr.com (pws@localhost) by news01.csr.com (8.13.7/8.13.7/Submit) with ESMTP id k8RGpNxB031437 for ; Wed, 27 Sep 2006 17:51:23 +0100 Message-Id: <200609271651.k8RGpNxB031437@news01.csr.com> X-Authentication-Warning: news01.csr.com: pws owned process doing -bs To: zsh-workers@sunsite.dk Subject: Re: mb_metacharlenconv vs. tokens In-reply-to: <200609272031.35901.arvidjaar@newmail.ru> References: <200609252146.00249.arvidjaar@newmail.ru> <200609262203.41210.arvidjaar@newmail.ru> <200609261810.k8QIAJ7a003189@news01.csr.com> <200609272031.35901.arvidjaar@newmail.ru> Comments: In-reply-to Andrey Borzenkov message dated "Wed, 27 Sep 2006 20:31:35 +0400." Date: Wed, 27 Sep 2006 17:51:23 +0100 From: Peter Stephenson X-OriginalArrivalTime: 27 Sep 2006 16:51:23.0762 (UTC) FILETIME=[2A5F3D20:01C6E255] Content-Type: text/plain MIME-Version: 1.0 X-Scanned-By: MailControl A-07-04-02 (www.mailcontrol.com) on 10.67.0.135 Andrey Borzenkov wrote: > I am afraid that mb_metastrlenconv is rather overloaded. As it looks like > traversing string character by character is valid operation for input as > well, so we cannot exclude tokens there. > > Let's put it differently - what we intend is to avoid passing bogus character > to mbrtowc(). If we *know* the context is tokenized we could just as well > pass a flag to itype_end() and mb_metacharlenconv() so they will check for > tokens and skip them. Does it actually make sense? Yes, that makes perfect sense. It's then a case of deciding on the context, but that's a lot less heavyweight than untokenizing. -- Peter Stephenson Software Engineer CSR PLC, Churchill House, Cambridge Business Park, Cowley Road Cambridge, CB4 0WZ, UK Tel: +44 (0)1223 692070 To access the latest news from CSR copy this link into a web browser: http://www.csr.com/email_sig.php