From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (qmail 18028 invoked by alias); 10 Dec 2015 05:00:08 -0000 Mailing-List: contact zsh-workers-help@zsh.org; run by ezmlm Precedence: bulk X-No-Archive: yes List-Id: Zsh Workers List List-Post: List-Help: X-Seq: 37372 Received: (qmail 17012 invoked from network); 10 Dec 2015 05:00:06 -0000 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on f.primenet.com.au X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM, HTML_MESSAGE,T_DKIM_INVALID autolearn=ham autolearn_force=no version=3.4.0 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=WO5QshlUjCXNPbHYdEK8ft2ZiMeNGsQVaTynLw7qaIA=; b=QZJ2J11AEMjgLZ6pPGom4xBNO4xAI5OsK0NHAVXtRVq+ndJ9cDsV57oGxT9KpyqZYA aOP0StcCa6VekjZR+ZCC497sDi9PTCfyoKc3A/c5eo3NOOG0W2k3Ok4xi+qf+RfAGEma U5lhXca1RusJK6GO+T8cQw++2q9jPFod+cohvN7T3GsbTQSnYpV5M27mMbnrwWFAbA3i woqX+TYlQF9fQ7jpkUWRdgaXIDyaHoaVy/aYx0PsY0Q+/yeyNIC3tAaAiPIf7u2WOHps IPoHB447ZJ5f9Y5jj+mr4oYla3EZ9/iJGkuwpSln/Ly6HT++XYCQL+Kt1tRuUSF53Jri Q1mg== MIME-Version: 1.0 X-Received: by 10.107.30.75 with SMTP id e72mr9668074ioe.5.1449723603557; Wed, 09 Dec 2015 21:00:03 -0800 (PST) In-Reply-To: <1926681449721747@web1m.yandex.ru> References: <2007121449719799@web8h.yandex.ru> <1926681449721747@web1m.yandex.ru> Date: Thu, 10 Dec 2015 15:30:03 +1030 Message-ID: Subject: Re: expr length "$val" returns the wrong length for values containing NULL (\\0) From: D Gowers To: "Nikolay Aleksandrovich Pavlov (ZyX)" Cc: "zsh-workers@zsh.org" Content-Type: multipart/alternative; boundary=001a1141977e69a83805268414a7 --001a1141977e69a83805268414a7 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I am aware of the prevalence of NUL-terminated strings, since I've coded in C in the past, that's why I wrote 'considerable bother to fix it'. Nevertheless, for a purpose such as argument passing, size + data is clearly better (easier to secure and more flexible) On Thu, Dec 10, 2015 at 2:59 PM, Nikolay Aleksandrovich Pavlov (ZyX) < kp-pav@yandex.ru> wrote: > 10.12.2015, 07:18, "D Gowers" : > > Ah, okay. That (commandline arguments not being able to contain NUL) > seems.. a bit anachronistic. But I guess it's never been enough of a > problem to warrant the considerable bother to fix it. Fair enough. > > This has nothing to do with the commandline itself. In some very earlier > days it was decided that strings will be NUL-terminated (in place of e.g. > being structs with size_t size and char *data) and this statement sneaked > into many parts of many standards. If you write C code you will have > problems when dealing with NUL-terminated string because every library > function that accepts something other then void* pointer with =E2=80=9Cge= neric > data=E2=80=9D assumes that string should terminate with NUL. Projects lik= e zsh or > almost every programming language have to write their own string > implementations: in zsh it is C strings with escaped characters, in most > other cases it is length+data pair. > > Since one of the functions having NUL convention is exec* function family > which is used to launch programs and another is main() function on the > other side that accepts NUL-terminated strings you cannot really do > anything to fix this: replacing one of the core conventions is *very* > expensive, especially since you must do this in a backward-compatible way= . > > > On Thu, Dec 10, 2015 at 2:26 PM, Nikolay Aleksandrovich Pavlov (ZyX) < > kp-pav@yandex.ru> wrote: > >> 10.12.2015, 04:52, "D Gowers" : > >>> Test case: > >>> > >>> v=3D$(printf foo\\0bar);expr length "$v";expr length $v > >>> > >>> alternatively: > >>> > >>> v=3Dfoo$'\0'bar;expr length "$v";expr length $v > >>> > >>> In zsh, the values returned are 3 and 3. > >>> In dash and zsh, the values returned are 6 and 6. > >>> > >>> Both of those results are wrong, AFAICS (foo$'0'bar is 7 characters > long). > >>> But the zsh result is more severely wrong. I could understand the > bash/dash > >>> result, at least, as 'NULL characters are not counted towards length'= . > >> > >> Both results are *right*. In both cases you ask the length of the > string and you get it. > >> > >> In dash (also posh, bash and busybox ash) zero byte is skipped when > storing. So length of the $v *is* six. You may question whether it is rig= ht > storing without zero byte, but the fact that all four shells have exactly > the same behaviour makes me think this is part of the POSIX standard. In > any case non-C strings are not on the list of features of these shells > unlike zsh (it also internally uses C NUL-terminated strings, but zero > bytes and some other characters are =E2=80=9Cmetafied=E2=80=9D (i.e. esca= ped) and > unmetafied when passed to the outer world e.g. by doing `echo $v` to pass > string to terminal). > >> > >> As I said in zsh zero byte is stored. But C strings which are the only > ones that can be arguments to any program are **NUL-terminated**. So what > you do is passing string "foo" because NUL terminates the string. You > cannot possibly get the answer you think is right here thus, unless you > reimplement `expr` as a zsh function. > >> > >>> > >>> In any case, it is easily demonstrated that the string is not 3 > characters > >>> long, by running 'echo "$V"' or 'print "$v"' or 'echo ${#v}' > >>> > >>> `zsh --version` =3D 'zsh 5.2 (x86_64-unknown-linux-gnu)' > --001a1141977e69a83805268414a7--