getopts doesn't update OPTIND when called from function

getopts doesn't update OPTIND when called from function

[Martijn Dekker]

I'm writing a shell function that extends the functionality of the
'getopts' builtin. For that to work, it is necessary to call the
'getopts' builtin from the shell function.

The POSIX standard specifies that OPTIND and OPTARG are global
variables, even if the positional parameters are local to the
function.[*] This makes it possible to call 'getopts' from a function by
simply passing the global positional parameters along by adding "$@".

My problem is that zsh does not update the global OPTIND variable when
getopts is called from a function, which defeats my function on zsh. (It
does update the global OPTARG variable, though.)

So, this is another bug report.

I made a little test program that demonstrates this; see below the
footnotes. It succeeds on bash, ksh93, pdksh, mksh, and yash, but not
zsh or (d)ash[*2].

Thanks,

- Martijn

[*] The POSIX standard specifies:
http://pubs.opengroup.org/onlinepubs/9699919799/utilities/getopts.html
"The shell variable specified by the name operand, OPTIND, and OPTARG
shall affect the current shell execution environment", which implies
that they are global variables.
    Confusingly, that same page also says: "The shell variables OPTIND
and OPTARG shall be local to the caller of getopts and shall not be
exported by default."
    But I believe that "caller" here means the program that calls
getopts, not the function; POSIX does not support function-local
variables. This interpretation is supported by the added phrase "... and
shall not be exported by default" and by the evidence that the majority
of popular shells pass my test script.
    (Of course it should be possible to explicitly make OPTIND and
OPTARG local using the non-standard 'local' or 'typeset' keyword, but I
believe they should be global by default.)

[*2] (d)ash has a hairier bug where, if getopts is called from a
function, it updates the global OPTIND for the first option but not for
subsequent options, so OPTIND gets stuck on 3. I shall be bothering
their developers separately.

#### begin test script ####

#! /bin/sh

expect() {
    if [ "X$2" = "X$3" ]; then
        printf '%s: OK, got "%s"\n' "$1" "$2"
    else
        printf '%s: BUG: expected "%s", got "%s"\n' "$1" "$2" "$3"
        return 1
    fi
}

callgetopts() {
    getopts 'D:ln:vhL' opt "$@"
}

testfn() {
    expect OPTIND 1 "$OPTIND"

    callgetopts "$@"
    expect opt D "$opt"
    expect OPTARG 'test' "$OPTARG"

    callgetopts "$@"
    expect opt h "$opt"
    expect OPTARG '' "$OPTARG"

    callgetopts "$@"
    expect OPTIND 5 "$OPTIND"
    expect opt n "$opt"
    expect OPTARG 1 "$OPTARG"

    callgetopts "$@"
    expect OPTIND 5 "$OPTIND"

    callgetopts "$@"
    expect OPTIND 5 "$OPTIND"
}

testfn -D test -hn 1 test arguments

#### end test script ####

Output on zsh 5.0.7-dev-4:

OPTIND: OK, got "1"
opt: OK, got "D"
OPTARG: OK, got "test"
opt: BUG: expected "h", got "D"
OPTARG: BUG: expected "", got "test"
OPTIND: BUG: expected "5", got "1"
opt: BUG: expected "n", got "D"
OPTARG: BUG: expected "1", got "test"
OPTIND: BUG: expected "5", got "1"
OPTIND: BUG: expected "5", got "1"

Expected output (on bash, *ksh*, yash):

OPTIND: OK, got "1"
opt: OK, got "D"
OPTARG: OK, got "test"
opt: OK, got "h"
OPTARG: OK, got ""
OPTIND: OK, got "5"
opt: OK, got "n"
OPTARG: OK, got "1"
OPTIND: OK, got "5"
OPTIND: OK, got "5"

Re: getopts doesn't update OPTIND when called from function

[Peter Stephenson]

On Thu, 28 May 2015 18:18:32 +0200
Martijn Dekker <martijn@inlv.org> wrote:
> I'm writing a shell function that extends the functionality of the
> 'getopts' builtin. For that to work, it is necessary to call the
> 'getopts' builtin from the shell function.
> 
> The POSIX standard specifies that OPTIND and OPTARG are global
> variables, even if the positional parameters are local to the
> function.[*] This makes it possible to call 'getopts' from a function by
> simply passing the global positional parameters along by adding "$@".
> 
> My problem is that zsh does not update the global OPTIND variable when
> getopts is called from a function, which defeats my function on zsh. (It
> does update the global OPTARG variable, though.)

This is documented behaviour (well, sort of -- the documentation didn't
say quite what we actually do) so this needs another compatibility fix.
POSIX_BUILTINS seems appropriate.

diff --git a/Doc/Zsh/builtins.yo b/Doc/Zsh/builtins.yo
index a2c03bc..f8e4edf 100644
--- a/Doc/Zsh/builtins.yo
+++ b/Doc/Zsh/builtins.yo
@@ -866,7 +866,8 @@ vindex(OPTARG, use of)
 
 The first option to be examined may be changed by explicitly assigning
 to tt(OPTIND).  tt(OPTIND) has an initial value of tt(1), and is
-normally reset to tt(1) upon exit from a shell function.  tt(OPTARG)
+normally set to tt(1) upon entry to a shell function and restored
+upon exit (this is disabled by the tt(POSIX_BUILTINS) option).  tt(OPTARG)
 is not reset and retains its value from the most recent call to
 tt(getopts).  If either of tt(OPTIND) or tt(OPTARG) is explicitly
 unset, it remains unset, and the index or option argument is not
diff --git a/Doc/Zsh/options.yo b/Doc/Zsh/options.yo
index 032423d..4c0ae12 100644
--- a/Doc/Zsh/options.yo
+++ b/Doc/Zsh/options.yo
@@ -2038,6 +2038,10 @@ tt(unset).
 In addition, various error conditions associated with the above builtins
 or tt(exec) cause a non-interactive shell to exit and an interactive
 shell to return to its top-level processing.
+
+Furthermore, the tt(getopts) builtin behaves in a POSIX-compatible
+fashion in that the associated variable tt(OPTIND) is not made
+local to functions.
 )
 pindex(POSIX_IDENTIFIERS)
 pindex(NO_POSIX_IDENTIFIERS)
diff --git a/Src/builtin.c b/Src/builtin.c
index a9afb45..18c22f2 100644
--- a/Src/builtin.c
+++ b/Src/builtin.c
@@ -4713,6 +4713,10 @@ bin_shift(char *name, char **argv, Options ops, UNUSED(int func))
     return ret;
 }
 
+/*
+ * Position of getopts option within OPTIND argument with multiple options.
+ */
+
 /**/
 int optcind;
 
diff --git a/Src/exec.c b/Src/exec.c
index 527d611..9f163a6 100644
--- a/Src/exec.c
+++ b/Src/exec.c
@@ -4910,9 +4910,11 @@ doshfunc(Shfunc shfunc, LinkList doshargs, int noreturnval)
     if (!(flags & PM_UNDEFINED))
 	scriptname = dupstring(name);
     oldzoptind = zoptind;
-    zoptind = 1;
     oldoptcind = optcind;
-    optcind = 0;
+    if (!isset(POSIXBUILTINS)) {
+	zoptind = 1;
+	optcind = 0;
+    }
 
     /* We need to save the current options even if LOCALOPTIONS is *
      * not currently set.  That's because if it gets set in the    *
@@ -5049,8 +5051,10 @@ doshfunc(Shfunc shfunc, LinkList doshargs, int noreturnval)
 	argzero = oargv0;
     }
     pparams = pptab;
-    optcind = oldoptcind;
-    zoptind = oldzoptind;
+    if (!isset(POSIXBUILTINS)) {
+	zoptind = oldzoptind;
+	optcind = oldoptcind;
+    }
     scriptname = oldscriptname;
     oflags = ooflags;
 

Re: getopts doesn't update OPTIND when called from function

[Roman Neuhauser]

# p.stephenson@samsung.com / 2015-05-28 18:17:40 +0100:
> On Thu, 28 May 2015 18:18:32 +0200
> Martijn Dekker <martijn@inlv.org> wrote:
> > I'm writing a shell function that extends the functionality of the
> > 'getopts' builtin. For that to work, it is necessary to call the
> > 'getopts' builtin from the shell function.
> > 
> > The POSIX standard specifies that OPTIND and OPTARG are global
> > variables, even if the positional parameters are local to the
> > function.[*] This makes it possible to call 'getopts' from a function by
> > simply passing the global positional parameters along by adding "$@".
> > 
> > My problem is that zsh does not update the global OPTIND variable when
> > getopts is called from a function, which defeats my function on zsh. (It
> > does update the global OPTARG variable, though.)
> 
> This is documented behaviour (well, sort of -- the documentation didn't
> say quite what we actually do) so this needs another compatibility fix.
> POSIX_BUILTINS seems appropriate.

thanks for the zshbuiltins(1) clarification.  

i have a problem similar to Martijn's, only there's no function in sight.
this diff shows how zsh (5.0.7, emulate -R sh) differs from other shells,
and the standard IIUC:

   $ getopts abc OPTNAM -a -b -- -c
   $ printf "%s=%s\n" OPTIND "$OPTIND" OPTNAM "$OPTNAM" OPTARG "$OPTARG"
-  OPTIND=2
+  OPTIND=1
   OPTNAM=a
   OPTARG=
   $ getopts abc OPTNAM -a -b -- -c
   $ printf "%s=%s\n" OPTIND "$OPTIND" OPTNAM "$OPTNAM" OPTARG "$OPTARG"
-  OPTIND=3
+  OPTIND=2
   OPTNAM=b
   OPTARG=
   $ getopts abc OPTNAM -a -b -- -c
   [1]
   $ printf "%s=%s\n" OPTIND "$OPTIND" OPTNAM "$OPTNAM" OPTARG "$OPTARG"
   OPTIND=4
-  OPTNAM=?
+  OPTNAM=b
   OPTARG=

it would be nice to have zsh aligned with other shells, at least under
posix_builtins or 'emulate sh'.

-- 
roman

Re: getopts doesn't update OPTIND when called from function

[Bart Schaefer]

On Jun 5,  8:46am, Roman Neuhauser wrote:
} Subject: Re: getopts doesn't update OPTIND when called from function
}
} i have a problem similar to Martijn's, only there's no function in sight.
} this diff shows how zsh (5.0.7, emulate -R sh) differs from other shells,
} and the standard IIUC:

I believe this was previously discussed here:

  http://www.zsh.org/mla/workers//2015/msg00196.html

The key text:

> In zsh, if you add a SECOND call to getopts in the example function,
> you will find that the second call DOES exit with a value of 1, and
> that WHEN zsh getopts exits with a return value greater than zero,
> THEN the value of OPTIND is set as POSIX specifies.

No one responded with an opinion of whether this conforms to the literal
specification or not.

-- 
Barton E. Schaefer

Re: getopts doesn't update OPTIND when called from function

[Roman Neuhauser]

# schaefer@brasslantern.com / 2015-06-06 09:46:41 -0700:
> On Jun 5,  8:46am, Roman Neuhauser wrote:
> } this diff shows how zsh (5.0.7, emulate -R sh) differs from other shells,
> } and the standard IIUC:
> 
> I believe this was previously discussed here:
> 
>   http://www.zsh.org/mla/workers//2015/msg00196.html
> 
> The key text:
> 
> > In zsh, if you add a SECOND call to getopts in the example function,
> > you will find that the second call DOES exit with a value of 1, and
> > that WHEN zsh getopts exits with a return value greater than zero,
> > THEN the value of OPTIND is set as POSIX specifies.
> 
> No one responded with an opinion of whether this conforms to the literal
> specification or not.

yeah, sorry.  i missed that conversation.

anyway, it seems perfectly unambiguous to me.  the spec[*] says

> Each time it is invoked, the getopts utility shall place the value of
> the next option in the shell variable specified by the name operand
> and the index of the next argument to be processed in the shell
> variable OPTIND.

*Each time* it is invoked ... shall place ... the index of *the next*
argument ... in ... OPTIND.

in zsh, OPTIND has the index of the last processed option instead, until
"the end of options is encountered".  so that's one clear violation of
the spec.

> When the end of options is encountered, the getopts utility shall exit
> with a return value greater than zero; the shell variable OPTIND shall
> be set to the index of the first operand, or the value "$#" +1 if
> there are no operands; the name variable shall be set to the
> <question-mark> character.

When the end of options is encountered ... the name variable shall be
set to the <question-mark> character.

in zsh, the name parameter has the value of the last processed option
instead.  that's another clear violation.

[*] http://pubs.opengroup.org/onlinepubs/9699919799/utilities/getopts.html

-- 
roman