Update _twisted completion

Update _twisted completion

[Eric P. Mangold]

Hi,

For a while now we've had a new version of the zsh tab-completion stub function. It should probably start being included in zsh, replacing the old deprecated stub function that lives at Completion/Unix/Command/_twisted.

Twisted ships a large number of commands, and e.g. `twistd' sub-commands are dynamically generated at runtime via plugin loading. As such, a static completion function is innappropriate for Twisted, and would be instantly out of date. So, the stub function just asks Twisted to send it a
completion function on stdout, which if well formed can be eval'd inside the stub function (_twisted) to produce results.

This is an improvement over the old (as exists in zsh tree, currently) stub function, which sought to locate a Twisted directory and refer to
a set of statically generated completion files that exist there (twisted/python/zsh/_* in our tree). These old stubs still exist in our tree for backward compatibility with the old stub function, and will continue to do so for a long time to come (though they are just wrappers for the new stub, now).

Please find the new stub function here, noting that it should be renamed _twisted to replace the existing one in zsh's tree,

http://twistedmatrix.com/trac/browser/trunk/twisted/python/twisted-completion.zsh

Thanks,
Eric Mangold
teratorn@twistedmatrix.com

Re: Update _twisted completion

[Phil Pennock]

On 2013-02-03 at 12:53 -0500, Eric P. Mangold wrote:
> Twisted ships a large number of commands, and e.g. `twistd' sub-commands are dynamically generated at runtime via plugin loading. As such, a static completion function is innappropriate for Twisted, and would be instantly out of date. So, the stub function just asks Twisted to send it a
> completion function on stdout, which if well formed can be eval'd inside the stub function (_twisted) to produce results.

Wait what, a function to be eval'd, so can do anything in the shell
environment, which runs because I press tab when seeing what options
this "command the user is not familiar with" might offer?

Is there no way to rationalize the options for expansion down to a
schema with "global options, sub-commands, sub-command options, tags for
known contexts" and have the _twisted function interpret those
accordingly?

There's a difference between running an unknown command and having your
shell execute, in its own context, any evaluable code supplied by that
command because the user hit _tab_, not _enter_.

-Phil

Re: Update _twisted completion

[Eric P. Mangold]

Hi Phil,

On Sun, Feb 03, 2013 at 11:26:09PM -0500, Phil Pennock wrote:
> On 2013-02-03 at 12:53 -0500, Eric P. Mangold wrote:
> > Twisted ships a large number of commands, and e.g. `twistd' sub-commands are dynamically generated at runtime via plugin loading. As such, a static completion function is innappropriate for Twisted, and would be instantly out of date. So, the stub function just asks Twisted to send it a
> > completion function on stdout, which if well formed can be eval'd inside the stub function (_twisted) to produce results.
> 
> Wait what, a function to be eval'd, so can do anything in the shell
> environment, which runs because I press tab when seeing what options
> this "command the user is not familiar with" might offer?

Well, this is what we do, and it's what we've been doing. Getting the code
dynamically over stdin isn't much different than locating arbitrary completion
functions inside the Twisted directory and executing them.

Other completions parse GNU --help output *shrug*. So I'm not really seeing
much of a difference here..

> Is there no way to rationalize the options for expansion down to a
> schema with "global options, sub-commands, sub-command options, tags for
> known contexts" and have the _twisted function interpret those
> accordingly?

There would be a way, with a lot of extra work, I suppose. Coming up with
a safe-data-only schema to represent completley arbitrary completion functionality
seems like a lot of work to me :)

If it were that easy, perhaps we would already have "completion descriptors"
instead of "completion functions" ;)

I think it's kind of a stretch to imagine a situation where the user is typing a
command-line and preparing to execute it, but hasn't decided to trust the
software in question yet.

Software often times will install its own zsh completion functions in to a
"site-functions" directory, and so arbitrary completion behavior (including
access to the user's shell environment) is already part of the game.

I am also planning on adding some caching to speed things up (it's a little
slow right now to load and import the various python modules and generate
the function), and storing cached data (as the current "old" completion stub
 does) in to the user's shell env is one option I am exploring. I guess the
other options is to write cache data to $HOME, somewhere.

> 
> There's a difference between running an unknown command and having your
> shell execute, in its own context, any evaluable code supplied by that
> command because the user hit _tab_, not _enter_.

Im glad my little innovations have raised these questions. I
would like very much for there to be a system whereby software authors
could more easily take charge of supplying their own shell-completion
functionality as part of the software distribution, instead of relying on
static functions shipped with zsh. I've seen several other projcts in the
past that ship/generate their own functions, and we could probably all
benefit from using an officially supported protocol.

Our ad-hoc "protocol" is to call the command-line and append
"--_shell-completion zsh:${CURRENT}" - this tells our options-parsing
subsystem (twisted.python.usage) to generate a completion function
on stdout appropriate for the given command-line.

I really do wish there was a better way for commands to supply completions
at run-time. More feedback would be much appreciated.

Cheers,
-E

Re: Update _twisted completion

[Peter Stephenson]

On Mon, 04 Feb 2013 11:48:44 -0500
"Eric P. Mangold" <eric@teratorn.org> wrote:
> Im glad my little innovations have raised these questions. I
> would like very much for there to be a system whereby software authors
> could more easily take charge of supplying their own shell-completion
> functionality as part of the software distribution, instead of relying on
> static functions shipped with zsh. I've seen several other projcts in the
> past that ship/generate their own functions, and we could probably all
> benefit from using an officially supported protocol.

It should be as easy as putting that function in
/usr/share/zsh/site-functions, or equivalent.  That's already in the
fpath ahead of the standard completion directory/directories, so will be
picked up by compinit, which already scans the entire fpath --- that
should be all you need.

There needs to be some interaction between the two utilities so
the utility can supply the information where zsh can find it.  I don't
think it's going to get much simpler than that.

The only difficulty would appear to be ensuring you've got the right
site-functions directory for the version or versions of zsh.  You could
do something like the following on installation:

for testdir in $path; do
  if [[ -x $testdir/zsh ]]; then
    sitefndir="$(zsh -cf 'print -l $fpath' | grep 'site-functions$' | head -1)"
    [[ -d $sitefndir ]] && cp my_completion_files $sitefndir
  fi
done

The alternative is to add some file to an initialisation directory that
sets the fpath for you early on (something run from /etc/zshrc would be
OK, something from /etc/zshenv would be good enough, something only from
/etc/zprofile wouldn't).  Distributions already have ways of doing this
and you probably wouldn't want to reinvent the wheel.

pws

Re: Update _twisted completion

[Phil Pennock]

On 2013-02-04 at 11:48 -0500, Eric P. Mangold wrote:
> Other completions parse GNU --help output *shrug*. So I'm not really seeing
> much of a difference here..

Pattern-matching on --help output and extracting strings to offer is
different from evaluating code which can modify the shell environment.
The child process can run anything itself, but that doesn't let it do
things like change $XAUTHORITY and $DISPLAY in the shell, to silently
repoint graphics interactions to a proxy, or set $http_proxy, or
anything else that affects launched processes.  Or changing $SSH_ASKPASS
to a wrapper which steals SSH key passphrases.

Software I really don't trust much gets run under passwordless sudo to
an account where it has no access to my main working account.  (Software
I don't trust at all obviously doesn't get run at all, or if I have no
choice but to run it, then in a VM).

Even without that caution, trust flows down the path from parent to
child processes.  Data coming back up the path should not become _code_
coming back up the path.

When eval happens, it needs to be explicit (eval `ssh-agent -s`) and not
happen because someone hit tab and didn't even ask for a command to be
run.

When both completion function and program ship as part of one project,
at least the completion function on disk can be read and audited and
people can tell what will run inside their shell.

When the code comes from a program, which may be a compiled binary, your
ability to audit and prove what you'll evaluate goes down significantly;
the output might change depending upon the day of the month.

-Phil

Re: Update _twisted completion

[Eric P. Mangold]

On Mon, Feb 04, 2013 at 12:31:56PM -0500, Phil Pennock wrote:
> On 2013-02-04 at 11:48 -0500, Eric P. Mangold wrote:
> > Other completions parse GNU --help output *shrug*. So I'm not really seeing
> > much of a difference here..
> 
> Pattern-matching on --help output and extracting strings to offer is
> different from evaluating code which can modify the shell environment.
> The child process can run anything itself, but that doesn't let it do
> things like change $XAUTHORITY and $DISPLAY in the shell, to silently
> repoint graphics interactions to a proxy, or set $http_proxy, or
> anything else that affects launched processes.  Or changing $SSH_ASKPASS
> to a wrapper which steals SSH key passphrases.

If you run a process via your shell, unless you have some kind of magic
sandbox I've never heard of, that process can stomp all over
your shell if it wants to.

If we're talking about malicious code, then you can't discount the power of
ptrace(2), /proc, etc.

> Software I really don't trust much gets run under passwordless sudo to
> an account where it has no access to my main working account.  (Software
> I don't trust at all obviously doesn't get run at all, or if I have no
> choice but to run it, then in a VM).
> 
> Even without that caution, trust flows down the path from parent to
> child processes.  Data coming back up the path should not become _code_
> coming back up the path.
> 
> When eval happens, it needs to be explicit (eval `ssh-agent -s`) and not
> happen because someone hit tab and didn't even ask for a command to be
> run.

Well, zsh ships our old stub, which loads arbitrary functions from the
Twisted directory and executes them.  Are you proposing to remove the
old stub? I'm not seeing any functional difference between the old and
new versions, from a security perspective :)
 
> When both completion function and program ship as part of one project,
> at least the completion function on disk can be read and audited and
> people can tell what will run inside their shell.

It's not different with Twisted's shell completion stub - it, and the python
code that gets executed, can be audited. And perhaps more easily
than if it were a huge convoluted shell script program.

If you upgrade your software, things could change - but that isn't really
any different than what we are used to.
 
> When the code comes from a program, which may be a compiled binary, your
> ability to audit and prove what you'll evaluate goes down significantly;
> the output might change depending upon the day of the month.

Yes - but that doesn't really apply exactly, to this particular function, as Twisted
doesn't include any binary programs, and all the source code is readily
available.

I mostly agree with everything you are saying - but I still think what we
are doing is acceptable, or at least *as acceptable* as what we've already
been doing. I'm very much open to doing things better, but I would need
some kind of concrete direction on what course to take. The problem of
allowing programs to dynamically supply their own, more-or-less arbitary,
completions in "safe" manner is certainly an interesting one.

The only mechanism provided by zsh is to "eval" the contents of a completion
function inside the user's shell, albeit in a "completion context". This works,
but provides for no safety between the consumer of completion functionality (zsh)
and the producer of completion functionality (in this case, a program).

It is already the case that programs, if "installed" can provide their own
completion functions by placing them in a "site-functions" directory.
Auditing these programs is still necessary as ever - if the program in question
delegates to some Python code to get its job done, I don't see that this
changes the fundamental nature of the problem...

If not "installed", then none of our Python code can hurt you. The completion
stub would fail because it would not find the command on the user's $PATH,
and the Python code in question could never be loaded because it doesn't
exist anywhere in sys.path.

And if you *have* "installed" Twisted, then presumably you already trust it, as
you've just contaminated your copy of Python with its modules.

Best regards,
-E

Re: Update _twisted completion

[Phil Pennock]

On 2013-02-04 at 14:01 -0500, Eric P. Mangold wrote:
> If you run a process via your shell, unless you have some kind of magic
> sandbox I've never heard of, that process can stomp all over
> your shell if it wants to.

Processes can't change environ of their parent; environ in proc, for
instance, is read-only.  If you can attach with ptrace, then yes.

You're right though, in that the security models of most operating
systems, including Unix, are pathetically inadequate for the modern
world; digression: this is because the only early funders of research
into computer security were branches of the US military, whose model
included users running programs decided upon and installed for them, and
needing to protect one user against another.  The models don't include
users not trusting code they themselves run.

We're finally starting to see progress here with stuff like Capsicum,
capability systems providing sandboxing.  That will take a while to help
by default.

In the meantime, I sudo and ensure stuff listening for network
connections from beyond localhost is not running as my working account
(and ensure I have packet filtering to enforce the strong end-system
host model, at least as regards localhost addresses, so that they can't
come over the wire).

> I mostly agree with everything you are saying - but I still think what we
> are doing is acceptable, or at least *as acceptable* as what we've already
> been doing. I'm very much open to doing things better, but I would need
> some kind of concrete direction on what course to take. The problem of
> allowing programs to dynamically supply their own, more-or-less arbitary,
> completions in "safe" manner is certainly an interesting one.

A former employer had standard options in binaries built against their
libraries, so that a certain command-line flag combination would emit
output in a form designed for shell parsing for completion, giving more
power than crude --help parsing.  I think I've seen this elsewhere, but
have been failing to recall where.

(That output was actually designed for use with bash, but I was able to
make it work with zsh easily enough).

> And if you *have* "installed" Twisted, then presumably you already trust it, as
> you've just contaminated your copy of Python with its modules.

VirtualEnv.

-Phil

Re: Update _twisted completion

[Bart Schaefer]

On Feb 4, 11:48am, Eric P. Mangold wrote:
}
} Well, this is what we do, and it's what we've been doing. Getting
} the code dynamically over stdin isn't much different than locating
} arbitrary completion functions inside the Twisted directory and
} executing them.

I have to agree with Phil here.  Completion functions in a directory can
be examined to increase confidence they're trustworthy, and are subject
to normal runtime security checks.  Eval'in a string provided on stdout
leaves the shell open to additional attacks.
 
} If it were that easy, perhaps we would already have "completion
} descriptors" instead of "completion functions" ;)

To a fair extent, we do.  That's what the parameters of _arguments are,
after all.

Re: Update _twisted completion

[Oliver Kiddle]

Phil Pennock wrote:
> Wait what, a function to be eval'd, so can do anything in the shell
> environment, which runs because I press tab when seeing what options
> this "command the user is not familiar with" might offer?

I don't think it is much worse than what the current _twisted does.
And in the end you have little choice than to trust all the software
you install and use on your system.

It'd be better if the packaging for Linux distributions (or BSD, opencsw
etc) coordinated things so that completion functions that aren't
distributed with zsh can go in somewhere suitable
(/usr/share/zsh/site-functions or /etc/zsh_completion.d) and then
get checked by compaudit.

I'm not especially familiar with twisted but in general, if the
completion function is going to be calling binaries to generate matches,
it is better to produce lists of things like sub-commands, options, etc
than a full completion function. These can then be used by any shell.

Another comment on _twisted is that using echo for error messages in
completion functions tends not to work too well because it mucks up
the line editor. `zle -M' or simply printing nothing (but generating no
matches) work better.

Oliver