realpath(3), symlinks, '..' components, and the ':A' word modifier

realpath(3), symlinks, '..' components, and the ':A' word modifier

[Daniel Shahaf]

Feedback is sought for a proposed behaviour change to the shell.

Currently, the ':a' word modifier removes '..' component from a path —
using a purely syntactic transformation, i.e., without consulting the
filesystem at all — and ':A' does the same and then resolves symlinks
[so no path component in the result is a symlink].

It has been proposed to change the semantics of :A to resolve symlinks
first and '..' components second, like the realpath(3) library function
does.

Under the incumbent semantics, $foo:A denotes the same file as $foo:a
(but not necessarily the same file as $foo).  Under the proposed
semantics, $foo:A denotes the same file as $foo (but not necesarily the
same file as $foo:a).

Would this change be a good idea?

----- Forwarded message from Daniel Shahaf <d.s@daniel.shahaf.name> -----

> Date: Fri, 10 Jun 2016 17:36:23 +0000
> From: Daniel Shahaf <d.s@daniel.shahaf.name>
> To: zsh-workers@zsh.org
> Subject: [PATCH 2/3] Fix the ':A' word modifier on paths with '..' components.
> Message-ID: <1465580184-3095-2-git-send-email-danielsh@tarsus.local2>
> X-Seq: 38650
> 
> This is an incompatible change; see the test and docs changes for details.
> 
> Daniel
> 
>  Doc/Zsh/expn.yo   |  6 ++++--
>  README            |  9 +++++++++
>  Src/hist.c        | 22 ++++++++++------------
>  Test/D02glob.ztst |  6 ++++++
>  4 files changed, 29 insertions(+), 14 deletions(-)
> 
> diff --git a/Doc/Zsh/expn.yo b/Doc/Zsh/expn.yo
> index c6e7b6f..50b8479 100644
> --- a/Doc/Zsh/expn.yo
> +++ b/Doc/Zsh/expn.yo
> @@ -225,9 +225,11 @@ intervening directories do not exist.
>  )
>  item(tt(A))(
>  As `tt(a)', but also resolve use of symbolic links where possible.
> -Note that resolution of `tt(..)' occurs em(before) resolution of symbolic
> -links.  This call is equivalent to tt(a) unless your system has the
> +This call is equivalent to tt(a) unless your system has the
>  tt(realpath) system call (modern systems do).
> +
> +em(Note): In zsh 5.2 and earlier, resolution of `tt(..)' occurred em(before)
> +resolution of symbolic links.
>  )
>  item(tt(c))(
>  Resolve a command name into an absolute path by searching the command
> diff --git a/README b/README
> index d5343db..84bb6bc 100644
> --- a/README
> +++ b/README
> @@ -79,6 +79,15 @@ Other aspects of EXIT trap handling have not changed --- there is still
>  only one EXIT trap at any point in a programme, so it is not generally
>  useful to combine POSIX and non-POSIX behaviour in the same script.
>  
> +4) On systems that have the realpath(3) library function, the ':A' word
> +modifier now resolves symbolic links before '..' path components.  This
> +could lead to different, but usually more desirable, results: the
> +tranformed value will now always identify the same directory entry as the
> +the pre-transformation value.
> +
> +The behaviour of 5.2 and older can be achieved by chaining modifiers:
> +'<expression>:a:A'.
> +
>  Incompatibilities between 5.0.8 and 5.2
>  ---------------------------------------
>  
> diff --git a/Test/D02glob.ztst b/Test/D02glob.ztst
> index 8618378..dc1a655 100644
> --- a/Test/D02glob.ztst
> +++ b/Test/D02glob.ztst
> @@ -670,3 +670,9 @@
>   () { set -- ${PWD}/$^@; print -l -- $@:A } glob.tmp/nonexistent/foo/bar/baz
>  0:modifier ':A' doesn't require existence
>  *>*/glob.tmp/nonexistent/foo/bar/baz
> +
> + ln -s dir3/subdir glob.tmp/link
> + () { print ${1:A} } glob.tmp/link/../../hello
> + rm glob.tmp/link
> +0:modifier ':A' resolves symlinks before '..' components
> +*>*glob.tmp/hello
> 

----- End forwarded message -----

Re: realpath(3), symlinks, '..' components, and the ':A' word modifier

[Roland Eggner]

[-- Attachment #1: Type: text/plain, Size: 1742 bytes --]

On 2016-07-05 Tuesday at 04:57 +0000 Daniel Shahaf wrote:
> Feedback is sought for a proposed behaviour change to the shell.
> 
> Currently, the ':a' word modifier removes '..' component from a path —
> using a purely syntactic transformation, i.e., without consulting the
> filesystem at all — and ':A' does the same and then resolves symlinks
> [so no path component in the result is a symlink].
> 
> It has been proposed to change the semantics of :A to resolve symlinks
> first and '..' components second, like the realpath(3) library function
> does.
> 
> Under the incumbent semantics, $foo:A denotes the same file as $foo:a
> (but not necessarily the same file as $foo).  Under the proposed
> semantics, $foo:A denotes the same file as $foo (but not necesarily the
> same file as $foo:a).
> 
> Would this change be a good idea?
> 

If voting for two different solutions is permitted,
I would vote for either (1) or (2):


(1)
(1.1)  We have already modifier “:a” for “..” removal.
(1.2)  Introduce a new modifier just for resolving
       filereferences (“symlinks”).
By splitting the two actions of modifier “:A” we could have both,
(1.2) after or before (1.1), both with the performance advantage over
external “realpath”.  The need to think about the desired order would
automatically comply with the goal of “least surprise”.

(2)    Leave it as is, keep backward compatibility.
The inconsistency with “realpath” can be viewed as advantage:
Both (1.2) after or before (1.1) is possible, the former by using “:A”,
the latter by using external “realpath”.



Thanks for taking care and for this democratic approach.
-- 
Best regards
Roland Eggner

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 213 bytes --]

Re: realpath(3), symlinks, '..' components, and the ':A' word modifier

[Vadim Zeitlin]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 1819 bytes --]

On Tue, 05 Jul 2016 04:57:56 +0000 Daniel Shahaf <d.s@daniel.shahaf.name> wrote:

DS> Feedback is sought for a proposed behaviour change to the shell.
DS> 
DS> Currently, the ':a' word modifier removes '..' component from a path —
DS> using a purely syntactic transformation, i.e., without consulting the
DS> filesystem at all — and ':A' does the same and then resolves symlinks
DS> [so no path component in the result is a symlink].
DS> 
DS> It has been proposed to change the semantics of :A to resolve symlinks
DS> first and '..' components second, like the realpath(3) library function
DS> does.
DS> 
DS> Under the incumbent semantics, $foo:A denotes the same file as $foo:a
DS> (but not necessarily the same file as $foo).  Under the proposed
DS> semantics, $foo:A denotes the same file as $foo (but not necesarily the
DS> same file as $foo:a).
DS> 
DS> Would this change be a good idea?

 Hello,

 I am not sure why would this be a good idea, the only argument for it I
see is compatibility with realpath(), but how much does it really matter?
There would seem to be quite a few flags/modifiers not corresponding to any
C library functions, so this doesn't seem like such an egregious exception.

 But this change would be backwards incompatible, if only marginally, and
my personal test for making breaking changes is whether I could see myself
justifying them reasonably well to someone whose script has got broken
after updating the shell. If you imagine yourself in such a situation, what
would your explanation be? I don't think that "we decided to make it
compatible with realpath(3)" quite cuts it. But this is just my personal
opinion, of course (and, FWIW, I don't think I personally have any script
which could be broken by this change anyhow).

 Regards,
VZ

[-- Attachment #2: Type: APPLICATION/PGP-SIGNATURE, Size: 196 bytes --]

Re: realpath(3), symlinks, '..' components, and the ':A' word modifier

[Filipe]

having a script broken by upgrading is such a painful, awful experience.

Couldn't we create another modifier that behaves like realpath? e.g. (:Ar) or even add an option to trigger this realpath compatible behaviour? 

> On 5 de jul de 2016, at 10:24, Vadim Zeitlin <vz-zsh@zeitlins.org> wrote:
> 
> On Tue, 05 Jul 2016 04:57:56 +0000 Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
> 
> DS> Feedback is sought for a proposed behaviour change to the shell.
> DS> 
> DS> Currently, the ':a' word modifier removes '..' component from a path —
> DS> using a purely syntactic transformation, i.e., without consulting the
> DS> filesystem at all — and ':A' does the same and then resolves symlinks
> DS> [so no path component in the result is a symlink].
> DS> 
> DS> It has been proposed to change the semantics of :A to resolve symlinks
> DS> first and '..' components second, like the realpath(3) library function
> DS> does.
> DS> 
> DS> Under the incumbent semantics, $foo:A denotes the same file as $foo:a
> DS> (but not necessarily the same file as $foo).  Under the proposed
> DS> semantics, $foo:A denotes the same file as $foo (but not necesarily the
> DS> same file as $foo:a).
> DS> 
> DS> Would this change be a good idea?
> 
> Hello,
> 
> I am not sure why would this be a good idea, the only argument for it I
> see is compatibility with realpath(), but how much does it really matter?
> There would seem to be quite a few flags/modifiers not corresponding to any
> C library functions, so this doesn't seem like such an egregious exception.
> 
> But this change would be backwards incompatible, if only marginally, and
> my personal test for making breaking changes is whether I could see myself
> justifying them reasonably well to someone whose script has got broken
> after updating the shell. If you imagine yourself in such a situation, what
> would your explanation be? I don't think that "we decided to make it
> compatible with realpath(3)" quite cuts it. But this is just my personal
> opinion, of course (and, FWIW, I don't think I personally have any script
> which could be broken by this change anyhow).
> 
> Regards,
> VZ

Re: realpath(3), symlinks, '..' components, and the ':A' word modifier

[Daniel Shahaf]

Roland Eggner wrote on Tue, Jul 05, 2016 at 14:54:30 +0200:
> (1.2)  Introduce a new modifier just for resolving
>        filereferences (“symlinks”).

This seems like a reasonable way forward.

To everyone who asked about compatibility: under the existing semantics
of :A, «$foo» and «$foo:A» might denote different files.  (Not just
different dirents, but different inodes, or possibly pathnames under
different mountpoints.)  That does not seem like a useful transformation;
it has no equivalent in other languages (including but not limited
to C); it's different from other common modifiers ('cat $foo:h/$foo:t'
== 'cat $foo:r.$foo:e' == 'cat $foo' != 'cat `f $foo:A`'); and it makes
it impossible to use the :A modifier to resolve symlinks in arguments
that may contain '..' segments (so if someone thinks of :A as "zsh's
interface to realpath(3)", their code would be broken).

So yes, backwards compatibility is important; I'm normally the first to
speak in its favour.  But when a behaviour has no obvious use-case, and
is unexpected to the point of possibly causing uses of it to be buggy,
then I do ask whether that documented behaviour is a feature or
a (documented) bug.

So, bottom line...  it seems the consensus on both lists is not to risk
breaking compatibility here, and to add another :x modifier letter
instead (as suggested by Roland here and by pws/Bart in workers/38798 et
seq).

I will add that to my list, or if someone beats me to implementing it
that's fine too.

Thanks all,

Daniel

Re: realpath(3), symlinks, '..' components, and the ':A' word modifier

[Bart Schaefer]

On Jul 7,  2:00am, Daniel Shahaf wrote:
}
} To everyone who asked about compatibility: under the existing semantics
} of :A, "$foo" and "$foo:A" might denote different files.

But that implies that "$foo" and "$foo:a" might also denote different
files, so why is *that* a useful transformation?

} So, bottom line...  it seems the consensus on both lists is not to risk
} breaking compatibility here, and to add another :x modifier letter
} instead (as suggested by Roland here and by pws/Bart in workers/38798 et
} seq).

There's no letter to use that won't in some way be confuse-able with a
parameter expansion flag, so here are some ideas:

:L - mnemonic "link" - corresponds to "ls -L"
:R - mnemonic "real" - think of realpath()
:P - mnemonic "path" - the other half of realpath(), and also in a
                       certain way comparable to the (P) parameter
		       flag in that it interprets the path name as
		       a further path name and yields the file at
		       the other path

Re: Next release (5.3)

[Roman Neuhauser]

Belatedly chiming in as someone who writes scripts in zsh
and uses :A in those.

I understand the issue is that although zshexpn(1) claims...

  This call is equivalent to `a` unless your system has the realpath
  system call (modern systems do).

... this is not the case.  Correct?  Well, I use it for this
exact purpose.

# schaefer@brasslantern.com / 2016-07-07 10:20:31 -0700:
> On Jul 7,  2:00am, Daniel Shahaf wrote:
> }
> } To everyone who asked about compatibility: under the existing semantics
> } of :A, "$foo" and "$foo:A" might denote different files.
> 
> But that implies that "$foo" and "$foo:a" might also denote different
> files, so why is *that* a useful transformation?

It's not, and I don't use it.

# p.stephenson@samsung.com / 2016-07-05 09:33:21 +0100:
> On Mon, 04 Jul 2016 08:04:24 -0700
> Bart Schaefer <schaefer@brasslantern.com> wrote:
> > (1) Daniel's suggested change to :A [care to offer an opinion?]
> 
> I'd be vaguely inclined to make sure it does what the doc currently says
> and leave it at that.

I'd prefer (it would *fix* my scripts) this to happen.

> But that's only because I've not worked out a case where I want anything
> different.  It's too difficult to come up with a categorical answer because
> it depends whether the user is used to CHASE_BLAH behaviour (I'm not
> suggesting option-specific behaviour, either).

Has anyone brought up any realistic (as in: I'm going to use it for
such-and-such) scenario for these CHASE_FOO-related variations?
It looks like you're spending energy on something no one has any use for.

-- 
roman

Re: Next release (5.3)

[Bart Schaefer]

On Jul 12,  9:58am, Roman Neuhauser wrote:
} Subject: Re: Next release (5.3)
}
} I understand the issue is that although zshexpn(1) claims...
} 
}   This call is equivalent to `a` unless your system has the realpath
}   system call (modern systems do).
} 
} ... this is not the case.  Correct?  Well, I use it for this
} exact purpose.

Your statement is going to require some clarification.  By "exact purpose"
do you mean "as a replacement for realpath"?

The documentation says:

1. resolution of `..' occurs _before_ resolution of symbolic links
2. equivalent to a unless your system has the realpath system call

These are not contraditory but they explicitly do NOT mean that :A
is a replacement for realpath.  All that (2) means is :A does NOT
follow symbolic links unless realpath is available to do that work.

You go on to say:

} > [...]  "$foo" and "$foo:a" might also denote different
} > files, so why is *that* a useful transformation?
} 
} It's not, and I don't use it.

Per my two points above, on any system that lacks realpath, you DO use
:a implicitly, because :A does not differ from :a when there is no
realpath underneath.

} > > (1) Daniel's suggested change to :A [care to offer an opinion?]
} > 
} > I'd be vaguely inclined to make sure it does what the doc currently
} > says and leave it at that.
} 
} I'd prefer (it would *fix* my scripts) this to happen.

You'd prefer what?  :A presently DOES what the doc currently says.
PWS's comment in effect means he's inclined to change nothing.

If what you mean is that you'd prefer that :A is a replacement for
realpath with all the same semantics as realpath, then you're now
requesting something that wasn't previously being discussed.

The present situation is:

    1. :a performs a string-manipulation on the path to remove any
       relative path segments.
    2. :A does (1) and then calls realpath on the result.

This matches the documentation.

Daniel is arguing that (1) is essentially useless and calling realpath
after that may give a different result than realpath on the original
path string.  His suggestion is:

    1. :a is as before
    2. :A calls realpath, and does (1) only if there is no realpath

(It's unclear to me whether there would be any reason to do (1) AFTER
calling realpath.)

The emerging consensus seems to be for:

    1. :a is as before
    2. :A is as before
    3. new modifier calls realpath and does (1) if no realpath

Is there one of those three cases with which you agree, or are you in
fact asking for zsh to re-implement realpath internally?

Does the re-statement above change anyone else's opinion about which
alternative should be chosen?  Please note that I don't think there
is any current zsh developer who is keen to rebuild realpath.

Re: Next release (5.3)

[Oliver Kiddle]

Bart wrote:

Thanks for the concise summary.

> The present situation is:
>
>     1. :a performs a string-manipulation on the path to remove any
>        relative path segments.
>     2. :A does (1) and then calls realpath on the result.

The only advantage I can think of that is that it might do what you want
in the case where you don't have read permissions on one of the
intermediate directories. Otherwise, the potential for the result being
a different file is not good.

> This matches the documentation.
>
> Daniel is arguing that (1) is essentially useless and calling realpath
> after that may give a different result than realpath on the original
> path string.  His suggestion is:
>
>     1. :a is as before
>     2. :A calls realpath, and does (1) only if there is no realpath

That seems like a good approach. It might also try (1) if realpath fails.

I don't have many uses in scripts to check but I'd expect it'd be
more likely to fix scripts than introduce a bug.

With that approach, I guess you could get the original :A behaviour by
chaining the modifiers - :a:A

And adding modifiers just makes it harder to work out which one you
want. I could see a use for a pure readlink one, however.

> (It's unclear to me whether there would be any reason to do (1) AFTER
> calling realpath.)
>
> The emerging consensus seems to be for:
>
>     1. :a is as before
>     2. :A is as before
>     3. new modifier calls realpath and does (1) if no realpath
>
> Is there one of those three cases with which you agree, or are you in
> fact asking for zsh to re-implement realpath internally?

If systems lacking realpath were common, it would be worth doing
but from a check of online man pages it seems to be there on currently
relevant systems. Our own implementation might handle unreadable
directories followed by ../ but there might be some OS specific
oddities: case-insensitivity on macOS and Solaris xattrs come to
mind.

Oliver

Re: Next release (5.3)

[Filipe Silva]

[-- Attachment #1: Type: text/plain, Size: 2381 bytes --]

Wait good people. Maybe I am missing something, but why the specific focus
on the non-posix `realpath`? Doesn't `readlink -f` emulates what realpath
does but it is always available because it is posix? Maybe :A should really
call `readlink -f`? Makes sense?

On Tue, Jul 12, 2016 at 5:23 PM, Oliver Kiddle <okiddle@yahoo.co.uk> wrote:

> Bart wrote:
>
> Thanks for the concise summary.
>
> > The present situation is:
> >
> >     1. :a performs a string-manipulation on the path to remove any
> >        relative path segments.
> >     2. :A does (1) and then calls realpath on the result.
>
> The only advantage I can think of that is that it might do what you want
> in the case where you don't have read permissions on one of the
> intermediate directories. Otherwise, the potential for the result being
> a different file is not good.
>
> > This matches the documentation.
> >
> > Daniel is arguing that (1) is essentially useless and calling realpath
> > after that may give a different result than realpath on the original
> > path string.  His suggestion is:
> >
> >     1. :a is as before
> >     2. :A calls realpath, and does (1) only if there is no realpath
>
> That seems like a good approach. It might also try (1) if realpath fails.
>
> I don't have many uses in scripts to check but I'd expect it'd be
> more likely to fix scripts than introduce a bug.
>
> With that approach, I guess you could get the original :A behaviour by
> chaining the modifiers - :a:A
>
> And adding modifiers just makes it harder to work out which one you
> want. I could see a use for a pure readlink one, however.
>
> > (It's unclear to me whether there would be any reason to do (1) AFTER
> > calling realpath.)
> >
> > The emerging consensus seems to be for:
> >
> >     1. :a is as before
> >     2. :A is as before
> >     3. new modifier calls realpath and does (1) if no realpath
> >
> > Is there one of those three cases with which you agree, or are you in
> > fact asking for zsh to re-implement realpath internally?
>
> If systems lacking realpath were common, it would be worth doing
> but from a check of online man pages it seems to be there on currently
> relevant systems. Our own implementation might handle unreadable
> directories followed by ../ but there might be some OS specific
> oddities: case-insensitivity on macOS and Solaris xattrs come to
> mind.
>
> Oliver
>

Re: Next release (5.3)

[Bart Schaefer]

On Jul 12, 11:56pm, Filipe Silva wrote:
}
} Wait good people. Maybe I am missing something, but why the specific focus
} on the non-posix `realpath`?

Sorry for the lack of clarity, but we aren't discussing the "realpath"
command, we're discussing the realpath C library function:

http://pubs.opengroup.org/onlinepubs/9699919799/functions/realpath.html

Re: Next release (5.3)

[Filipe Silva]

[-- Attachment #1: Type: text/plain, Size: 476 bytes --]

Oh, right, sorry.

On Wed, Jul 13, 2016 at 1:45 AM, Bart Schaefer <schaefer@brasslantern.com>
wrote:

> On Jul 12, 11:56pm, Filipe Silva wrote:
> }
> } Wait good people. Maybe I am missing something, but why the specific
> focus
> } on the non-posix `realpath`?
>
> Sorry for the lack of clarity, but we aren't discussing the "realpath"
> command, we're discussing the realpath C library function:
>
> http://pubs.opengroup.org/onlinepubs/9699919799/functions/realpath.html
>

Re: Next release (5.3)

[Peter Stephenson]

On Tue, 12 Jul 2016 22:23:08 +0200
Oliver Kiddle <okiddle@yahoo.co.uk> wrote:
> Bart wrote:
> 
> Thanks for the concise summary.
> 
> > The present situation is:
> >
> >     1. :a performs a string-manipulation on the path to remove any
> >        relative path segments.
> >     2. :A does (1) and then calls realpath on the result.
> 
> The only advantage I can think of that is that it might do what you want
> in the case where you don't have read permissions on one of the
> intermediate directories. Otherwise, the potential for the result being
> a different file is not good.

I've seen that last statement a lot recently and I think it's misleading.
It's not wrong but it's far from the whole story given the complexities
involved.

It's long-standing behaviour of zsh that by default directories are
managed logically, not physically, and hence that a ".." takes you up in
the logical hierarchy, i.e. the one you see in the string.

It's true that this case here isn't to do with $PWD or changing
directory etc. etc., but you don't know where the path you're applying
to the expression that's getting the :A has come from.  It might very
well have been something that was previously used for directory
management (it's a directory, after all).  In *that* case, changing the
behaviour of :A introduces bugs.  (Rather than it being a different
file, which is possible but unlikely, it's much more likely to refer to
something that doesn't exist at all --- less dangerous, but still
unhelpful.)

No one can actually point to a specific error case being fixed by the
change.

I suggest we focus on improving the documentation of :A, as the current
".." behaviour is one thing it *is* clear about so it's a little
bizarre to change *that*, to point out problems, and decide if we
actually want a new qualifier, which I think would be fairly
uncontroversial even if it's not frequently used.

pws

Re: Next release (5.3)

[Peter Stephenson]

Here's a script that's broken by the change.

I'm not saying *any* of the following:

- that this is a way I'd recommend doing things
- that there are no ways round the problem
- that leaving resolution of ".." to the last minute is a good idea

--- but I *am* saying that this script works according to the currently
documented behaviour and wouldn't any more.

pws


#!/usr/local/bin/zsh

# No CHASE_*.
emulate zsh

top=~/tmp/A_dir/data
real=$top/blah/real
link=$top/link

rm -rf $top
mkdir -p $real
ln -s $real $link

cd $link
print "In symbolic link:"
pwd
pwd -P

dir=$PWD
parent=$PWD/..

print "\nIn parent:"
cd $parent
pwd
pwd -P

print "\nIn symbolic link:"
cd $dir
pwd
pwd -P

print "This is the correct file" >$top/file
print "This is not the correct file.  I wouldn't be seeing this
if someone hadn't made any arbitrary change to the :A modifier that
changes the documented behaviour." >$real/../file

file=$parent/file

# We know :A resolves using the same rules as directory changing,
# so use that.
resolved=($file(:A))
print "\nUsing $file with :A => $resolved"
cat $resolved

Re: Next release (5.3)

[Bart Schaefer]

On Jul 13, 10:59am, Peter Stephenson wrote:
}
} Here's a script that's broken by the change.

Here's a question:  Of what significance is the use of :A in this case,
as opposed to :a ?  That is, why use :A at all here?  Can you come up
with an example where :a and :A differ that would still be broken by
the change?


} # No CHASE_*.
} emulate zsh
} 
} # We know :A resolves using the same rules as directory changing,
} # so use that.
} resolved=($file(:A))
} print "\nUsing $file with :A => $resolved"
} cat $resolved

I don't want this discussion to become circular, but don't the comments
there imply that what's desired/intended is for :A to respect CHASE_*?

(That's still different from what Daniel originally suggested, because
he wants the default to act like CHASE_LINKS, but at least it's been
touched upon in the -workers side of this discussion.)

If I were going to re-design this from scratch, I think I'd aim for a
parallel with "cd -L" and "cd -P", and thus

    $var:L	- like current :a, never chase links
    $var:P	- like Daniel's :A, always chase links
    $var:A	- one of the above, depending on CHASE_LINKS

(and never have :a in the first place).  Incidentally, outside of an
actual "cd" there's no point to examining CHASE_DOTS.

Given where we are right now, adding :P and shrugging our shoulders 
about :a and :A seems the most reasonable escape hatch.  Plus adding
a bit to the :A documentation, perhaps.  I suppose we could also make
:L a synonym for :a just for mnemonic.

Re: Next release (5.3)

[Peter Stephenson]

On Wed, 13 Jul 2016 09:35:35 -0700
Bart Schaefer <schaefer@brasslantern.com> wrote:
> On Jul 13, 10:59am, Peter Stephenson wrote:
> }
> } Here's a script that's broken by the change.
> 
> Here's a question:  Of what significance is the use of :A in this case,
> as opposed to :a ?  That is, why use :A at all here?  Can you come up
> with an example where :a and :A differ that would still be broken by
> the change?

As there are no additional symbolic links in this case, there's no
difference.

Why use :A --- for exactly the same reason you'd ever use it, that you
want the physical path for later reference rather than one that still
has symbolic links in it.  Why you'd do *that* is a vaguer question: you
might believe that the symbolic link was temporary and the physical
path not (which doesn't reflect the situtation I've got here in the
example).

The result would differ if you had a symbolic link higher up, e.g. if my
home directory was a symbolic link (from an automounter, for example),
which would be resolved in one case but not the other.  But the question
why I'd *want* that is still open; I think we're back in the woolly
world of intention that's somewhat bedevilled this discussion.

> } # No CHASE_*.
> } emulate zsh
> } 
> } # We know :A resolves using the same rules as directory changing,
> } # so use that.
> } resolved=($file(:A))
> } print "\nUsing $file with :A => $resolved"
> } cat $resolved
> 
> I don't want this discussion to become circular, but don't the comments
> there imply that what's desired/intended is for :A to respect CHASE_*?

Indeed, that was one of the things in my mind, although whether it's the
right fix is another matter.

pws

Re: Next release (5.3)

[Bart Schaefer]

On Jul 13,  5:59pm, Peter Stephenson wrote:
}
} > I don't want this discussion to become circular, but don't the comments
} > there imply that what's desired/intended is for :A to respect CHASE_*?
} 
} Indeed, that was one of the things in my mind, although whether it's the
} right fix is another matter.

As a point of reference, none of the current uses of :A in functions
distributed with the shell would expect to have behavior that might
be affected by CHASE_LINKS.  On the other hand, I suspect they're
using :A with the expectation that links *are* chased, so if we do
introduce a new :P option the scripts might want to employ it.

So we could *probably* get away with adding $var:P and introducing an
option dependency to :A.  (Sorry, throwing out all these alternatives
probably isn't getting us much closer to a resolution.)

Re: Next release (5.3)

[Daniel Shahaf]

Peter Stephenson wrote on Wed, Jul 13, 2016 at 10:59:10 +0100:
> Here's a script that's broken by the change.

It was never a question that the change *was* backwards incompatible;
the question was just whether breaking compatibility in this case was
acceptable.

The bottom line of the threads on that question is that both lists
clearly wish :A's semantics to remain unchanged, so I don't expect them
to change any sooner than our roads' lanes stop being as wide as two
Roman horses.  Any "resolve symlinks first" functionality will be added
through a new :${LETTER} modifier.

[PATCH] improve :A docs (was: Re: Next release (5.3))

[Daniel Shahaf]

Peter Stephenson wrote on Wed, Jul 13, 2016 at 10:32:33 +0100:
> No one can actually point to a specific error case being fixed by the
> change.
> 
> I suggest we focus on improving the documentation of :A, as the current
> ".." behaviour is one thing it *is* clear about so it's a little
> bizarre to change *that*, to point out problems, and decide if we
> actually want a new qualifier, which I think would be fairly
> uncontroversial even if it's not frequently used.

diff --git a/Doc/Zsh/expn.yo b/Doc/Zsh/expn.yo
index c6e7b6f..c5f3c00 100644
--- a/Doc/Zsh/expn.yo
+++ b/Doc/Zsh/expn.yo
@@ -224,10 +224,15 @@ Note that the transformation takes place even if the file or any
 intervening directories do not exist.
 )
 item(tt(A))(
-As `tt(a)', but also resolve use of symbolic links where possible.
-Note that resolution of `tt(..)' occurs em(before) resolution of symbolic
-links.  This call is equivalent to tt(a) unless your system has the
-tt(realpath) system call (modern systems do).
+Turn a file name into an absolute path as the `tt(a)' modifier does, and
+em(then) pass the result through the tt(realpath+LPAR()3+RPAR()) library
+function to resolve symbolic links.
+
+Note: on systems that do not have a tt(realpath+LPAR()3+RPAR()) library
+function, symbolic links are not resolved, so on those systems `tt(a)' and
+`tt(A)' are equivalent.
+
+Note: tt(foo:A) and tt(realpath+LPAR()foo+RPAR()) are different on some inputs.
 )
 item(tt(c))(
 Resolve a command name into an absolute path by searching the command

Re: [PATCH] improve :A docs (was: Re: Next release (5.3))

[Daniel Shahaf]

Daniel Shahaf wrote on Sun, Jul 17, 2016 at 14:59:31 +0000:
> Peter Stephenson wrote on Wed, Jul 13, 2016 at 10:32:33 +0100:
> > I suggest we focus on improving the documentation of :A,

:a too.

diff --git a/Doc/Zsh/expn.yo b/Doc/Zsh/expn.yo
index c6e7b6f..7e78869 100644
--- a/Doc/Zsh/expn.yo
+++ b/Doc/Zsh/expn.yo
@@ -219,9 +219,12 @@ noted.
 startitem()
 item(tt(a))(
 Turn a file name into an absolute path:  prepends the current directory,
-if necessary, and resolves any use of `tt(..)' and `tt(.)' in the path.
-Note that the transformation takes place even if the file or any
-intervening directories do not exist.
+if necessary; remove `tt(.)' path segments; and remove `tt(..)' path segments
+and the segments that immediately precede them.
+
+This transformation is agnostic of the filesystem: `tt(/foo/bar/../baz)' shall
+always be transformed to `tt(/foo/baz)', regardless of whether `tt(/foo)'
+exists or what kind of object (dir, file, symlink, etc.) it is.
 )
 item(tt(A))(
 As `tt(a)', but also resolve use of symbolic links where possible.

Re: Next release (5.3)

[Roman Neuhauser]

# schaefer@brasslantern.com / 2016-07-12 09:40:17 -0700:
> On Jul 12,  9:58am, Roman Neuhauser wrote:
> } I understand the issue is that although zshexpn(1) claims...
> } 
> }   This call is equivalent to `a` unless your system has the realpath
> }   system call (modern systems do).
> } 
> } ... this is not the case.  Correct?  Well, I use it for this
> } exact purpose.
> 
> Your statement is going to require some clarification.  By "exact purpose"
> do you mean "as a replacement for realpath"?

yes.

> The documentation says:
> 
> 1. resolution of `..' occurs _before_ resolution of symbolic links
> 2. equivalent to a unless your system has the realpath system call
> 
> These are not contraditory but they explicitly do NOT mean that :A
> is a replacement for realpath.  All that (2) means is :A does NOT
> follow symbolic links unless realpath is available to do that work.

alright.  i gotta admit that i misunderstood the manpage as saying
that if :A is equivalent to realpath(3) if i have it, and :a otherwise.
i know, i should have known better and read the manpage with my lawyer
hat on.  sloppy reading on my part.

however, re "explicit": the :A description is anything but!

: A      As `a', but also resolve use of symbolic links where possible.
:        Note that  resolution of `..' occurs before resolution of
:        symbolic links.  This call is equivalent to a unless your
:        system has the realpath system call (modern systems do).

so what exactly does it do if i *do* have realpath(3)?
 
> You go on to say:
> 
> } > [...]  "$foo" and "$foo:a" might also denote different
> } > files, so why is *that* a useful transformation?
> } 
> } It's not, and I don't use it.
> 
> Per my two points above, on any system that lacks realpath, you DO use
> :a implicitly, because :A does not differ from :a when there is no
> realpath underneath.

yes in principle, not in practice.  i may be lucky but it's all
relatively modern GNU/Linux or *BSD around me.  my scripts have
no users on systems without realpath(3), and wouldn't work on such
systems for other reasons anyway.
 
> } > > (1) Daniel's suggested change to :A [care to offer an opinion?]
> } > 
> } > I'd be vaguely inclined to make sure it does what the doc currently
> } > says and leave it at that.
> } 
> } I'd prefer (it would *fix* my scripts) this to happen.
> 
> You'd prefer what?  :A presently DOES what the doc currently says.
> PWS's comment in effect means he's inclined to change nothing.

right, i was writing that based on my misunderstanding of the manpage
(see above).

> If what you mean is that you'd prefer that :A is a replacement for
> realpath with all the same semantics as realpath, then you're now
> requesting something that wasn't previously being discussed.

right, sorry for that.  i'll crawl back under my rock shortly. :)
 
> The present situation is:
> 
>     1. :a performs a string-manipulation on the path to remove any
>        relative path segments.
>     2. :A does (1) and then calls realpath on the result.
> 
> This matches the documentation.
> 
> Daniel is arguing that (1) is essentially useless and calling realpath
> after that may give a different result than realpath on the original
> path string.  His suggestion is:
> 
>     1. :a is as before
>     2. :A calls realpath, and does (1) only if there is no realpath

i agree with Daniel regarding limited utility of :a, and i already
(mis)use :A as if it was doing 2.  i don't care about the fallback
to :a since systems where that happens are outside my universe.
 
> The emerging consensus seems to be for:
> 
>     1. :a is as before
>     2. :A is as before
>     3. new modifier calls realpath and does (1) if no realpath

re "does (1) if no realpath": really (1), not (2)?

> Is there one of those three cases with which you agree, or are you in
> fact asking for zsh to re-implement realpath internally?

in order of decreasing preference:

1 change the meaning of :A so that it is equivalent to realpath(3)
  where it exists and to :a on systems without realpath(3)
2 new modifier that calls realpath(3) with fallback to :A
3 new modifier that calls realpath(3) with fallback to :a

if i could have a pony then the most favorable solution would be

0 change the meaning of :A so that it is always equivalent to
  realpath(3), using an implementation bundled with Zsh where
  necessary

-- 
roman

Re: Next release (5.3)

[Bart Schaefer]

On Jul 13,  9:41am, Roman Neuhauser wrote:
}
} i agree with Daniel regarding limited utility of :a, and i already
} (mis)use :A as if it was doing 2.

Cf.  PWS's example script RE utility of :a / current :A.
 
} > The emerging consensus seems to be for:
} > 
} >     1. :a is as before
} >     2. :A is as before
} >     3. new modifier calls realpath and does (1) if no realpath
} 
} re "does (1) if no realpath": really (1), not (2)?

Well, if no realpath, then (1) and (2) are the same, so ...

} in order of decreasing preference:
} 
} 1 change the meaning of :A so that it is equivalent to realpath(3)
}   where it exists and to :a on systems without realpath(3)
} 2 new modifier that calls realpath(3) with fallback to :A
} 3 new modifier that calls realpath(3) with fallback to :a

... there's no useful distinction between your #2 and #3.

} if i could have a pony then the most favorable solution would be
} 
} 0 change the meaning of :A so that it is always equivalent to
}   realpath(3), using an implementation bundled with Zsh where
}   necessary

Noted.  Somewhat more likely would be a new modifier that implements
realpath by stealing code from somewhere that is licensing-compatible.

Re: Next release (5.3)

[Vincent Lefevre]

On 2016-07-12 09:40:17 -0700, Bart Schaefer wrote:
> On Jul 12,  9:58am, Roman Neuhauser wrote:
> } Subject: Re: Next release (5.3)
> }
> } I understand the issue is that although zshexpn(1) claims...
> } 
> }   This call is equivalent to `a` unless your system has the realpath
> }   system call (modern systems do).
> } 
> } ... this is not the case.  Correct?  Well, I use it for this
> } exact purpose.
> 
> Your statement is going to require some clarification.  By "exact purpose"
> do you mean "as a replacement for realpath"?
> 
> The documentation says:
> 
> 1. resolution of `..' occurs _before_ resolution of symbolic links
> 2. equivalent to a unless your system has the realpath system call
> 
> These are not contraditory but they explicitly do NOT mean that :A
> is a replacement for realpath.  All that (2) means is :A does NOT
> follow symbolic links unless realpath is available to do that work.

But it does not say how ".." is resolved. Using the realpath system
call is a way (the only correct one) to resolve "..".

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: Next release (5.3)

[Vincent Lefevre]

On 2016-07-20 15:05:12 +0200, Vincent Lefevre wrote:
> On 2016-07-12 09:40:17 -0700, Bart Schaefer wrote:
> > On Jul 12,  9:58am, Roman Neuhauser wrote:
> > } Subject: Re: Next release (5.3)
> > }
> > } I understand the issue is that although zshexpn(1) claims...
> > } 
> > }   This call is equivalent to `a` unless your system has the realpath
> > }   system call (modern systems do).
> > } 
> > } ... this is not the case.  Correct?  Well, I use it for this
> > } exact purpose.
> > 
> > Your statement is going to require some clarification.  By "exact purpose"
> > do you mean "as a replacement for realpath"?
> > 
> > The documentation says:
> > 
> > 1. resolution of `..' occurs _before_ resolution of symbolic links
> > 2. equivalent to a unless your system has the realpath system call
> > 
> > These are not contraditory but they explicitly do NOT mean that :A
> > is a replacement for realpath.  All that (2) means is :A does NOT
> > follow symbolic links unless realpath is available to do that work.
> 
> But it does not say how ".." is resolved. Using the realpath system
> call is a way (the only correct one) to resolve "..".

BTW, Bart, you even got confused in the past:

  http://www.zsh.org/mla/users/2013/msg00675.html

with an incorrect solution.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)

Re: Next release (5.3)

[Peter Stephenson]

On Wed, 20 Jul 2016 16:00:06 +0200
Vincent Lefevre <vincent@vinc17.net> wrote:
> But it does not say how ".." is resolved.

Yes, we should be clear it's the way zsh has done it with the default
option settings for directories in the past.

pws

diff --git a/Doc/Zsh/expn.yo b/Doc/Zsh/expn.yo
index c5f3c00..eb90e8d 100644
--- a/Doc/Zsh/expn.yo
+++ b/Doc/Zsh/expn.yo
@@ -220,6 +220,13 @@ startitem()
 item(tt(a))(
 Turn a file name into an absolute path:  prepends the current directory,
 if necessary, and resolves any use of `tt(..)' and `tt(.)' in the path.
+The resolution takes in the same manner as when changing directories
+when neither of the options tt(CHASE_DOTS) or tt(CHASE_LINKS) is
+set, i.e. this is on the logical, not the physical, directory.
+For example, the components tt(before/here/../after) resolve
+to tt(before/after) regardless of any symbolic links that may
+occur when traversing the path.
+
 Note that the transformation takes place even if the file or any
 intervening directories do not exist.
 )

Re: Next release (5.3)

[Peter Stephenson]

On Wed, 20 Jul 2016 15:15:17 +0100
Peter Stephenson <p.stephenson@samsung.com> wrote:
> +The resolution takes in the same manner as when changing directories
                       ^
		       place --- fixed that.

pws

Re: Next release (5.3)

[Daniel Shahaf]

Peter Stephenson wrote on Wed, Jul 20, 2016 at 15:15:17 +0100:
> On Wed, 20 Jul 2016 16:00:06 +0200
> Vincent Lefevre <vincent@vinc17.net> wrote:
> > But it does not say how ".." is resolved.
> 
> Yes, we should be clear it's the way zsh has done it with the default
> option settings for directories in the past.

This patch conflicts with my 21779 [I did not push immediately after
posting to allow for feedback].

I think both of them make sense: 21779 specifies the behaviour in
absolute terms and 21784 provides the "this is the algorithm 'cd' uses"
information.

Cheers,

Daniel

> pws
> 
> diff --git a/Doc/Zsh/expn.yo b/Doc/Zsh/expn.yo
> index c5f3c00..eb90e8d 100644
> --- a/Doc/Zsh/expn.yo
> +++ b/Doc/Zsh/expn.yo
> @@ -220,6 +220,13 @@ startitem()
>  item(tt(a))(
>  Turn a file name into an absolute path:  prepends the current directory,
>  if necessary, and resolves any use of `tt(..)' and `tt(.)' in the path.
> +The resolution takes in the same manner as when changing directories
> +when neither of the options tt(CHASE_DOTS) or tt(CHASE_LINKS) is
> +set, i.e. this is on the logical, not the physical, directory.
> +For example, the components tt(before/here/../after) resolve
> +to tt(before/after) regardless of any symbolic links that may
> +occur when traversing the path.
> +
>  Note that the transformation takes place even if the file or any
>  intervening directories do not exist.
>  )
> 

Re: Next release (5.3)

[Peter Stephenson]

On Thu, 21 Jul 2016 05:40:19 +0000
Daniel Shahaf <d.s@daniel.shahaf.name> wrote:
> Peter Stephenson wrote on Wed, Jul 20, 2016 at 15:15:17 +0100:
> > On Wed, 20 Jul 2016 16:00:06 +0200
> > Vincent Lefevre <vincent@vinc17.net> wrote:
> > > But it does not say how ".." is resolved.
> > 
> > Yes, we should be clear it's the way zsh has done it with the default
> > option settings for directories in the past.
> 
> This patch conflicts with my 21779 [I did not push immediately after
> posting to allow for feedback].

Yes, Bart pointed that out.

> I think both of them make sense: 21779 specifies the behaviour in
> absolute terms and 21784 provides the "this is the algorithm 'cd' uses"
> information.

I can wait.

pws

Re: Next release (5.3)

[Bart Schaefer]

On Jul 20,  3:05pm, Vincent Lefevre wrote:
} 
} But it does not say how ".." is resolved. Using the realpath system
} call is a way (the only correct one) to resolve "..".

Well, no, that's not true.  Using something that implements the same
algorithm as realpath is the only correct way to resolve a possible
chain of several relative and symbolic links.

But :a never said that it resolves symbolic links.  It says it "turns
a file name into an absolute path".  The doc probably should have said
"REMOVES any use of `..'" rather than "resolves", but there's no mention
of symbolic links at all.

You could argue that opendir("..") is the only correct way to "resolve"
".." but zsh has NEVER done that with "cd" except when CHASE_DOTS.  As
has become apparent, PWS wanted :a to predict the result of "cd", not
to resolve symbolic links.

And :A starts out with "as :a" before it gets to the part about links.

Re: Next release (5.3)

[Daniel Shahaf]

Bart Schaefer wrote on Tue, Jul 12, 2016 at 09:40:17 -0700:
> His suggestion is:
> 
>     1. :a is as before
>     2. :A calls realpath, and does (1) only if there is no realpath
> 
> (It's unclear to me whether there would be any reason to do (1) AFTER
> calling realpath.)

There shouldn't be.  POSIX specifies:

       The realpath() function shall derive, from the pathname pointed to by
       file_name, an absolute pathname that names the same file, whose
       resolution does not involve '.' , '..'  , or symbolic links.
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^

(my emphasis)

> Does the re-statement above change anyone else's opinion about which
> alternative should be chosen?  Please note that I don't think there
> is any current zsh developer who is keen to rebuild realpath.

Re-build, of course not.  Copy a *BSD libc implementation, perhaps :-)

Re: Next release (5.3)

[Daniel Shahaf]

Roman Neuhauser wrote on Tue, Jul 12, 2016 at 09:58:49 +0200:
> # schaefer@brasslantern.com / 2016-07-07 10:20:31 -0700:
> > But that's only because I've not worked out a case where I want anything
> > different.  It's too difficult to come up with a categorical answer because
> > it depends whether the user is used to CHASE_BLAH behaviour (I'm not
> > suggesting option-specific behaviour, either).
> 
> Has anyone brought up any realistic (as in: I'm going to use it for
> such-and-such) scenario for these CHASE_FOO-related variations?
> It looks like you're spending energy on something no one has any use for.

Not that I know of.  The behaviour of :A was originally reported as
a bug downstream (debian bug #607615 by Vincent), but that ticket does
not refer to CHASE_*.

Cheers,

Daniel

> -- 
> roman
> 

Re: realpath(3), symlinks, '..' components, and the ':A' word modifier

[Vincent Lefevre]

On 2016-07-07 10:20:31 -0700, Bart Schaefer wrote:
> On Jul 7,  2:00am, Daniel Shahaf wrote:
> } To everyone who asked about compatibility: under the existing semantics
> } of :A, "$foo" and "$foo:A" might denote different files.
> 
> But that implies that "$foo" and "$foo:a" might also denote different
> files, so why is *that* a useful transformation?

IMHO, the only useful transformation is to the same file (the usual
realpath).

Note that some scripts may use ":A" thinking that one would get the
same file. So, changing the behavior may also fix these scripts!

> } So, bottom line...  it seems the consensus on both lists is not to risk
> } breaking compatibility here, and to add another :x modifier letter
> } instead (as suggested by Roland here and by pws/Bart in workers/38798 et
> } seq).
> 
> There's no letter to use that won't in some way be confuse-able with a
> parameter expansion flag, so here are some ideas:
> 
> :L - mnemonic "link" - corresponds to "ls -L"
> :R - mnemonic "real" - think of realpath()
> :P - mnemonic "path" - the other half of realpath(), and also in a
>                        certain way comparable to the (P) parameter
> 		       flag in that it interprets the path name as
> 		       a further path name and yields the file at
> 		       the other path

If you do not want to change the behavior of ":A", then ":R" is
probably the best one. ":L" would definitely be a bad choice as
it could be confused with "realpath -L" from GNU coreutils, which
does the opposite (the default being "realpath -P", so using ":P"
could be OK too, where "P" means "physical").

In any case, the documentation should be clarified.

-- 
Vincent Lefèvre <vincent@vinc17.net> - Web: <https://www.vinc17.net/>
100% accessible validated (X)HTML - Blog: <https://www.vinc17.net/blog/>
Work: CR INRIA - computer arithmetic / AriC project (LIP, ENS-Lyon)