PSA: Mac OS X El Capitan upgrade might break your $PATH

PSA: Mac OS X El Capitan upgrade might break your $PATH

[Kurtis Rader]

[-- Attachment #1: Type: text/plain, Size: 904 bytes --]

A public service announcement for those running Zsh on Mac OS X.

Upgrading to El Capitan (OS X 10.11) will install a /etc/zprofile that
contains

    # system-wide environment settings for zsh(1)
    if [ -x /usr/libexec/path_helper ]; then
       eval `/usr/libexec/path_helper -s`
    fi

On a new login shell that will be sourced after your $HOME/.zshenv. If you
set your $PATH in your .zshenv the /usr/libexec/path_helper program will
alter the order of the directories. The order appears random so it's
probably using a hashed set to avoid having the same directory appear more
than once in the result.

You can't remove /usr/libexec/path_helper due to the new security
mechanisms (at least not without booting from a recovery disk) but you can
remove /etc/zprofile which is what I did to keep my $PATH from being mucked
with.

-- 
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank

Re: PSA: Mac OS X El Capitan upgrade might break your $PATH

[Bart Schaefer]

On Jul 31,  9:35am, Kurtis Rader wrote:
}
} On a new login shell that will be sourced after your $HOME/.zshenv.
} If you set your $PATH in your .zshenv the /usr/libexec/path_helper
} program will alter the order of the directories.
} 
} You can't remove /usr/libexec/path_helper due to the new security
} mechanisms (at least not without booting from a recovery disk) but you can
} remove /etc/zprofile which is what I did

Another option is to add to your ~/.zshenv the line:

    setopt no_global_rcs

And then if desired in your ~/.zprofile add:

    setopt global_rcs

This pair of commands placed in these two files will disable reading of
/etc/zprofile and then re-enable reading of /etc/zshrc and /etc/zlogin,
without needing to remove system files (which might re-appear on an update
anyway).

Re: PSA: Mac OS X El Capitan upgrade might break your $PATH

[Mikael Magnusson]

On Fri, Jul 31, 2015 at 6:35 PM, Kurtis Rader <krader@skepticism.us> wrote:
> A public service announcement for those running Zsh on Mac OS X.
>
> Upgrading to El Capitan (OS X 10.11) will install a /etc/zprofile that
> contains
>
>     # system-wide environment settings for zsh(1)
>     if [ -x /usr/libexec/path_helper ]; then
>        eval `/usr/libexec/path_helper -s`
>     fi
>
> On a new login shell that will be sourced after your $HOME/.zshenv. If you
> set your $PATH in your .zshenv the /usr/libexec/path_helper program will
> alter the order of the directories. The order appears random so it's
> probably using a hashed set to avoid having the same directory appear more
> than once in the result.
>
> You can't remove /usr/libexec/path_helper due to the new security
> mechanisms (at least not without booting from a recovery disk) but you can
> remove /etc/zprofile which is what I did to keep my $PATH from being mucked
> with.

That's good news, they used to have that code in /etc/zshenv which
meant you had _no_ way to override it. It only took them a few years
to fix this.

As a sidenote, it's not a great idea to set your path in .zshenv, you
should probably move those settings to your .zprofile. (Otherwise,
running a script that uses zsh will not use the $PATH from the
environment, which can break things in some situations).

-- 
Mikael Magnusson

Re: PSA: Mac OS X El Capitan upgrade might break your $PATH

[Andrew Janke]

On 7/31/15 2:53 PM, Mikael Magnusson wrote:
> On Fri, Jul 31, 2015 at 6:35 PM, Kurtis Rader <krader@skepticism.us> wrote:
>> A public service announcement for those running Zsh on Mac OS X.
>>
>> Upgrading to El Capitan (OS X 10.11) will install a /etc/zprofile that
>> contains
>>
>>      # system-wide environment settings for zsh(1)
>>      if [ -x /usr/libexec/path_helper ]; then
>>         eval `/usr/libexec/path_helper -s`
>>      fi
>>
>> On a new login shell that will be sourced after your $HOME/.zshenv. If you
>> set your $PATH in your .zshenv the /usr/libexec/path_helper program will
>> alter the order of the directories. The order appears random so it's
>> probably using a hashed set to avoid having the same directory appear more
>> than once in the result.
>>
>> You can't remove /usr/libexec/path_helper due to the new security
>> mechanisms (at least not without booting from a recovery disk) but you can
>> remove /etc/zprofile which is what I did to keep my $PATH from being mucked
>> with.
> That's good news, they used to have that code in /etc/zshenv which
> meant you had _no_ way to override it. It only took them a few years
> to fix this.
>
> As a sidenote, it's not a great idea to set your path in .zshenv, you
> should probably move those settings to your .zprofile. (Otherwise,
> running a script that uses zsh will not use the $PATH from the
> environment, which can break things in some situations).
>

I suspect the order of entries in path_helper is determined by 
alphabetical ordering of the filenames in /etc/paths.d, where 
path_helper locates the files that come after the default system paths. 
That "40-" in XQuartz looks like an rcdir-style technique to enforce 
ordering, and I think the XQuartz folks know what they're doing with the 
OS X system stuff. (This is on 10.9.)

$ ls -l /etc/paths.d
total 16
-rw-r--r--   1 root  wheel    13 Aug 11  2014 40-XQuartz
-rw-r--r--   1 root  wheel    12 May  7 21:06 TeX

Playing around with this seems to support that theory. Files read in 
order, duplicate entries removed after the first occurrence.

#/bin/sh
# install-hello-paths - script to test path_helper behavior
echo /hello/world > /etc/paths.d/01-hello
echo /hello/world > /etc/paths.d/99-hello


One other El Capitan change: the system-supplied zsh (5.0.8) appears to 
be compiled with /usr/local/share/zsh/site-functions in the default 
$fpath, which was not the case for earlier versions of OS X. This is 
probably related to the "rootless" stuff that locks down /usr outside 
/usr/local/.

Cheers,
Andrew

Re: PSA: Mac OS X El Capitan upgrade might break your $PATH

[Kurtis Rader]

[-- Attachment #1: Type: text/plain, Size: 1920 bytes --]

On Fri, Jul 31, 2015 at 2:05 PM, Andrew Janke <andrew@apjanke.net> wrote:
>
>
> I suspect the order of entries in path_helper is determined by
> alphabetical ordering of the filenames in /etc/paths.d, where path_helper
> locates the files that come after the default system paths. That "40-" in
> XQuartz looks like an rcdir-style technique to enforce ordering, and I
> think the XQuartz folks know what they're doing with the OS X system stuff.
> (This is on 10.9.)
>

That would be fine if it simply appended those directories in that order to
the existing PATH as the man page claims. But it is clearly using a hash
based ordering and ignoring even the apparent ordering implied by the file
names in /etc/paths.d. From my system:

14:45 macbook  coff  ~  ls /etc/paths.d
40-XQuartz go
14:50 macbook  coff  ~  cat /etc/paths.d/*
/opt/X11/bin
/usr/local/go/bin
14:52 macbook  coff  ~  /usr/libexec/path_helper -s
PATH="/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/go/bin:/Users/krader/bin:/Users/krader/sbin:/Users/krader/symlinks:/usr/local/sbin";
export PATH;
14:52 macbook  coff  ~  echo $PATH
/Users/krader/bin:/Users/krader/sbin:/Users/krader/symlinks:/usr/local/bin:/usr/local/go/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/opt/X11/bin


One other El Capitan change: the system-supplied zsh (5.0.8) appears to be
> compiled with /usr/local/share/zsh/site-functions in the default $fpath,
> which was not the case for earlier versions of OS X. This is probably
> related to the "rootless" stuff that locks down /usr outside /usr/local/.
>

Yes, I noticed that as well and should have mentioned it as it causes zsh
running as root to complain about "compinit: insecure files" since those
files are managed by HomeBrew and owned by me.

Computer Security: If it isn't getting in your way you're doing it wrong :-)

-- 
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank

Re: PSA: Mac OS X El Capitan upgrade might break your $PATH

[Andrew Janke]

[-- Attachment #1: Type: text/plain, Size: 3151 bytes --]



On 7/31/15 6:01 PM, Kurtis Rader wrote:
> On Fri, Jul 31, 2015 at 2:05 PM, Andrew Janke <andrew@apjanke.net 
> <mailto:andrew@apjanke.net>> wrote:
>
>
>     I suspect the order of entries in path_helper is determined by
>     alphabetical ordering of the filenames in /etc/paths.d, where
>     path_helper locates the files that come after the default system
>     paths. That "40-" in XQuartz looks like an rcdir-style technique
>     to enforce ordering, and I think the XQuartz folks know what
>     they're doing with the OS X system stuff. (This is on 10.9.)
>
>
> That would be fine if it simply appended those directories in that 
> order to the existing PATH as the man page claims. But it is clearly 
> using a hash based ordering and ignoring even the apparent ordering 
> implied by the file names in /etc/paths.d. From my system:
>
> 14:45 macbook  coff  ~  ls /etc/paths.d
> 40-XQuartzgo
> 14:50 macbook  coff  ~  cat /etc/paths.d/*
> /opt/X11/bin
> /usr/local/go/bin
> 14:52 macbook  coff  ~  /usr/libexec/path_helper -s
> PATH="/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/go/bin:/Users/krader/bin:/Users/krader/sbin:/Users/krader/symlinks:/usr/local/sbin"; 
> export PATH;
> 14:52 macbook  coff  ~  echo $PATH
> /Users/krader/bin:/Users/krader/sbin:/Users/krader/symlinks:/usr/local/bin:/usr/local/go/bin:/usr/local/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/opt/X11/bin
>
That path_helper output doesn't look random or hash based. That looks like:

- the system default paths (from /etc/paths) come first, in the order 
specified in /etc/paths
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:

- then the stuff from /etc/paths.d, in filename alphabetical order
/opt/X11/bin:/usr/local/go/bin:

- then any other item it found in $PATH at the time of its invocation, 
in the order in which they appeared in $PATH.
/Users/krader/bin:/Users/krader/sbin:/Users/krader/symlinks:/usr/local/sbin

That lastpart isn't documented in `man path_helper`, but it behaved this 
way consistently when I was testing it with various $PATH values. (on 10.9)

The man page could stand tobe more thorough and clear, but I don't think 
it's randomly rearranging your path. And the `man path_helper` page does 
indicate where the default paths come from, though it's buried a few 
paragraphs down.

      Prior to reading these directories, default PATH and MANPATH 
values are
      obtained from the files /etc/paths and /etc/manpaths respectively.

>
>     One other El Capitan change: the system-supplied zsh (5.0.8)
>     appears to be compiled with /usr/local/share/zsh/site-functions in
>     the default $fpath, which was not the case for earlier versions of
>     OS X. This is probably related to the "rootless" stuff that locks
>     down /usr outside /usr/local/.
>
>
> Yes, I noticed that as well and should have mentioned it as it causes 
> zsh running as root to complain about "compinit: insecure files" since 
> those files are managed by HomeBrew and owned by me.
> Computer Security: If it isn't getting in your way you're doing it 
> wrong :-)
> -- 
> Kurtis Rader
> Caretaker of the exceptional canines Junior and Hank

Re: PSA: Mac OS X El Capitan upgrade might break your $PATH

[Kurtis Rader]

[-- Attachment #1: Type: text/plain, Size: 1159 bytes --]

On Fri, Jul 31, 2015 at 3:29 PM, Andrew Janke <andrew@apjanke.net> wrote:

> That path_helper output doesn't look random or hash based. That looks like:
>
[snip]

> The man page could stand to be more thorough and clear, but I don't think
> it's randomly rearranging your path. And the `man path_helper` page does
> indicate where the default paths come from, though it's buried a few
> paragraphs down.
>

Yes, your analysis appears correct. In all likelihood the program is
working as intended. Clearly the man page does not match the actual
behavior as the very first sentence says "...appends their contents to the
PATH...". The man page says nothing at all about the sequence the
directories appear being changed. And while the man page goes on to explain
that the "default PATH ... values are obtained from /etc/paths..." that is
misleading. The use of the word "default" implies to me that if the current
PATH already contains those directories the existing PATH will be used as
is.

Yet I think we can all agree that the man page is crap and the design of
path_helper is crap :-)

-- 
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank

Re: PSA: Mac OS X El Capitan upgrade might break your $PATH

[Kurtis Rader]

[-- Attachment #1: Type: text/plain, Size: 1481 bytes --]

On Fri, Jul 31, 2015 at 11:53 AM, Mikael Magnusson <mikachu@gmail.com>
wrote:

> That's good news, they used to have that code in /etc/zshenv which
> meant you had _no_ way to override it. It only took them a few years
> to fix this.
>
> As a sidenote, it's not a great idea to set your path in .zshenv, you
> should probably move those settings to your .zprofile. (Otherwise,
> running a script that uses zsh will not use the $PATH from the
> environment, which can break things in some situations).
>

It's a trade-off. Back when ksh93 was my primary shell I had a ~/.environ
file where I put such things. Then I sourced that from ~/.profile and my
crontab entries. That way my cron jobs would have an environment consistent
with my interactive shells. Zsh's .zshenv makes things much easier. It also
means that when I do something like

ssh home "some_command"

that command gets the same environment it would have if I had gotten an
interactive shell and run the command.

I protect against the problem you mention by checking if PATH contains
$HOME/bin. If it does I assume it's already been initialized to a
reasonable value and leave it alone. If not I install my own PATH.

Of course this is only a problem if you shadow commands -- something most
people don't do. In my case I have wrappers around things like ssh in my
~/bin directory that I want to have precedence over the system supplied
version.

-- 
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank