[ISSUE] [RFC] mark python3 site-packages as externally managed

[ISSUE] [RFC] mark python3 site-packages as externally managed

[classabbyamp]

[-- Attachment #1: Type: text/plain, Size: 1675 bytes --]

New issue by classabbyamp on void-packages repository

https://github.com/void-linux/void-packages/issues/43703

Description:
https://peps.python.org/pep-0668/

By adding a file `/usr/lib/python3.X/EXTERNALLY_MANAGED`, pip will not let users install python modules with `pip` outside of virtual environments

![image](https://user-images.githubusercontent.com/5366828/235501708-689a19a2-a6bf-451a-ab58-cba48f53bb5a.png)

### Pros
- will prevent people from breaking xbps-installed python modules by using pip outside a venv

### Cons
- breaks `pip install --user` too (can be [solved](https://bugs.gentoo.org/895410#c4))
- may break some void-based containers (or similar things) that install things with pip
    - solutions:
        - `pip --break-system-packages`
        - `doas pip config set install.break-system-packages True`
        - `noextract` on the EXTERNALLY-MANAGED file (could be done by default in void's official containers)

### Prior Art

- gentoo implements this, see [here](https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/gentoo-common/gentoo-common-1.ebuild#n17) and [here](https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-lang/python/python-3.11.3.ebuild#n462)
- ubuntu [apparently does this](https://ubuntuforums.org/showthread.php?t=2485257), but I can't find their source packages to check
- Alpine implemented this for ~1 day until people complained that it broke their containers (this is probably less of a concern for void, and could be mitigated)
- arch has not done this, from what I can tell
- debian and fedora put distro-packaged python modules in a different directory

cc @void-linux/pkg-committers 

Re: [RFC] mark python3 site-packages as externally managed

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 985 bytes --]

New comment by ahesford on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1530095046

Comment:
I think I'm OK with this. Having to configure pip for user installs is a little annoying but worth the protections for system packages. Still, I'm heavily dependent on pip per-user installation for my regular work and would want to do a bit of testing to see how painful this is in practice.

As for containers... when I deploy custom Python in containers, I think it's better to wrap things in a venv anyway (sometimes with `--system-site-packages`) to make installation of packaged stuff easier. If `--system-site-packages` is compatible with `EXTERNALLY-MANAGED`, maybe that's good enough; otherwise, defaulting to removing that file or setting the system-wide pip config to allow breaking system packages in a container seems fine. (We could even let container builds break and force users to take this action if we really wanted to.)

Re: [RFC] mark python3 site-packages as externally managed

[chrysos349]

[-- Attachment #1: Type: text/plain, Size: 522 bytes --]

New comment by chrysos349 on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1530103296

Comment:
> ubuntu [apparently does this](https://ubuntuforums.org/showthread.php?t=2485257), but I can't find their source packages to check

yes, it does starting with 23.04. see below:

https://git.launchpad.net/ubuntu/+source/python3.11/tree/debian/rules?h=ubuntu/lunar#n1241
https://git.launchpad.net/ubuntu/+source/python3.11/tree/debian/EXTERNALLY-MANAGED.in?h=ubuntu/lunar

Re: [RFC] mark python3 site-packages as externally managed

[icp1994]

[-- Attachment #1: Type: text/plain, Size: 293 bytes --]

New comment by icp1994 on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1530183002

Comment:
The relevant Debian [NEWS file](https://salsa.debian.org/python-team/packages/python-pip/-/blob/315bcd6f4cdb5bcfb8a74f1e599739cc74c86432/debian/NEWS)

Re: [RFC] mark python3 site-packages as externally managed

[CtrlC-Root]

[-- Attachment #1: Type: text/plain, Size: 1189 bytes --]

New comment by CtrlC-Root on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1627759680

Comment:
As a heavy Python user I'll add my two cents. I think this is generally a great idea as long as `pip install --user` remains accessible. I already use virtual environments for all project packages in order to avoid breaking system packages. However I've noticed there are lots of scripts or instructions (which ask to be copy and pasted) on the internet that include `sudo pip install` in them. As a user it's certainly possible to carefully inspect these before running them but you can still make a mistake and miss something. By the time you notice the system wide packages are likely contaminated or broken in a way that's annoying to fix. So requiring an opt-in mechanism makes a lot of sense to me.

I do find `pip install --user` necessary for non-project specific tools though (ex. [virtualfish](https://github.com/justinmayer/virtualfish), [powerline-status](https://pypi.org/project/powerline-status/)). The alternative would be to create system packages for these tools but in my opinion that would be a higher burden all around.

Re: [RFC] mark python3 site-packages as externally managed

[CtrlC-Root]

[-- Attachment #1: Type: text/plain, Size: 1361 bytes --]

New comment by CtrlC-Root on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1627759680

Comment:
As a heavy Python user I'll add my two cents. I think this is generally a great idea as long as `pip install --user` remains accessible. I already use virtual environments for all project packages in order to avoid breaking system packages. However I've noticed there are lots of scripts or instructions (which ask to be copy and pasted) on the internet that include `sudo pip install` in them. As a user it's certainly possible to carefully inspect these before running them but you can still make a mistake and miss something. By the time you notice the system wide packages are likely contaminated or broken in a way that's annoying to fix. So requiring an opt-in mechanism makes a lot of sense to me.

I do find `pip install --user` necessary for non-project specific tools though (ex. [virtualfish](https://github.com/justinmayer/virtualfish), [powerline-status](https://pypi.org/project/powerline-status/)). The alternative would be to create system packages for these tools but in my opinion that would be a higher burden all around.

EDIT: Perhaps if this was implemented it would be a good idea to add a new section to the handbook for Python that mentions the workarounds above for various use cases.

Re: [RFC] mark python3 site-packages as externally managed

[icp1994]

[-- Attachment #1: Type: text/plain, Size: 369 bytes --]

New comment by icp1994 on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1627765934

Comment:
I recommend using [pipx](https://github.com/void-linux/void-packages/blob/master/srcpkgs/python3-pipx/template) instead. I also plan to package [rye](https://github.com/mitsuhiko/rye#global-tools) when it's a bit more mature.

Re: [RFC] mark python3 site-packages as externally managed

[classabbyamp]

[-- Attachment #1: Type: text/plain, Size: 340 bytes --]

New comment by classabbyamp on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1627772223

Comment:
`pip --user` can cause the same issues as `sudo pip` because it will be added to Python's module path ahead of the system modules dirs, so the EXTERNALLY-MANAGED file also breaks `pip --user`

Re: [RFC] mark python3 site-packages as externally managed

[CtrlC-Root]

[-- Attachment #1: Type: text/plain, Size: 788 bytes --]

New comment by CtrlC-Root on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1627774644

Comment:
> `pip --user` can cause the same issues as `sudo pip` because it will be added to Python's module path ahead of the system modules dirs, so the EXTERNALLY-MANAGED file also breaks `pip --user`

Indeed but it's relatively easy to fix by removing the appropriate `site-packages` folder under `.local/lib`. Especially if you only use it for a few system wide tools and keep everything else in virtual environments. I think this is something people who use it regularly (on a rolling distro at least) would be familiar with since every time the `python3` package is updated to a new major version you need to reinstall these packages anyways.

Re: [RFC] mark python3 site-packages as externally managed

[0x5c]

[-- Attachment #1: Type: text/plain, Size: 1541 bytes --]

New comment by 0x5c on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1628688907

Comment:
> I do find pip install --user necessary for non-project specific tools though (ex. [virtualfish](https://github.com/justinmayer/virtualfish), [powerline-status](https://pypi.org/project/powerline-status/)). The alternative would be to create system packages for these tools but in my opinion that would be a higher burden all around.

I have had tools like that in my workflow in the past, but never would it have been impossible to simply pass `--break-system-packages` like the error message suggests. Also as icp1994 mentioned earlier, tools like `pipx` are designed for this specific purpose of installing python-based tools in venvs and exposing them in the path.

> Indeed but it's relatively easy to fix by removing the appropriate `site-packages` folder under `.local/lib`. Especially if you only use it for a few system wide tools and keep everything else in virtual environments. I think this is something people who use it regularly (on a rolling distro at least) would be familiar with since every time the `python3` package is updated to a new major version you need to reinstall these packages anyways.

While easy to fix, this kind of bug has the potential to be very difficult to diagnose properly and take much valuable time, including in the distro's support places (bug tracker, IRC, etc).
This is the main reason the "externally managed" feature exists in the first place.

Re: [RFC] mark python3 site-packages as externally managed

[CtrlC-Root]

[-- Attachment #1: Type: text/plain, Size: 1970 bytes --]

New comment by CtrlC-Root on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1629373289

Comment:
> > I do find pip install --user necessary for non-project specific tools though (ex. [virtualfish](https://github.com/justinmayer/virtualfish), [powerline-status](https://pypi.org/project/powerline-status/)). The alternative would be to create system packages for these tools but in my opinion that would be a higher burden all around.
> 
> I have had tools like that in my workflow in the past, but never would it have been impossible to simply pass `--break-system-packages` like the error message suggests. Also as icp1994 mentioned earlier, tools like `pipx` are designed for this specific purpose of installing python-based tools in venvs and exposing them in the path.
> 
> > Indeed but it's relatively easy to fix by removing the appropriate `site-packages` folder under `.local/lib`. Especially if you only use it for a few system wide tools and keep everything else in virtual environments. I think this is something people who use it regularly (on a rolling distro at least) would be familiar with since every time the `python3` package is updated to a new major version you need to reinstall these packages anyways.
> 
> While easy to fix, this kind of bug has the potential to be very difficult to diagnose properly and take much valuable time, including in the distro's support places (bug tracker, IRC, etc). This is the main reason the "externally managed" feature exists in the first place.

I think we are actually in complete agreement here. I think this change is good. I think advanced users who know what they are doing can use the escape hatch / workarounds listed above to continue using `pip install --user`. Some users may choose an alternative like `pipx` if it suits them. The only thing I would add is to consider documenting these options in the handbook so they are more discoverable.

Re: [RFC] mark python3 site-packages as externally managed

[classabbyamp]

[-- Attachment #1: Type: text/plain, Size: 244 bytes --]

New comment by classabbyamp on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1629376210

Comment:
if you look at the PR I made for this, the error message pip gives mentions everything already 

Re: [RFC] mark python3 site-packages as externally managed

[classabbyamp]

[-- Attachment #1: Type: text/plain, Size: 302 bytes --]

New comment by classabbyamp on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1629376210

Comment:
if you look at the [PR](https://github.com/void-linux/void-packages/pull/43735) I made for this, the error message pip gives mentions everything already 

Re: [RFC] mark python3 site-packages as externally managed

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 170 bytes --]

New comment by ahesford on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1751053447

Comment:
Adopted with Python 3.12.

Re: [ISSUE] [CLOSED] [RFC] mark python3 site-packages as externally managed

[ahesford]

[-- Attachment #1: Type: text/plain, Size: 1700 bytes --]

Closed issue by classabbyamp on void-packages repository

https://github.com/void-linux/void-packages/issues/43703

Description:
https://peps.python.org/pep-0668/

By adding a file `/usr/lib/python3.X/EXTERNALLY_MANAGED`, pip will not let users install python modules with `pip` outside of virtual environments

![image](https://user-images.githubusercontent.com/5366828/235501708-689a19a2-a6bf-451a-ab58-cba48f53bb5a.png)

### Pros
- will prevent people from breaking xbps-installed python modules by using pip outside a venv

### Cons
- breaks `pip install --user` too (can be [solved](https://bugs.gentoo.org/895410#c4))
- may break some void-based containers (or similar things) that install things with pip
    - solutions:
        - `pip --break-system-packages`
        - `doas pip config set install.break-system-packages True`
        - `noextract` on the EXTERNALLY-MANAGED file (could be done by default in void's official containers)

### Prior Art

- gentoo implements this, see [here](https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/gentoo-common/gentoo-common-1.ebuild#n17) and [here](https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-lang/python/python-3.11.3.ebuild#n462)
- ubuntu [apparently does this](https://ubuntuforums.org/showthread.php?t=2485257), but I can't find their source packages to check
- Alpine implemented this for ~1 day until people complained that it broke their containers (this is probably less of a concern for void, and could be mitigated)
- arch has not done this, from what I can tell
- debian and fedora put distro-packaged python modules in a different directory, according to the PEP

cc @void-linux/pkg-committers