Github messages for voidlinux
 help / color / mirror / Atom feed
* [ISSUE] [RFC] mark python3 site-packages as externally managed
@ 2023-05-01 18:27 classabbyamp
  2023-05-01 19:12 ` ahesford
                   ` (13 more replies)
  0 siblings, 14 replies; 15+ messages in thread
From: classabbyamp @ 2023-05-01 18:27 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 1675 bytes --]

New issue by classabbyamp on void-packages repository

https://github.com/void-linux/void-packages/issues/43703

Description:
https://peps.python.org/pep-0668/

By adding a file `/usr/lib/python3.X/EXTERNALLY_MANAGED`, pip will not let users install python modules with `pip` outside of virtual environments

![image](https://user-images.githubusercontent.com/5366828/235501708-689a19a2-a6bf-451a-ab58-cba48f53bb5a.png)

### Pros
- will prevent people from breaking xbps-installed python modules by using pip outside a venv

### Cons
- breaks `pip install --user` too (can be [solved](https://bugs.gentoo.org/895410#c4))
- may break some void-based containers (or similar things) that install things with pip
    - solutions:
        - `pip --break-system-packages`
        - `doas pip config set install.break-system-packages True`
        - `noextract` on the EXTERNALLY-MANAGED file (could be done by default in void's official containers)

### Prior Art

- gentoo implements this, see [here](https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/gentoo-common/gentoo-common-1.ebuild#n17) and [here](https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-lang/python/python-3.11.3.ebuild#n462)
- ubuntu [apparently does this](https://ubuntuforums.org/showthread.php?t=2485257), but I can't find their source packages to check
- Alpine implemented this for ~1 day until people complained that it broke their containers (this is probably less of a concern for void, and could be mitigated)
- arch has not done this, from what I can tell
- debian and fedora put distro-packaged python modules in a different directory

cc @void-linux/pkg-committers 

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [RFC] mark python3 site-packages as externally managed
  2023-05-01 18:27 [ISSUE] [RFC] mark python3 site-packages as externally managed classabbyamp
@ 2023-05-01 19:12 ` ahesford
  2023-05-01 19:20 ` chrysos349
                   ` (12 subsequent siblings)
  13 siblings, 0 replies; 15+ messages in thread
From: ahesford @ 2023-05-01 19:12 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 985 bytes --]

New comment by ahesford on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1530095046

Comment:
I think I'm OK with this. Having to configure pip for user installs is a little annoying but worth the protections for system packages. Still, I'm heavily dependent on pip per-user installation for my regular work and would want to do a bit of testing to see how painful this is in practice.

As for containers... when I deploy custom Python in containers, I think it's better to wrap things in a venv anyway (sometimes with `--system-site-packages`) to make installation of packaged stuff easier. If `--system-site-packages` is compatible with `EXTERNALLY-MANAGED`, maybe that's good enough; otherwise, defaulting to removing that file or setting the system-wide pip config to allow breaking system packages in a container seems fine. (We could even let container builds break and force users to take this action if we really wanted to.)

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [RFC] mark python3 site-packages as externally managed
  2023-05-01 18:27 [ISSUE] [RFC] mark python3 site-packages as externally managed classabbyamp
  2023-05-01 19:12 ` ahesford
@ 2023-05-01 19:20 ` chrysos349
  2023-05-01 20:23 ` icp1994
                   ` (11 subsequent siblings)
  13 siblings, 0 replies; 15+ messages in thread
From: chrysos349 @ 2023-05-01 19:20 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 522 bytes --]

New comment by chrysos349 on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1530103296

Comment:
> ubuntu [apparently does this](https://ubuntuforums.org/showthread.php?t=2485257), but I can't find their source packages to check

yes, it does starting with 23.04. see below:

https://git.launchpad.net/ubuntu/+source/python3.11/tree/debian/rules?h=ubuntu/lunar#n1241
https://git.launchpad.net/ubuntu/+source/python3.11/tree/debian/EXTERNALLY-MANAGED.in?h=ubuntu/lunar

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [RFC] mark python3 site-packages as externally managed
  2023-05-01 18:27 [ISSUE] [RFC] mark python3 site-packages as externally managed classabbyamp
  2023-05-01 19:12 ` ahesford
  2023-05-01 19:20 ` chrysos349
@ 2023-05-01 20:23 ` icp1994
  2023-07-09 16:05 ` CtrlC-Root
                   ` (10 subsequent siblings)
  13 siblings, 0 replies; 15+ messages in thread
From: icp1994 @ 2023-05-01 20:23 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 293 bytes --]

New comment by icp1994 on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1530183002

Comment:
The relevant Debian [NEWS file](https://salsa.debian.org/python-team/packages/python-pip/-/blob/315bcd6f4cdb5bcfb8a74f1e599739cc74c86432/debian/NEWS)

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [RFC] mark python3 site-packages as externally managed
  2023-05-01 18:27 [ISSUE] [RFC] mark python3 site-packages as externally managed classabbyamp
                   ` (2 preceding siblings ...)
  2023-05-01 20:23 ` icp1994
@ 2023-07-09 16:05 ` CtrlC-Root
  2023-07-09 16:12 ` CtrlC-Root
                   ` (9 subsequent siblings)
  13 siblings, 0 replies; 15+ messages in thread
From: CtrlC-Root @ 2023-07-09 16:05 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 1189 bytes --]

New comment by CtrlC-Root on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1627759680

Comment:
As a heavy Python user I'll add my two cents. I think this is generally a great idea as long as `pip install --user` remains accessible. I already use virtual environments for all project packages in order to avoid breaking system packages. However I've noticed there are lots of scripts or instructions (which ask to be copy and pasted) on the internet that include `sudo pip install` in them. As a user it's certainly possible to carefully inspect these before running them but you can still make a mistake and miss something. By the time you notice the system wide packages are likely contaminated or broken in a way that's annoying to fix. So requiring an opt-in mechanism makes a lot of sense to me.

I do find `pip install --user` necessary for non-project specific tools though (ex. [virtualfish](https://github.com/justinmayer/virtualfish), [powerline-status](https://pypi.org/project/powerline-status/)). The alternative would be to create system packages for these tools but in my opinion that would be a higher burden all around.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [RFC] mark python3 site-packages as externally managed
  2023-05-01 18:27 [ISSUE] [RFC] mark python3 site-packages as externally managed classabbyamp
                   ` (3 preceding siblings ...)
  2023-07-09 16:05 ` CtrlC-Root
@ 2023-07-09 16:12 ` CtrlC-Root
  2023-07-09 16:31 ` icp1994
                   ` (8 subsequent siblings)
  13 siblings, 0 replies; 15+ messages in thread
From: CtrlC-Root @ 2023-07-09 16:12 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 1361 bytes --]

New comment by CtrlC-Root on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1627759680

Comment:
As a heavy Python user I'll add my two cents. I think this is generally a great idea as long as `pip install --user` remains accessible. I already use virtual environments for all project packages in order to avoid breaking system packages. However I've noticed there are lots of scripts or instructions (which ask to be copy and pasted) on the internet that include `sudo pip install` in them. As a user it's certainly possible to carefully inspect these before running them but you can still make a mistake and miss something. By the time you notice the system wide packages are likely contaminated or broken in a way that's annoying to fix. So requiring an opt-in mechanism makes a lot of sense to me.

I do find `pip install --user` necessary for non-project specific tools though (ex. [virtualfish](https://github.com/justinmayer/virtualfish), [powerline-status](https://pypi.org/project/powerline-status/)). The alternative would be to create system packages for these tools but in my opinion that would be a higher burden all around.

EDIT: Perhaps if this was implemented it would be a good idea to add a new section to the handbook for Python that mentions the workarounds above for various use cases.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [RFC] mark python3 site-packages as externally managed
  2023-05-01 18:27 [ISSUE] [RFC] mark python3 site-packages as externally managed classabbyamp
                   ` (4 preceding siblings ...)
  2023-07-09 16:12 ` CtrlC-Root
@ 2023-07-09 16:31 ` icp1994
  2023-07-09 16:55 ` classabbyamp
                   ` (7 subsequent siblings)
  13 siblings, 0 replies; 15+ messages in thread
From: icp1994 @ 2023-07-09 16:31 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 369 bytes --]

New comment by icp1994 on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1627765934

Comment:
I recommend using [pipx](https://github.com/void-linux/void-packages/blob/master/srcpkgs/python3-pipx/template) instead. I also plan to package [rye](https://github.com/mitsuhiko/rye#global-tools) when it's a bit more mature.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [RFC] mark python3 site-packages as externally managed
  2023-05-01 18:27 [ISSUE] [RFC] mark python3 site-packages as externally managed classabbyamp
                   ` (5 preceding siblings ...)
  2023-07-09 16:31 ` icp1994
@ 2023-07-09 16:55 ` classabbyamp
  2023-07-09 17:07 ` CtrlC-Root
                   ` (6 subsequent siblings)
  13 siblings, 0 replies; 15+ messages in thread
From: classabbyamp @ 2023-07-09 16:55 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 340 bytes --]

New comment by classabbyamp on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1627772223

Comment:
`pip --user` can cause the same issues as `sudo pip` because it will be added to Python's module path ahead of the system modules dirs, so the EXTERNALLY-MANAGED file also breaks `pip --user`

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [RFC] mark python3 site-packages as externally managed
  2023-05-01 18:27 [ISSUE] [RFC] mark python3 site-packages as externally managed classabbyamp
                   ` (6 preceding siblings ...)
  2023-07-09 16:55 ` classabbyamp
@ 2023-07-09 17:07 ` CtrlC-Root
  2023-07-10 10:41 ` 0x5c
                   ` (5 subsequent siblings)
  13 siblings, 0 replies; 15+ messages in thread
From: CtrlC-Root @ 2023-07-09 17:07 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 788 bytes --]

New comment by CtrlC-Root on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1627774644

Comment:
> `pip --user` can cause the same issues as `sudo pip` because it will be added to Python's module path ahead of the system modules dirs, so the EXTERNALLY-MANAGED file also breaks `pip --user`

Indeed but it's relatively easy to fix by removing the appropriate `site-packages` folder under `.local/lib`. Especially if you only use it for a few system wide tools and keep everything else in virtual environments. I think this is something people who use it regularly (on a rolling distro at least) would be familiar with since every time the `python3` package is updated to a new major version you need to reinstall these packages anyways.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [RFC] mark python3 site-packages as externally managed
  2023-05-01 18:27 [ISSUE] [RFC] mark python3 site-packages as externally managed classabbyamp
                   ` (7 preceding siblings ...)
  2023-07-09 17:07 ` CtrlC-Root
@ 2023-07-10 10:41 ` 0x5c
  2023-07-10 17:07 ` CtrlC-Root
                   ` (4 subsequent siblings)
  13 siblings, 0 replies; 15+ messages in thread
From: 0x5c @ 2023-07-10 10:41 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 1541 bytes --]

New comment by 0x5c on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1628688907

Comment:
> I do find pip install --user necessary for non-project specific tools though (ex. [virtualfish](https://github.com/justinmayer/virtualfish), [powerline-status](https://pypi.org/project/powerline-status/)). The alternative would be to create system packages for these tools but in my opinion that would be a higher burden all around.

I have had tools like that in my workflow in the past, but never would it have been impossible to simply pass `--break-system-packages` like the error message suggests. Also as icp1994 mentioned earlier, tools like `pipx` are designed for this specific purpose of installing python-based tools in venvs and exposing them in the path.

> Indeed but it's relatively easy to fix by removing the appropriate `site-packages` folder under `.local/lib`. Especially if you only use it for a few system wide tools and keep everything else in virtual environments. I think this is something people who use it regularly (on a rolling distro at least) would be familiar with since every time the `python3` package is updated to a new major version you need to reinstall these packages anyways.

While easy to fix, this kind of bug has the potential to be very difficult to diagnose properly and take much valuable time, including in the distro's support places (bug tracker, IRC, etc).
This is the main reason the "externally managed" feature exists in the first place.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [RFC] mark python3 site-packages as externally managed
  2023-05-01 18:27 [ISSUE] [RFC] mark python3 site-packages as externally managed classabbyamp
                   ` (8 preceding siblings ...)
  2023-07-10 10:41 ` 0x5c
@ 2023-07-10 17:07 ` CtrlC-Root
  2023-07-10 17:08 ` classabbyamp
                   ` (3 subsequent siblings)
  13 siblings, 0 replies; 15+ messages in thread
From: CtrlC-Root @ 2023-07-10 17:07 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 1970 bytes --]

New comment by CtrlC-Root on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1629373289

Comment:
> > I do find pip install --user necessary for non-project specific tools though (ex. [virtualfish](https://github.com/justinmayer/virtualfish), [powerline-status](https://pypi.org/project/powerline-status/)). The alternative would be to create system packages for these tools but in my opinion that would be a higher burden all around.
> 
> I have had tools like that in my workflow in the past, but never would it have been impossible to simply pass `--break-system-packages` like the error message suggests. Also as icp1994 mentioned earlier, tools like `pipx` are designed for this specific purpose of installing python-based tools in venvs and exposing them in the path.
> 
> > Indeed but it's relatively easy to fix by removing the appropriate `site-packages` folder under `.local/lib`. Especially if you only use it for a few system wide tools and keep everything else in virtual environments. I think this is something people who use it regularly (on a rolling distro at least) would be familiar with since every time the `python3` package is updated to a new major version you need to reinstall these packages anyways.
> 
> While easy to fix, this kind of bug has the potential to be very difficult to diagnose properly and take much valuable time, including in the distro's support places (bug tracker, IRC, etc). This is the main reason the "externally managed" feature exists in the first place.

I think we are actually in complete agreement here. I think this change is good. I think advanced users who know what they are doing can use the escape hatch / workarounds listed above to continue using `pip install --user`. Some users may choose an alternative like `pipx` if it suits them. The only thing I would add is to consider documenting these options in the handbook so they are more discoverable.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [RFC] mark python3 site-packages as externally managed
  2023-05-01 18:27 [ISSUE] [RFC] mark python3 site-packages as externally managed classabbyamp
                   ` (9 preceding siblings ...)
  2023-07-10 17:07 ` CtrlC-Root
@ 2023-07-10 17:08 ` classabbyamp
  2023-07-10 17:09 ` classabbyamp
                   ` (2 subsequent siblings)
  13 siblings, 0 replies; 15+ messages in thread
From: classabbyamp @ 2023-07-10 17:08 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 244 bytes --]

New comment by classabbyamp on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1629376210

Comment:
if you look at the PR I made for this, the error message pip gives mentions everything already 

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [RFC] mark python3 site-packages as externally managed
  2023-05-01 18:27 [ISSUE] [RFC] mark python3 site-packages as externally managed classabbyamp
                   ` (10 preceding siblings ...)
  2023-07-10 17:08 ` classabbyamp
@ 2023-07-10 17:09 ` classabbyamp
  2023-10-06 16:26 ` [ISSUE] [CLOSED] " ahesford
  2023-10-06 16:26 ` ahesford
  13 siblings, 0 replies; 15+ messages in thread
From: classabbyamp @ 2023-07-10 17:09 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 302 bytes --]

New comment by classabbyamp on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1629376210

Comment:
if you look at the [PR](https://github.com/void-linux/void-packages/pull/43735) I made for this, the error message pip gives mentions everything already 

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [RFC] mark python3 site-packages as externally managed
  2023-05-01 18:27 [ISSUE] [RFC] mark python3 site-packages as externally managed classabbyamp
                   ` (12 preceding siblings ...)
  2023-10-06 16:26 ` [ISSUE] [CLOSED] " ahesford
@ 2023-10-06 16:26 ` ahesford
  13 siblings, 0 replies; 15+ messages in thread
From: ahesford @ 2023-10-06 16:26 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 170 bytes --]

New comment by ahesford on void-packages repository

https://github.com/void-linux/void-packages/issues/43703#issuecomment-1751053447

Comment:
Adopted with Python 3.12.

^ permalink raw reply	[flat|nested] 15+ messages in thread

* Re: [ISSUE] [CLOSED] [RFC] mark python3 site-packages as externally managed
  2023-05-01 18:27 [ISSUE] [RFC] mark python3 site-packages as externally managed classabbyamp
                   ` (11 preceding siblings ...)
  2023-07-10 17:09 ` classabbyamp
@ 2023-10-06 16:26 ` ahesford
  2023-10-06 16:26 ` ahesford
  13 siblings, 0 replies; 15+ messages in thread
From: ahesford @ 2023-10-06 16:26 UTC (permalink / raw)
  To: ml

[-- Attachment #1: Type: text/plain, Size: 1700 bytes --]

Closed issue by classabbyamp on void-packages repository

https://github.com/void-linux/void-packages/issues/43703

Description:
https://peps.python.org/pep-0668/

By adding a file `/usr/lib/python3.X/EXTERNALLY_MANAGED`, pip will not let users install python modules with `pip` outside of virtual environments

![image](https://user-images.githubusercontent.com/5366828/235501708-689a19a2-a6bf-451a-ab58-cba48f53bb5a.png)

### Pros
- will prevent people from breaking xbps-installed python modules by using pip outside a venv

### Cons
- breaks `pip install --user` too (can be [solved](https://bugs.gentoo.org/895410#c4))
- may break some void-based containers (or similar things) that install things with pip
    - solutions:
        - `pip --break-system-packages`
        - `doas pip config set install.break-system-packages True`
        - `noextract` on the EXTERNALLY-MANAGED file (could be done by default in void's official containers)

### Prior Art

- gentoo implements this, see [here](https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/gentoo-common/gentoo-common-1.ebuild#n17) and [here](https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-lang/python/python-3.11.3.ebuild#n462)
- ubuntu [apparently does this](https://ubuntuforums.org/showthread.php?t=2485257), but I can't find their source packages to check
- Alpine implemented this for ~1 day until people complained that it broke their containers (this is probably less of a concern for void, and could be mitigated)
- arch has not done this, from what I can tell
- debian and fedora put distro-packaged python modules in a different directory, according to the PEP

cc @void-linux/pkg-committers 

^ permalink raw reply	[flat|nested] 15+ messages in thread

end of thread, other threads:[~2023-10-06 16:26 UTC | newest]

Thread overview: 15+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-05-01 18:27 [ISSUE] [RFC] mark python3 site-packages as externally managed classabbyamp
2023-05-01 19:12 ` ahesford
2023-05-01 19:20 ` chrysos349
2023-05-01 20:23 ` icp1994
2023-07-09 16:05 ` CtrlC-Root
2023-07-09 16:12 ` CtrlC-Root
2023-07-09 16:31 ` icp1994
2023-07-09 16:55 ` classabbyamp
2023-07-09 17:07 ` CtrlC-Root
2023-07-10 10:41 ` 0x5c
2023-07-10 17:07 ` CtrlC-Root
2023-07-10 17:08 ` classabbyamp
2023-07-10 17:09 ` classabbyamp
2023-10-06 16:26 ` [ISSUE] [CLOSED] " ahesford
2023-10-06 16:26 ` ahesford

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).