Post History

## Overview

Modern Linux distributions generally come with Python pre-installed; but fundamentally this copy of Python is not provided for experimentation or general software development, but rather to support other applications and tools in the distro.

As a developer, it won't meet your needs as-is unless all your dependencies are already available as system packages, and this route is still probably only a good idea if you have the means to distribute a system package. (In fact, some of these distributions of Python are even missing parts of the standard library.) As an end user, while Python is generally cross-platform, if you want to install code not packaged by your distro, you will have to take extra steps.

There are really two separate concerns here: the Python *executable* itself, and the system Python *environment* — meaning, basically, the path(s) where system packages will install Python code, and where Python will look for libraries by default at startup.

## The executable

Generally, there's no problem with *using* the provided `python` executable (or any symlinks etc. it may have). In fact, the most common strategies for dealing with environment problems depend on this. However, attempting to *upgrade* this Python, or to install an *additional* version in the usual place, may cause serious breakages. If for example you build a separate version of Python from source, it will be better to `./configure` it to use a different install prefix (such as `/opt` or `/usr/local`, rather than the default `/usr`) and `make altinstall` to put the installed build there instead. You may also want to rename the executable, or otherwise ensure that it doesn't shadow the system Python: make sure that system scripts with a `#!/usr/bin/python` or `#!/usr/bin/python3` shebang will run with the original system Python executable.

## The environment

Installer tools like `pip` often default to installing in a folder relative to the target Python, which Python automatically configures at startup to be searched for third-party code. This can still work as an unprivileged user (and you should *never use `sudo` with these tools*), installing into a user-specific folder; but when Python (and therefore these system scripts) run as you, they will pick up that user-specific folder, which can cause interference.

Modern Linux distros therefore implement an [additional protection](https://peps.python.org/pep-0668/), implemented as of the Python 3.11 release in 2022. This represents cooperation between third-party Python tools (`pip` is *not* made by the core Python development team) and Linux distros: the distro places a marker file to signal that a system package manager is responsible for packages installed there, and Python package tools respect that by default.

[Circumventing this protection is possible, but strongly discouraged](https://software.codidact.com/posts/291839). In the worst case, interference with the system Python environment could break (among other things) your written-in-Python system package manager, making recovery difficult. Also, if you happen to install malware into the system environment, ordinary system tools could be tricked into running that malware, when it might otherwise sit inactive until you explicitly `import`ed some code from your own project.

### How to avoid the system environment

The standard isolation technique is the [*virtual environment*](https://software.codidact.com/posts/291740). There are many tools you can use to create these, manage them, control when they're used, etc.; but fundamentally they all work in the same way. A separate folder hierarchy is created similar to the one for system packages, and some symlinks to the "base" Python are put in appropriate places in this hierarchy. Additionally, a short `pyvenv.cfg` config file is added, as well as some "activation" scripts that allow for some convenient (and generally optional!) manipulation of environment variables[^1]. When Python starts from the symlink, it looks for the corresponding config file; if it's found and validated, Python will configure its default search path for Python code, to use the virtual environment's folders instead of the system's.

The Python standard library provides a `venv` module (on Debian and derivatives, you will likely need to install a separate package for this, even though it's supposed to be part of the standard library) that creates virtual environments for you. This includes a simple command-line interface: e.g. `python -m venv my-venv`. By default, these virtual environments will include a bootstrapped installation of pip; the bootstrap process may take several seconds and use ~15 megabytes of additional disk space, per environment. You can avoid this by passing `--without-pip` to the command, but then you'll need a separate copy of pip (or another installer) to install into that environment. With pip, pass the `--python` argument (with the path to the virtual envrionment's symlink) to choose the target installation environment.[^2]



[^1]: Pedantically, a virtual environment doesn't need to contain the activation scripts to be valid — but all the standard ways to set up a virtual environment will create them.

[^2]: This requires pip version 22.3 or later, and requires that your copy of pip is compatible with the version of Python that the virtual environment uses. For most people this will rarely if ever cause a problem. Using pip with `--python` is also slightly slower, since the program will basically restart itself in a new process under the specified Python. Other installation tools may avoid issues like these.