Getting a Module Error When Running Pytest Even Though the Module is Installed in the Current Virtual Environment
Hello everyone,
I am encountering an issue when running pytest in my virtual environment. Although I have installed the required modules in my virtual environment using pip, I am still getting a ModuleNotFoundError when running pytest. I had to install some packages globally using the package manager to stop seeing these errors. However, this only works with popular packages since not all of them are available on the package manager.
As a workaround, I had to uninstall pytest from the package manager and install it in the virtual environment. However, I would prefer to use the pytest from the package manager since it makes sense to me to install a package I use often globally instead of installing it in each virtual environment.
In summary, I would like to install only pytest globally while all other packages are installed in the virtual environment. And I want to know how to use the global pytest from the virtual environment without getting any errors.
Thank you in advance for the help.
Here is the output of pip freeze in my virtual environment:
venv ❯ pip freeze
backoff==2.2.1
certifi==2023.5.7
charset-normalizer==3.2.0
feedparser==6.0.10
idna==3.4
pythorhead==0.12.3
requests==2.31.0
schedule==1.2.0
sgmllib3k==1.0.0
urllib3==2.0.3
And here is the error I am getting when running pytest:
venv ❯ pytest my_module.py
E ModuleNotFoundError: No module named 'schedule'
2 answers
As a workaround, I had to uninstall pytest from the package manager and install it in the virtual environment. However, I would prefer to use the pytest from the package manager since it makes sense to me to install a package I use often globally instead of installing it in each virtual environment.
I'd say that wasn't the workaround, it was the solution. A python package from outside your venv does not know what's installed inside. To run pytest inside a venv, with the venv, you need to install it inside of it.
0 comment threads
Although the system package manager was involved for your setup, this is really just a special case of a pure Python issue.
When you run a program like Pytest that is itself written in Python, generally the "executable" is just a wrapper that invokes Python to import the corresponding library and call some entry-point function. On Linux, these are simply Python scripts with a shebang line; for example, if you do cat `which pytest` you should see something like
#!/absolute/path/to/some/installation/of/python
# -*- coding: utf-8 -*-
import re
import sys
from pytest import console_main
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
sys.exit(console_main())
If you use the system package manager to install Pytest, then the Pytest copy that you get will be set up to use the system Python (i.e. the shebang line will specify /usr/bin/python or similar), and therefore the code will run in the system Python's environment with the system Python's installed third-party libraries. It has no awareness of your "current" venv (virtual environment) - because there isn't really such a thing in the first place ("activating" a venv doesn't do very much, and isn't strictly required in order to use it anyway), and because that isn't the copy of Python that's running. When the system Python runs, its default sys.path initialization will include the system packages (including the system-installed Pytest) - not anything that you installed into the venv.
Further, it will not work to do something like python `which pytest` (i.e., running the wrapper executable as a script, with the venv's Python), because then the import of the Pytest library will fail, because that isn't installed for the venv.
On Linux, you could edit the wrapper so that the shebang points at /usr/bin/env python instead. This is a brutal hack, of course: it's not portable (on Windows, the wrappers are actual native-code executables that make a Windows system call to launch Python) and it breaks convention. (People often ask why these wrappers aren't generated using relative paths, which along with a tweak to the activate script would allow the venv folders to be relocated; the answer, apparently, is that some users depend on being able to copy or symlink the wrappers.)
You can try arranging for the venv's site-packages directory to be added to sys.path - for example by using the PYTHONPATH environment variable, or by using one of the configuration file options for Pytest to supply a pythonpath value. However, trying to force Python to "share" libraries from another environment like this is not safe. It can easily break if the venv's Python is a different version from the system Python, and you won't be able to prevent Pytest from accessing the system Python's libraries (which could spoil your test results, e.g. by fixing broken imports in your code that you were trying to detect) without even trickier hacking (i.e. using PYTHONHOME).
However, the expected and normal way to solve the problem is exactly what you called a "workaround": install Pytest into the venv that you're using to test your code. (You should also consider doing an "editable" install of the code itself into the venv, so that Pytest can reliably import your code for testing without worrying about the current working directory.) As much sense as you might think it makes to reuse a system-installed Pytest, it really isn't designed to work that way.
(Pip does have this functionality, BTW; but the UI for it is clumsy and a lot of the rest of the ecosystem is designed with the expectation that each venv will get its own copy of Pip. And besides, this is only possible because Pip doesn't need to import anything from the environment it's installing into. If you use the clunky UI and get a system installation of Pip to install into a venv, from Pip's perspective, the venv's site-packages is just another directory. But the whole point of Pytest is that it orchestrates running your code.)
0 comment threads
1 comment thread