Dependency management in any language is hard, best way to avoid a dependency hell is to avoid any dependencies. :) But that’s never practical.
These are few suggestions to make dependency management easier. This blog post is mainly focused on dependency management in a web applications. Hope this will be useful.
Use venv and split out requirements similar to:
Requirements file Use
Direct dependencies of the application.
Version locked list of direct dependncies and their dependencies
Dependencies for testing
Dependencies for security audits and tests
Development tools and libraries for the project
Always use venv
Python virtual environment (venv) module comes with your standard Python installation. venv helps you to keep your development environment clean without leaking Python libraries to other projects or environments. This also allows you to keep different versions of Python and dependencies for testing.
To create a new Python virtual environment use following command, this will
create a Python venv in
python -m venv venv
To activate your python venv:
If you want to deactivate your venv, use
I thought it’s good idea to mention about tox here.
toxis very useful tool run tests for different Python environments. When you have a properly configured
tox.inifile and all the required Python versions that you need to support installed. Just running
toxcommand will take care of things.
Use requirements.txt file for direct dependencies
Use a requirements.txt file for your direct dependencies, depending on your project requirements you may add version information as well. But I like to keep the requirements.txt file without any version information. Let’s call this main requirements file because there’ll be more.
You can use
pip inside venv to install your dependencies.
pip install -r requirements.txt
It’s important to keep only the libraries that required to run your application in this main requirements file. So don’t include libraries that’s required for testing, deployment etc in this one.
What about dependencies of direct dependencies?
Now it’s important to version lock dependencies for a production release.
It’s a bad idea to let your production environment decide which version to
install because that can lead to breaking changes or unintended side effects.
For that use
Create this file after installing all your main requirements. You can use
pip freeze command for that.
pip freeze > requirements.txt.lock
requirements.txt.lock must be used when you are deploying to a
production or test server or running tests on a CI/CD pipeline.
If you have to debug a production issue use this file to install dependencies,
so you have same versions of libraries as your production environment has.
Note that, some libraries depends on operating system and other external ibraries, so this does not guarantee that your work environment will be 100% imilar to production.
imilar to python version, pip version matters too. So it’s important to make ure all environments use same pip version, and keep that up to date.
What about testing and other tools?
Keep other tools that you use for development like
I like to keep tools used for security vulnerability test in a seperate file
requirements.sec.txt, but this could be part of
requirements.test.txt as well. I like to keep it seperate because that allows
me to set up a seperate pipeline that only checks for security vulnerabilities
and dependency audits daily.
Talking about security, it’s important audit your
daily. You can use safety for that. When
is installed it’s simple as running:
safety check -r requirements.txt.lock
Another good habit is to keep pip up to
pip will let you know if there’s an update. You can upgrade
with following command.
pip install --upgrade pip
I haven’t used Constraint files in my projects so far. These files allows you to main a certain version of a dependency through out.
If you have a constraint file, you can use that with
pip install as follow.
pip install -c constraints.txt -r requirements.txt