Dealing with dependancies in Python

Dependancy Management, Programming, Python, conda, pip

Application dependencies are libraries — other than those that are a part of your project code — required to build and run your application. When working on a collaborative project, managing dependencies becomes essential.

This EdPresso shot walks you through the use of conda to manage dependencies for an application. It will show you how to install and use the necessary tools and make strong recommendations on best practices.

Motivation

First, let us generate a list of all packages that are a currently a part of our development environment.

1
2
$ cd /PATH/TO/PROJECT
$ pip freeze > requirements.txt && cat requirements.txt

Without a proper framework for dependency management, this is what one sees.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
alabaster==0.7.12
appnope==0.1.0
appscript==1.0.1
asn1crypto==0.24.0
astroid==2.2.5
astropy==3.2.1
atomicwrites==1.3.0
attrs==19.1.0
Babel==2.7.0
backcall==0.1.0
backports.functools-lru-cache==1.5
backports.os==0.1.1
backports.shutil-get-terminal-size==1.0.0
backports.tempfile==1.0
backports.weakref==1.0.post1
beautifulsoup4==4.7.1
bitarray==0.9.3
bkcharts==0.2
bleach==3.1.0
bokeh==1.2.0
boto==2.49.0
Bottleneck==1.2.1
certifi==2019.6.16
cffi==1.12.3
chardet==3.0.4
Click==7.0
cloudpickle==1.2.1
clyent==1.2.2
colorama==0.4.1
cryptography==2.7
cycler==0.10.0
Cython==0.29.12
cytoolz==0.10.0
dask==2.1.0
... 428 more

That’s a lot of packages. Now you might be wondering what kind of project uses so many libraries? The answer is no kind. At least from what I have seen. No Python project uses 400+ packages. So what is the reason this type of output? Well, this list includes all packages in the environment including those that you do not use in your current project.

This is not the right way to deal with application dependancies. We can’t give this list to our users. There’s no need to make everybody who wants to use this application download and install every single package we have on our system. We want to create a list of only those packages relevant that our application and we want that list to detail the correct versions of every package.

Using a virtual environment

A virtual environment is a tool that helps to keep dependencies required by different projects separate by creating isolated environments for them, containing only the required packages.

There are many ways to create virtual environments. In fact, every modern Python distribution has built-in creation capabilities. This tutorial will use conda.

Installation

1
$ pip install conda

Creating your first virtual environment

1
$ conda create -n new_venv python=3.7

We just created a new virtual environment called new_venv and specified that we’ll be using Python 3.7 with it — you can use any other version compatible with your project. Now let’s activate this environment and see our requirements.txt file.

1
2
3
$ cd /PATH/TO/PROJECT
$ conda activate new_venv
$ pip freeze > requirements.txt && cat requirements.txt

So what does the new output look like?

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
chardet==3.0.4
Click==7.0
dash==0.30.0
dash-core-components==0.38.0
dash-html-components==0.13.2
dash-renderer==0.15.0
decorator==4.3.0
Flask==1.0.2
Flask-Compress==1.4.0
Flask-SQLAlchemy==2.3.2
ipython-genutils==0.2.0
Jinja2==2.10
jsonschema==2.6.0
jupyter-core==4.4.0
MarkupSafe==1.1.0
numpy==1.15.4
pandas==0.22.0
plotly==3.4.1
python-dateutil==2.7.5
requests==2.20.1
retrying==1.3.3
six==1.11.0
SQLAlchemy==1.2.14
urllib3==1.24.1
Werkzeug==0.14.1

A much more concise version. Now we can supply this list to our users and all they have to do is install the listed packages.

1
2
$ cd /PATH/TO/PROJECT
$ pip install -r requirements.txt

Let there be light!