Problem: You have a PC or an SSH account in a computation server and you want to use the latest and greatest Python stack for your work. The main problem is that Python has a lot of external libraries which aren’t generally packaged by distro maintainers. One way to overcome this issue is to use pip, a command-line utility that fetches Python packages from PyPI (Python Package Index) and installs them globally (system-wide) or locally (user-wide). Although pip and PyPI duo seem to solve the dependency to distro packages, they have serious problems: Package dependencies and keeping your packages up-to-date can generally be messy and libraries may not be correctly packaged so that they work out-of-the-box in a scientific computing environment. Also, there isn’t the concept of isolated environments in pip.
Today we’ll be talking about another solution called Anaconda which is an advanced scientific Python suite developed by Continuum Analytics.
What is Anaconda?
Anaconda is nothing but a curated Python package repository and relevant command-line utilities to manage custom Python environments. By using Anaconda, you can:
- Install everything (literally everything) related to Python in your HOME folder
- Keep update packages easily
- Create and manage several isolated environments in your HOME folder. A common example would be to have environments for Python 2 and Python 3.
- Take advantage of many other great features if you’d like
There are 2 variants: Anaconda and Miniconda:
- Anaconda contains the majority of the packages (A total of 195 packages as of June 2015) in the installer which is ~300MB.
- Miniconda only ships the bare minimum and it is up to you to install the packages that you’d like to use afterwards. I always prefer this one.
So let’s head up to miniconda download page. Here you’ll see Python 2.x and Python 3.x links. The difference between them is that miniconda3 by default creates Python 3.4 installation environments. Note that this doesn’t mean you can’t create a Python 2.7 environment with miniconda3. Only their defaults are different. Since I’m still not familiar at all with Python 3.x, I’m downloading Python 2.7 64-bit bash installer for my laptop which is running Ubuntu 14.04 64-bit.
Run the installer script from a terminal:
ozan@ozan-Lenovo:~$ bash Miniconda-latest-Linux-x86_64.sh Welcome to Miniconda 3.10.1 (by Continuum Analytics, Inc.) In order to continue the installation process, please review the license agreement. Please, press ENTER to continue >>>
The rest of the installation is easy: Accept the license and the default installation location ($HOME/miniconda). Once the packages are installed, the installer will ask you whether to prepend Miniconda install location to PATH or no. Saying yes to this (recommended) will automatically make your shell conda-aware. You can always to this afterwards by editing your .bashrc file. Now open a new terminal screen and type conda, if everything went well you’ll see the help page of the conda command.
The name of the default environment is root. Let’s list the available conda environments:
ozan@ozan-Lenovo:~$ conda env list # conda environments: # root * /home/ozan/testconda
Good! Now the last step is to activate the environment:
ozan@ozan-Lenovo:~$ source activate root discarding /home/ozan/testconda/bin from PATH prepending /home/ozan/testconda/bin to PATH
source is a built-in shell command which applies a shell script to the current shell, e.g. the changes that the script does remains active in your shell. activate is a script installed by miniconda ($HOME/miniconda/bin/activate). When you run the above command, conda sets up Python related paths and variables for the running shell. From now on, you will be accessing the interpreter and packages installed in your local conda environment instead of the system-wide ones. This is actually what we were trying to achieve: You have total control of a Python stack installed locally inside your $HOME folder.
I highly recommend to add the activation command to your .bashrc as well so that firing up a new shell automatically puts you in your conda environment. The end of your .bashrc should look like this:
export PATH=/home/ozan/miniconda/bin:$PATH source activate root
Let’s launch the Python interpreter to see the results:
ozan@ozan-Lenovo:~$ python Python 2.7.9 |Continuum Analytics, Inc.| (default, Apr 14 2015, 12:54:25) [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. Anaconda is brought to you by Continuum Analytics. Please check out: http://continuum.io/thanks and https://binstar.org >>>
If you see Continuum Analytics next to the Python version then everything is OK 🙂
Upgrade and Install
The first thing to do is to upgrade your environment so that new packages are fetched and installed from conda repositories:
ozan@ozan-Lenovo:~$ conda update --all
Just type yes and sit back. Once the update completes, you now have the latest and greatest bare minimum Python stack at your orders. Now let’s install ipython:
ozan@ozan-Lenovo:~$ conda install ipython Fetching package metadata: .... Solving package specifications: . Package plan for installation in environment /home/ozan/testconda: The following packages will be downloaded: package | build ---------------------------|----------------- ipython-3.2.0 | py27_0 3.4 MB The following NEW packages will be INSTALLED: ipython: 3.2.0-py27_0 Proceed ([y]/n)? y ozan@ozan-Lenovo:~$ ipython Python 2.7.10 |Continuum Analytics, Inc.| (default, May 28 2015, 17:02:03) Type "copyright", "credits" or "license" for more information. IPython 3.2.0 -- An enhanced Interactive Python. Anaconda is brought to you by Continuum Analytics. Please check out: http://continuum.io/thanks and https://anaconda.org ? -> Introduction and overview of IPython's features. %quickref -> Quick reference. help -> Python's own help system. object? -> Details about 'object', use 'object??' for extra details.
Creating a new environment
There are several advanced features of conda regarding environments but here I will only show an example of creating a Python 3.4 environment named py34 and installing ipython inside of it:
ozan@ozan-Lenovo:~$ conda create -n py3 python=3.4 ipython
That’s it! Once the installation completes, you can switch to your new environment using the source activate <env name> trick:
ozan@ozan-Lenovo:~$ source activate py3 discarding /home/ozan/testconda/bin from PATH prepending /home/ozan/testconda/envs/py3/bin to PATH ozan@ozan-Lenovo:~$ ipython Python 3.4.3 |Continuum Analytics, Inc.| (default, Jun 4 2015, 15:29:08) Type "copyright", "credits" or "license" for more information. IPython 3.2.0 -- An enhanced Interactive Python. Anaconda is brought to you by Continuum Analytics. Please check out: http://continuum.io/thanks and https://anaconda.org
A “source deactivate” will switch back to your default root environment.
(Note that since we have by default activated the root environment in .bashrc, launching a new shell will always bring you your Python 2.x environment.)
Where is Science?
Python by default is slow for doing scientific computations due to numerous design choices. NumPy (Numerical Python) overcomes this limitation and enriches Python with Matlab-like facilities. You can easily install NumPy by typing conda install numpy but instead let’s proceed to talk about some technical details.
BLAS (Basic Linear Algebra Subprograms)
BLAS (Basic Linear Algebra Subprograms) is a specification that prescribes a set of low-level routines for performing common linear algebra operations such as vector addition, scalar multiplication, dot products, linear combinations, and matrix multiplication. They are the de facto API for linear algebra libraries, with bindings for both C and Fortran.
Any serious computation which is numerically expensive is guaranteed to rely on optimized BLAS libraries in background. Linux distributions (at least Fedora and Ubuntu) ships with ATLAS but you can find some other BLAS implementations in their repositories too. Intel has a proprietary BLAS implementation called MKL (Math Kernel Library) which is aggressively optimized for Intel CPU’s. If you are a student you can get an academic license from Intel and download MKL without a fee.
Nowadays, another implementation called OpenBLAS is highly recommended. Generally OpenBLAS and MKL are quite close in terms of performance. Note that it would be better to download and compile OpenBLAS on your computer instead of relying on distro packages in order to achieve best performance.
So now you may ask “What does the installed NumPy use as BLAS implementation?”. It is ATLAS but since conda creates a brand new environment in your $HOME folder, it can’t rely on distribution provided ATLAS which may not be installed at all. That’s why it is statically linked against ATLAS during compile time.
Now let’s try a simple dot product operation between large matrices using the NumPy installed from conda. Run the below snippet in ipython (%timeit is a special ipython magic which correctly calculates elapsed time for an operation):
import numpy as np a = np.random.random_sample((1000, 1000)) b = np.random.random_sample((1000, 1000)) c = np.random.random_sample((2000, 2000)) d = np.random.random_sample((2000, 2000)) %timeit np.dot(a,b) 10 loops, best of 3: 74.8 ms per loop %timeit np.dot(c,d) 1 loops, best of 3: 1.04 s per loop
On my 3 year old Core i5-2467M 1.6GHz laptop (2 physical cores), dot product of two 1000×1000 matrices took approximately ~75ms while the same operation for 2000×2000 matrices took ~1s.
(Note: Although BLAS implementations greatly benefits from multi-core CPU’s, Hyper-Threading (an extra execution thread per physical core) doesn’t help at all. That is why you’ll generally see that only 2/4 CPU’s are under heavy use during a BLAS operation if you have a dual-core CPU for example. Don’t panic, this is expected.)
(Update: As of February 2016, Anaconda started to ship with MKL without a need of an academic license.)
Accelerate is an Anaconda Add-On that allows Anaconda to take advantage of modern multi-core and GPU architectures. Accelerate optimizes common operations like linear algebra, random-number generation, and Fourier transforms in standard Anaconda packages. Accelerate also contains revolutionary programming patterns in NumbaPro to enable Python developers to easily compile their own Python code to many-core, and GPU architectures.
Although Conda Accelerate is a paid add-on, it is free for academic usage. So if you apply for an academic license and put the license file in your $HOME/.continuum folder, “conda install accelerate” will install the Accelerate add-on which will automatically switch your several packages like NumPy, SciPy, etc. to MKL versions. Accelerate also installs CUDA & GPU related stuff which is fantastic if you are lucky enough to have a CUDA capable GPU.
Now let’s try the dot product benchmark again with MKL-enabled NumPy:
%timeit np.dot(a,b) 10 loops, best of 3: 75.1 ms per loop %timeit np.dot(c,d) 1 loops, best of 3: 585 ms per loop
For the larger matrice case, MKL-enabled NumPy performed 2x better than the standard ATLAS one.
Numba & NumbaPro
Finally, If you’re really into optimizing the performance of your scientific solution, you can try Numba which is a JIT (Just-in-time) compiler for Python. (NumbaPro is the pro version shipped withing Accelerate). Numba uses a compiler framework called LLVM to dynamically generate native code for decorated Python snippets.
- Use always a solution like Anaconda to ease your Python installations. Other alternatives may be Canopy, Pythonxy but I strongly recommend Anaconda as the team behind it not only does packaging but also develops innovative products, libraries, add-ons, etc. Take a look at their website and blogs.
- Always keep in mind that the bare default installation will generally be suboptimal in terms of BLAS performance. Use MKL or OpenBLAS where possible. On GPU side, NVIDIA has a BLAS implementation called cuBLAS.
- If you are diving into machine learning, especially deep learning, try hard to get some GPU’s as CPU-only deep learning is very time-consuming.
- Take a look at other languages and frameworks. For deep learning current popular entities are:
- Theano (GPU/CPU symbolic expression compiler for Python)
- Pylearn2 (Neural network framework based on Theano)
- Lasagne (Neural network framework based on Theano)
- Keras (Neural network framework based on Theano)
- Neon (A highly GPU optimized neural network framework in Python)
- Torch (Neural network framework in Lua mainly developed by Facebook AI)
- Caffe (Berkeley’s C++/Python CPU/GPU neural network toolbox)
- cuDNN (NVIDIA’s CUDA optimized deep neural network library)