Setting up our Django site environment with pythonbrew and virtualenv

My problem: I have a server running my Django apps that I set up when I had very little experience with Django, and just some more with Python. The configuraton is using Apache and mod_wsgi on Ubuntu, running on the system’s default Python installation (2.6) which has Django 1.1 installed.

I need to update my site, which has been upgraded to use Django 1.2, and would benefit from Python 2.7 (although the site itself runs on 2.6 as well, running the tests requires 2.7, since I use some of the new convenient unittest assertions from that version, e.g. assertIsNotNone or assertIn).

Upgrading the site will require some migrations, which are not backwards compatible (e.g. some table and field renames). Since all in all will be a delicate intervention, I’d like to create a totally separate environment for testing the revamped site while the previous version is still online. Even if it was not risky, now I know better than when I started, and it’s a good time to switch to a virtual environment setup.

Therefore, in order to proceed we will:

1. Install python 2.7 with pythonbrew
2. Install virtualenv and virtualenwrapper
3. Create a virtualenv with that python version
4. Install all requirements with pip
5. Make a Copy of the production DB, and migrate
6. Run unit tests and test using the development server
7. Switch to a production setup with mod_wsgi, and the new virtualenv

This article covers all steps but the mod_wsgi setup, which will have its specific followup.

All instructions and command-line examples are biased towards Ubuntu.

Install pythonbrew and Python 2.7

pythonbrew automates the building and installation of Python in the users $HOME. It is very convenient in order to e.g. install versions of python newer than those supported by your OS version (in my case the server still runs Ubuntu 9.04, which uses python 2.6).

Since pythonbrew builds python on your system, and I will need sqlite3 support, I will need to install its development packages (I know this because I failed earlier). Same applies to libxml and bz2 development packages, because I’ll be using lxml and bz2. While I’m at it, I’ll also install curl:

sudo aptitude install libsqlite3-dev libbz2-dev libxml2-dev libxslt-dev curl

Installing pythonbrew:

$ curl -kL http://github.com/utahta/pythonbrew/raw/master/pythonbrew-install | bash
$ echo "source $HOME/.pythonbrew/etc/bashrc" >> ~/.bashrc

… and reload the shell for the command to be available.

Installing python 2.7 with pythonbrew may take quite a while, since it downloads and compiles from source (and runs tests):

$ pythonbrew install --verbose 2.7.2

After that, you may activate and run your new python interpreter in the current shell with:

$ python -V
Python 2.6.2
$ pythonbrew use 2.7.2
Using `Python-2.7.2`
$ python -V
Python 2.7.2

Install virtualenv and virtualenvwrapper

virtualenv is a tool to create isolated Python environments. An isolated environment includes a specific version of python plus a set of installed packages. With virtualenv you can easily manage multiple such environments with controlled subsets of package versions in the same machine.

virtualenvwrapper is a set of extensions to virtualenv to simplify the organisation and switching between multiple virtual environments.

Install on linux:

sudo aptitude install python-virtualenv
pythonbrew use 2.7.2
pip install virtualenvwrapper

[EDIT: make sure that you install virtualenvwrapper within each of your python brews, i.e. your specific version was activated (with pythonbrew use) before installing.]

Create a directory to hold your virtual environments:

mkdir ~/.virtualenvs

… then edit ~/.bashrc, and add at the end:

export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=$HOME/.pythonbrew/pythons/Python-2.7.2/bin/python
source $HOME/.pythonbrew/pythons/Python-2.7.2/bin/virtualenvwrapper.sh

Relogin, or apply with . .bashrc

Create the new virtual environment

We are now ready to create and switch to the new virtual environment. We want to make sure it will use the proper python we installed earlier with pythonbrew.

$ pythonbrew use 2.7.2
Using `Python-2.7.2`
$ mkvirtualenv --no-site-packages fabula
(fabula)$ python -V
Python 2.7.2
(fabula)$ which python
/home/carles/.virtualenvs/fabula/bin/python

(Note that virtualenv adds the name of the active environment within parenthesis in your prompt, for easy identification.)

Additional goodies… Install yolk, so that we can obtain a useful list of installed packages.

$ pip install yolk
$ yolk -l
Python          - 2.7.2        - active development (/home/carles/.pythonbrew/pythons/Python-2.7.2/lib/python2.7/lib-dynload)
pip             - 1.0.1        - active 
setuptools      - 0.6c11       - active 
wsgiref         - 0.1.2        - active development (/home/carles/.pythonbrew/pythons/Python-2.7.2/lib/python2.7)
yolk            - 0.4.1        - active 

Install the project’s dependencies

All of the required packages are in the project’s requirements.txt file used by pip (if you’re still not using this, you should). My requirements file contains:

django==1.2.5
South
lxml

Installing all of these with pip is a breeze.

(fabula)$ pip install -r requirements.txt

We can check that all’s fine, and we have separate environments as expected (deactivate and pythonbrew off are virtualenv’s and pythonbrew’s respective commands to leave the controlled environment and return to default versions):

# go back to default versions
(fabula)$ deactivate 
$ pythonbrew off
$ python -V
Python 2.6.2
$ python -c "import django; print(django.VERSION)"
(1, 1, 0, 'final', 0)

# reactivate controlled environments
$ workon fabula
(fabula)$ python -V
Python 2.7.2
(fabula)$ python -c "import django; print(django.VERSION)"
(1, 2, 5, 'final', 0)

We’re cool! Let’s proceed with testing our new version of the application.

Migrate data and test

Our goal now is to set up a different database than production, migrate it and run our tests. We will test the migrations with a replica of production data, by exporting the production database and importing it in the staging site. With postgreSQL replicating the DB may be as easy as:

create database fabula_v11 with template fabula_v10;

We change our settings in staging to point to the new DB (DATABASE_NAME=fabula_v11), and perform the validations (manage is my alias for django’s manage.py/django-admin.py command):

(fabula)$ manage migrate
(fabula)$ manage test
.......................................................................................................
----------------------------------------------------------------------
Ran 103 tests in 15.033s
(fabula)$ manage runserver 0.0.0.0:8186

… and we have a test server running on port 8186, we may open our browser to http://hostname:8186/admin/ and voilà! there we have a running test site prefilled with production data that we may further test. Once we have verified that the site functionally works, we can stop the development server and get ready to go live with a real production setup with mod_wsgi, which I will cover on a follow-up article .

• deployment
• django
• mod_wsgi
• python
• pythonbrew
• virtualenv