Setting up our Django site environment with pythonbrew and virtualenv

My problem: I have a server running my Django apps that I set up when I had very little experience with Django, and just some more with Python. The configuraton is using Apache and mod_wsgi on Ubuntu, running on the system’s default Python installation (2.6) which has Django 1.1 installed.

I need to update my site, which has been upgraded to use Django 1.2, and would benefit from Python 2.7 (although the site itself runs on 2.6 as well, running the tests requires 2.7, since I use some of the new convenient unittest assertions from that version, e.g. assertIsNotNone or assertIn).

Upgrading the site will require some migrations, which are not backwards compatible (e.g. some table and field renames). Since all in all will be a delicate intervention, I’d like to create a totally separate environment for testing the revamped site while the previous version is still online. Even if it was not risky, now I know better than when I started, and it’s a good time to switch to a virtual environment setup.

Therefore, in order to proceed we will:

  1. Install python 2.7 with pythonbrew
  2. Install virtualenv and virtualenwrapper
  3. Create a virtualenv with that python version
  4. Install all requirements with pip
  5. Make a Copy of the production DB, and migrate
  6. Run unit tests and test using the development server
  7. Switch to a production setup with mod_wsgi, and the new virtualenv

This article covers all steps but the mod_wsgi setup, which will have its specific followup.

All instructions and command-line examples are biased towards Ubuntu.

Install pythonbrew and Python 2.7

pythonbrew automates the building and installation of Python in the users $HOME. It is very convenient in order to e.g. install versions of python newer than those supported by your OS version (in my case the server still runs Ubuntu 9.04, which uses python 2.6).

Since pythonbrew builds python on your system, and I will need sqlite3 support, I will need to install its development packages (I know this because I failed earlier). Same applies to libxml and bz2 development packages, because I’ll be using lxml and bz2. While I’m at it, I’ll also install curl:

sudo aptitude install libsqlite3-dev libbz2-dev libxml2-dev libxslt-dev curl

Installing pythonbrew:

$ curl -kL http://github.com/utahta/pythonbrew/raw/master/pythonbrew-install | bash
$ echo "source $HOME/.pythonbrew/etc/bashrc" >> ~/.bashrc

… and reload the shell for the command to be available.

Installing python 2.7 with pythonbrew may take quite a while, since it downloads and compiles from source (and runs tests):

$ pythonbrew install --verbose 2.7.2

After that, you may activate and run your new python interpreter in the current shell with:

$ python -V
Python 2.6.2
$ pythonbrew use 2.7.2
Using `Python-2.7.2`
$ python -V
Python 2.7.2

Install virtualenv and virtualenvwrapper

virtualenv is a tool to create isolated Python environments. An isolated environment includes a specific version of python plus a set of installed packages. With virtualenv you can easily manage multiple such environments with controlled subsets of package versions in the same machine.

virtualenvwrapper is a set of extensions to virtualenv to simplify the organisation and switching between multiple virtual environments.

Install on linux:

sudo aptitude install python-virtualenv
pythonbrew use 2.7.2
pip install virtualenvwrapper

[EDIT: make sure that you install virtualenvwrapper within each of your python brews, i.e. your specific version was activated (with pythonbrew use) before installing.]

Create a directory to hold your virtual environments:

mkdir ~/.virtualenvs

… then edit ~/.bashrc, and add at the end:

export WORKON_HOME=$HOME/.virtualenvs
export VIRTUALENVWRAPPER_PYTHON=$HOME/.pythonbrew/pythons/Python-2.7.2/bin/python
source $HOME/.pythonbrew/pythons/Python-2.7.2/bin/virtualenvwrapper.sh

Relogin, or apply with . .bashrc

Create the new virtual environment

We are now ready to create and switch to the new virtual environment. We want to make sure it will use the proper python we installed earlier with pythonbrew.

$ pythonbrew use 2.7.2
Using `Python-2.7.2`
$ mkvirtualenv --no-site-packages fabula
(fabula)$ python -V
Python 2.7.2
(fabula)$ which python
/home/carles/.virtualenvs/fabula/bin/python

(Note that virtualenv adds the name of the active environment within parenthesis in your prompt, for easy identification.)

Additional goodies… Install yolk, so that we can obtain a useful list of installed packages.

$ pip install yolk
$ yolk -l
Python          - 2.7.2        - active development (/home/carles/.pythonbrew/pythons/Python-2.7.2/lib/python2.7/lib-dynload)
pip             - 1.0.1        - active 
setuptools      - 0.6c11       - active 
wsgiref         - 0.1.2        - active development (/home/carles/.pythonbrew/pythons/Python-2.7.2/lib/python2.7)
yolk            - 0.4.1        - active 

Install the project’s dependencies

All of the required packages are in the project’s requirements.txt file used by pip (if you’re still not using this, you should). My requirements file contains:

django==1.2.5
South
lxml

Installing all of these with pip is a breeze.

(fabula)$ pip install -r requirements.txt

We can check that all’s fine, and we have separate environments as expected (deactivate and pythonbrew off are virtualenv’s and pythonbrew’s respective commands to leave the controlled environment and return to default versions):

# go back to default versions
(fabula)$ deactivate 
$ pythonbrew off
$ python -V
Python 2.6.2
$ python -c "import django; print(django.VERSION)"
(1, 1, 0, 'final', 0)

# reactivate controlled environments
$ workon fabula
(fabula)$ python -V
Python 2.7.2
(fabula)$ python -c "import django; print(django.VERSION)"
(1, 2, 5, 'final', 0)

We’re cool! Let’s proceed with testing our new version of the application.

Migrate data and test

Our goal now is to set up a different database than production, migrate it and run our tests. We will test the migrations with a replica of production data, by exporting the production database and importing it in the staging site. With postgreSQL replicating the DB may be as easy as:

create database fabula_v11 with template fabula_v10;

We change our settings in staging to point to the new DB (DATABASE_NAME=fabula_v11), and perform the validations (manage is my alias for django’s manage.py/django-admin.py command):

(fabula)$ manage migrate
(fabula)$ manage test
.......................................................................................................
----------------------------------------------------------------------
Ran 103 tests in 15.033s
(fabula)$ manage runserver 0.0.0.0:8186

… and we have a test server running on port 8186, we may open our browser to http://hostname:8186/admin/ and voilà! there we have a running test site prefilled with production data that we may further test. Once we have verified that the site functionally works, we can stop the development server and get ready to go live with a real production setup with mod_wsgi, which I will cover on a follow-up article .

About these ads

, , , , ,

  1. #1 by diogo on 2011/07/30 - 14:12

    Can’t you use pythonbrew venv for virtual environments? just getting to know pythonbrew

    • #2 by Carles Barrobés on 2011/07/30 - 14:50

      I’ll try that option, it looks interesting, I’d need to see how it works when dealing with virtual environments for python versions that have not necessarily been installed with pythonbrew.

      But virtual environments are my bread and butter in machines where I don’t need to use pythonbrew. Therefore I prefer to use the virtualenvwrapper commands directly. They also require less typing (e.g. workon vs pythonbrew venv).

  2. #3 by diogo on 2011/09/13 - 17:51

    hi again,

    In your example you have installed a single python version with pythonbrew. But how could i use your system with several python versions installed?

    In other words, i’m interested in having virtualenvs linked to a python path according to the pythonbrew’s in-use-python at the time of the creation of those same virtualenvs.

    Don’t know if i made myself clear.

    • #4 by Carles Barrobés on 2011/09/14 - 20:25

      I see no problem in creating as many pythonbrew installs as desired, and different virtual environments, one or more for each of the pytonbrew installs.

      You only have to install a new python with pythonbrew, and then create a virtualenv to point to that specific python version. You can either do it as in my example (pythonbrew use and then mkvirtualenv), or use mkvirtualenv –python=/path/to/python

      • #5 by David on 2011/09/22 - 01:24

        I found that your method as described (“pythonbrew use …” then “mkvirtualenv”) did not pick up the ‘current’ version of python – it reverted to the system python. I had to use “pythonbrew use 2.7.2 ; mkvirtualenv –python=python2.7″ instead, which seemed to work.

      • #6 by diogo on 2011/09/30 - 08:54

        Found out that my problem was due to a previously system-wide installed pip (which i didn’t know about), which messed up with the built-in pythonbrew’s pip.

  3. #7 by Carles Barrobés on 2011/09/22 - 08:34

    I added an edit a few days ago because I also found myself struggling with this issue. The virtualenv wrapper must be installed for each of your python brews. If you call “which mkvirtualenv” it will tell you where it is running from. If it is not installed for your brewed python 2.7.2 it will point to the one in the system python, and will create a virtualenv for that version

    • #8 by David on 2011/09/23 - 01:32

      That seemed to fix it – thanks. I just had to do ‘pip install virtualenvwrapper’ for each pythonbrew. Note that I didn’t need to set VIRTUALENVWRAPPER_PYTHON, but that might be because I’m using the same version for all brews. It could get ugly if I’m not.

      I note that the path to ‘virtualenv’ changes with each brew, but ‘which mkvirtualenv’ doesn’t tell me anything as it’s a shell function. Anyway, thanks again, seems to be working better now.

  4. #9 by Amer (@BunnyHumping) on 2012/08/03 - 16:14

    Hi, so I am trying to set this up as well. Just like @diogo, I had pip system-wide install before. When I installed brewed python 3.2.3 it says no setuptools were installed. So I tought it might be conflicting with system wide install. Did you have to remove system-wide pip installation and virtualenvwrapper system-wide installation? I am stuck with the point in your post where you say that you install different virtualenvwrappers for different brews. Thanks!

  1. Measure of Justice » Blog Archive » Resources on Django and D3
  2. Setting up a virtual environment with Ipython, numpy and pandas | Measure of Justice
  3. Emacs IPython Notebook and “ESS in the Cloud” | Measure of Justice
  4. Ubuntu: Set up a virtual environment with Ipython, numpy and pandas | Measure of Justice

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: