Running PyPy in a virtualenv

This is a quick guide to running your [python/django] project on PyPy, the fast JIT-based Python interpreter (and optionally benchmark re: cPython)

Install PyPy

Follow the clearly detailed instructions in: http://pypy.readthedocs.org

For example, for 32-bit linux I used:

$ wget https://bitbucket.org/pypy/pypy/downloads/pypy-1.8-linux.tar.bz2
$ tar xvf pypy-1.8-linux.tar.bz2

This will create a folder named pypy-1.8, which matches Python 2.7.2:

$ ./pypy-1.8/bin/pypy --version
Python 2.7.2 (0e28b379d8b3, Feb 09 2012, 19:41:19)
[PyPy 1.8.0 with GCC 4.4.3]

Create your virtual environment

Install distribute and pip for pypy:

NOTE: These instructions no longer work.
http://python-distribute.org is a domain for sale.

$ # wget http://python-distribute.org/distribute_setup.py  ## OBSOLETE
$ wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py
$ ./pypy-1.8/bin/pypy distribute_setup.py
$ ./pypy-1.8/bin/pypy get-pip.py

Make sure you have at least virtualenv 1.6, else it won’t work with PyPy:

$ virtualenv --version
1.6.4

Install virtualenvwrapper, and create a new virtual environment for your projectx:

$ ./pypy-1.8/bin/pip install virtualenvwrapper
$ mkvirtualenv --no-site-packages --distribute --python=/path/to/pypy-1.8/bin/pypy projectx-pypy
$ python --version
Python 2.7.2 (0e28b379d8b3, Feb 09 2012, 19:41:19)
[PyPy 1.8.0 with GCC 4.4.3]

Now we have a virtual environment named projectx-pypy based on pypy that we can use as just any other virtualenv.

Setup your project

Install your project’s requirements as usual:

$ workon projectx-pypy
$ pip install -r requirements.txt -r test-requirements.txt

Benchmark if desired

Now I can easily compare timings between cPython and PyPy. I created another identical virtualenv based on cPython (2.7.2) and run the same tests (one testcase replicated 100000 times using nose test generators, just to have something time consuming and CPU-intensive).

PyPy:

$ workon projectx-pypy
$ nosetests
...
----------------------------------------------------------------------
Ran 100000 tests in 8.624s

cPython:

$ workon projectx
$ nosetests
...
----------------------------------------------------------------------
Ran 100000 tests in 38.180s

Of course this is not a representative sample, but just a simple test. In this one case, PyPy takes approximately 20% the execution time of cPython. Not bad, huh?

The PyPy people have more comprehensive benchmarks in their speed center.

So give PyPy a try, and share your results.

Publishing Django Test Coverage Reports in Jenkins

This post is built on some assumptions.

First, I assume that you already know that writing unit tests is good for you. Well, to be honest, if you are not systematically writing tests for your code, you shouldn’t be calling yourself a software engineer anyway. No excuses.

In consequence I also assume that your latest Django project includes its dose of unit testing. But do you have a clear idea of which parts of your Django site are not being tested? Are you taking action to improve on that area? In other words, are you already obtaining and analysing coverage data for your project?

If so, lucky you. I didn’t, decided it was about time, and set out to the task.

I will try to demistify the process, since it takes very little effort and you can reap substantial benefits from it – provided, of course, that you take a look at the coverage reports on a regular basis, and add tests for the uncovered methods… but you promise you will do that, won’t you? Great!

We will start by generating the reports manually, and will then move on to automating them into Jenkins, our friendly build butler.

Running our Django site with mod_wsgi and virtualenv (part 2)

These are the steps remaining from our previous article. In order to complete our desired setup we must configure Apache with mod_wsgi pointing to the new virtual environment.

But before we can even do that, we need to setup mod_wsgi, which in our case will require building it and installing it from source.

Setting up mod_wsgi with our virtual Environment

Detailed explanations for using virtual environments in mod_wsgi can be found here.

Just by reading over a bit, and based on some prior experience (2+ years ago, though), I was expecting this to be the main pain area of the whole process. From the docs I read that mod_wsgi has to be compiled against the same version of Python your code will be running under. Which means I will have to build from source, since Ubuntu 9.04’s version of mod_wsgi is linked with its included version of Python 2.6.

Build mod_wsgi

First, we must build and install mod_wsgi to use the correct version of Python. The installation guide for mod_wsgi is very clear, you just need to follow it. Below are the distilled steps my installation needed.

We start by installing the Apache2 development libraries (which we will need in order to install mod_wsgi), and downloading mod_wsgi from its subversion repository.

sudo apt-get install apache2-dev
svn co http://modwsgi.googlecode.com/svn/branches/mod_wsgi-2.X mod_wsgi-2.x
cd !$

Setting up our Django site environment with pythonbrew and virtualenv

My problem: I have a server running my Django apps that I set up when I had very little experience with Django, and just some more with Python. The configuraton is using Apache and mod_wsgi on Ubuntu, running on the system’s default Python installation (2.6) which has Django 1.1 installed.

I need to update my site, which has been upgraded to use Django 1.2, and would benefit from Python 2.7 (although the site itself runs on 2.6 as well, running the tests requires 2.7, since I use some of the new convenient unittest assertions from that version, e.g. assertIsNotNone or assertIn).

Upgrading the site will require some migrations, which are not backwards compatible (e.g. some table and field renames). Since all in all will be a delicate intervention, I’d like to create a totally separate environment for testing the revamped site while the previous version is still online. Even if it was not risky, now I know better than when I started, and it’s a good time to switch to a virtual environment setup.

Therefore, in order to proceed we will:

1. Install python 2.7 with pythonbrew
2. Install virtualenv and virtualenwrapper
3. Create a virtualenv with that python version
4. Install all requirements with pip
5. Make a Copy of the production DB, and migrate
6. Run unit tests and test using the development server
7. Switch to a production setup with mod_wsgi, and the new virtualenv

This article covers all steps but the mod_wsgi setup, which will have its specific followup.