My problem: I have a server running my Django apps that I set up when I had very little experience with Django, and just some more with Python. The configuraton is using Apache and mod_wsgi on Ubuntu, running on the system’s default Python installation (2.6) which has Django 1.1 installed.
I need to update my site, which has been upgraded to use Django 1.2, and would benefit from Python 2.7 (although the site itself runs on 2.6 as well, running the tests requires 2.7, since I use some of the new convenient unittest assertions from that version, e.g. assertIsNotNone or assertIn).
Upgrading the site will require some migrations, which are not backwards compatible (e.g. some table and field renames). Since all in all will be a delicate intervention, I’d like to create a totally separate environment for testing the revamped site while the previous version is still online. Even if it was not risky, now I know better than when I started, and it’s a good time to switch to a virtual environment setup.
Therefore, in order to proceed we will:
- Install python 2.7 with
pythonbrew - Install
virtualenvandvirtualenwrapper - Create a
virtualenvwith that python version - Install all requirements with
pip - Make a Copy of the production DB, and migrate
- Run unit tests and test using the development server
- Switch to a production setup with
mod_wsgi, and the newvirtualenv
This article covers all steps but the mod_wsgi setup, which will have its specific followup.
All instructions and command-line examples are biased towards Ubuntu.
Install pythonbrew and Python 2.7
pythonbrew automates the building and installation of Python in the users $HOME. It is very convenient in order to e.g. install versions of python newer than those supported by your OS version (in my case the server still runs Ubuntu 9.04, which uses python 2.6).
Since pythonbrew builds python on your system, and I will need sqlite3 support, I will need to install its development packages (I know this because I failed earlier). Same applies to libxml and bz2 development packages, because I’ll be using lxml and bz2. While I’m at it, I’ll also install curl:
sudo aptitude install libsqlite3-dev libbz2-dev libxml2-dev libxslt-dev curl
Installing pythonbrew:
$ curl -kL http://github.com/utahta/pythonbrew/raw/master/pythonbrew-install | bash $ echo "source $HOME/.pythonbrew/etc/bashrc" >> ~/.bashrc
… and reload the shell for the command to be available.
Installing python 2.7 with pythonbrew may take quite a while, since it downloads and compiles from source (and runs tests):
$ pythonbrew install --verbose 2.7.2
After that, you may activate and run your new python interpreter in the current shell with:
$ python -V Python 2.6.2 $ pythonbrew use 2.7.2 Using `Python-2.7.2` $ python -V Python 2.7.2
Install virtualenv and virtualenvwrapper
virtualenv is a tool to create isolated Python environments. An isolated environment includes a specific version of python plus a set of installed packages. With virtualenv you can easily manage multiple such environments with controlled subsets of package versions in the same machine.
virtualenvwrapper is a set of extensions to virtualenv to simplify the organisation and switching between multiple virtual environments.
Install on linux:
sudo aptitude install python-virtualenv pythonbrew use 2.7.2 pip install virtualenvwrapper
[EDIT: make sure that you install virtualenvwrapper within each of your python brews, i.e. your specific version was activated (with pythonbrew use) before installing.]
Create a directory to hold your virtual environments:
mkdir ~/.virtualenvs
… then edit ~/.bashrc, and add at the end:
export WORKON_HOME=$HOME/.virtualenvs export VIRTUALENVWRAPPER_PYTHON=$HOME/.pythonbrew/pythons/Python-2.7.2/bin/python source $HOME/.pythonbrew/pythons/Python-2.7.2/bin/virtualenvwrapper.sh
Relogin, or apply with . .bashrc
Create the new virtual environment
We are now ready to create and switch to the new virtual environment. We want to make sure it will use the proper python we installed earlier with pythonbrew.
$ pythonbrew use 2.7.2 Using `Python-2.7.2` $ mkvirtualenv --no-site-packages fabula (fabula)$ python -V Python 2.7.2 (fabula)$ which python /home/carles/.virtualenvs/fabula/bin/python
(Note that virtualenv adds the name of the active environment within parenthesis in your prompt, for easy identification.)
Additional goodies… Install yolk, so that we can obtain a useful list of installed packages.
$ pip install yolk $ yolk -l Python - 2.7.2 - active development (/home/carles/.pythonbrew/pythons/Python-2.7.2/lib/python2.7/lib-dynload) pip - 1.0.1 - active setuptools - 0.6c11 - active wsgiref - 0.1.2 - active development (/home/carles/.pythonbrew/pythons/Python-2.7.2/lib/python2.7) yolk - 0.4.1 - active
Install the project’s dependencies
All of the required packages are in the project’s requirements.txt file used by pip (if you’re still not using this, you should). My requirements file contains:
django==1.2.5 South lxml
Installing all of these with pip is a breeze.
(fabula)$ pip install -r requirements.txt
We can check that all’s fine, and we have separate environments as expected (deactivate and pythonbrew off are virtualenv’s and pythonbrew’s respective commands to leave the controlled environment and return to default versions):
# go back to default versions (fabula)$ deactivate $ pythonbrew off $ python -V Python 2.6.2 $ python -c "import django; print(django.VERSION)" (1, 1, 0, 'final', 0) # reactivate controlled environments $ workon fabula (fabula)$ python -V Python 2.7.2 (fabula)$ python -c "import django; print(django.VERSION)" (1, 2, 5, 'final', 0)
We’re cool! Let’s proceed with testing our new version of the application.
Migrate data and test
Our goal now is to set up a different database than production, migrate it and run our tests. We will test the migrations with a replica of production data, by exporting the production database and importing it in the staging site. With postgreSQL replicating the DB may be as easy as:
create database fabula_v11 with template fabula_v10;
We change our settings in staging to point to the new DB (DATABASE_NAME=fabula_v11), and perform the validations (manage is my alias for django’s manage.py/django-admin.py command):
(fabula)$ manage migrate (fabula)$ manage test ....................................................................................................... ---------------------------------------------------------------------- Ran 103 tests in 15.033s (fabula)$ manage runserver 0.0.0.0:8186
… and we have a test server running on port 8186, we may open our browser to http://hostname:8186/admin/ and voilà! there we have a running test site prefilled with production data that we may further test. Once we have verified that the site functionally works, we can stop the development server and get ready to go live with a real production setup with mod_wsgi, which I will cover on a follow-up article .
Can’t you use pythonbrew venv for virtual environments? just getting to know pythonbrew
I’ll try that option, it looks interesting, I’d need to see how it works when dealing with virtual environments for python versions that have not necessarily been installed with pythonbrew.
But virtual environments are my bread and butter in machines where I don’t need to use pythonbrew. Therefore I prefer to use the virtualenvwrapper commands directly. They also require less typing (e.g.
workonvspythonbrew venv).hi again,
In your example you have installed a single python version with pythonbrew. But how could i use your system with several python versions installed?
In other words, i’m interested in having virtualenvs linked to a python path according to the pythonbrew’s in-use-python at the time of the creation of those same virtualenvs.
Don’t know if i made myself clear.
I see no problem in creating as many pythonbrew installs as desired, and different virtual environments, one or more for each of the pytonbrew installs.
You only have to install a new python with pythonbrew, and then create a virtualenv to point to that specific python version. You can either do it as in my example (pythonbrew use and then mkvirtualenv), or use mkvirtualenv –python=/path/to/python
I found that your method as described (“pythonbrew use …” then “mkvirtualenv”) did not pick up the ‘current’ version of python – it reverted to the system python. I had to use “pythonbrew use 2.7.2 ; mkvirtualenv –python=python2.7″ instead, which seemed to work.
Found out that my problem was due to a previously system-wide installed pip (which i didn’t know about), which messed up with the built-in pythonbrew’s pip.
I added an edit a few days ago because I also found myself struggling with this issue. The virtualenv wrapper must be installed for each of your python brews. If you call “which mkvirtualenv” it will tell you where it is running from. If it is not installed for your brewed python 2.7.2 it will point to the one in the system python, and will create a virtualenv for that version
That seemed to fix it – thanks. I just had to do ‘pip install virtualenvwrapper’ for each pythonbrew. Note that I didn’t need to set VIRTUALENVWRAPPER_PYTHON, but that might be because I’m using the same version for all brews. It could get ugly if I’m not.
I note that the path to ‘virtualenv’ changes with each brew, but ‘which mkvirtualenv’ doesn’t tell me anything as it’s a shell function. Anyway, thanks again, seems to be working better now.
Hi, so I am trying to set this up as well. Just like @diogo, I had pip system-wide install before. When I installed brewed python 3.2.3 it says no setuptools were installed. So I tought it might be conflicting with system wide install. Did you have to remove system-wide pip installation and virtualenvwrapper system-wide installation? I am stuck with the point in your post where you say that you install different virtualenvwrappers for different brews. Thanks!