Chris Adams

At work we're busy trying to get a Django site out the door. This time around, we've been enjoying the modest time invested in setting up a Hudson continuous integration server - see Chris Shenton's presentation here for why you should and how quickly you can - and one of the areas we've really expanded was the use of automated testing. I've already described the test runner we're using but wanted to describe the overall process which we're using.

First, some background notes:

• Create a script which does all of the hard work and manage that as part of your project - some of the examples show people dumping 20+ line shell scripts into a Hudson config but if you're serious, this should be versioned like everything else. If you're careful, some of the setup tasks can even be shared with other scripts you use to setup new developers or create RPMs.
• Our process relies on virtualenv and pip. If you're not familiar with these, all you need to know in order to follow along is that virtualenv creates a virtual Python instance which allows us to keep this project separate from everything else and avoids the need for pip to have privileged access to install software.

Roughly in order, this is what our automated job does:

1. If our virtualenv doesn't exist or requirements.pip has changed since we initialized the virtualenv, remove it and recreate it. In Bash this is roughly:

   if [ ! -d .virtualenv -o requirements.pip -nt .virtualenv ]; then 
     rm -r .virtualenv;
     virtualenv .virtualenv;
     pip install -r requirements.pip --environment=.virtualenv --download-cache=.pip-download-cache;
   fi
   

   One important note: using the download cache makes your installs a lot faster and avoids wasting other people's resources on the distribution servers.
2. To avoid issues with a failure leaving the database in an inconsistent state, we drop and recreate the database before every run and clear the Solr full-text search index.
3. Start Solr as a background task:
   java -DSTOP.PORT=<arbitrary high port> -DSTOP.KEY=<arbitrary key> -jar start.jar
4. django-admin.py syncdb
5. django-admin.py loaddata clean_site (on our projects, we name fixtures clean_site rather than initial_data to avoid overwriting changes when syncdb runs)
6. At this point, we're ready to actually run the tests, which we do using our custom test runner which runs our full Django test suite, saving the output and coverage.py's report to a directory which is available through a local Apache instance for convenience. This also generates coverage.py's XML report so the Hudson Cobertura plugin can generate pretty charts showing our progress over time.
   We save the return code from the test suite (i.e. TEST_RC=$?) so we can report failures after running our cleanup code (see below) 
7. Assuming that the test suite ran correctly, we then launch some additional tests using Eric Holscher's excellent django-test-utils:
   django-admin.py crawlurls -v0 > logdir/crawler.log
   This also allows us to collect some basic performance numbers - I want to start visualizing per-page performance using something like dygraphs but we haven't had time to set that up yet.
8. Shutdown Solr:
   java -DSTOP.PORT=<arbitrary high port> -DSTOP.KEY=<arbitrary key> -jar start.jar --stop
9. Exit with the value returned by the Django test suite

That might sound like a lot of work but on our test system it currently takes well under 5 minutes. In addition to helping us stay on top of test coverage it's been really helpful for flushing out obsolete fixture data (i.e. crawurls will show 404 links) and has alerted us to several upstream version changes - we use pip freeze to track version numbers so we've found out quickly when the version of something we're using has been removed from PyPI. Most importantly, we know that our install instructions actually work because we're testing them on a regular basis - when something changes, it breaks quickly and is linked directly to a commit, making it easy to update the instructions and the deployment script - and when the time comes to put the code into production there's no question that the script is accurate because it's been run hundreds of times.