Chris Adams

I've moved my earlier speed tracer middleware from my django-sugar fork to a separate repository: http://github.com/acdha/django-speedtracer

The modern web browsers have increasingly good tools for monitoring client-side performance - perhaps the best being the WebKit Timeline but there's only so much you can improve when most of the processing happens on the server side.

For those of us writing Django apps, there are fairly good tools for monitoring database performance by tracking slow queries (e.g. django-debug-toolbar or django-devserver) but there's not much else out there for measuring any of the other sources of delay (filesystem or network I/O, inefficient Python code, etc.) except the lower-level profile module. Most of these tools have the problem that getting useful data out requires a bit of work manipulating data, filtering out noise and assigning blame to the correct place (e.g. it's more likely that your code is calling an API poorly rather than that underlying API's implementation) - simply sorting a two-dimensional list is not enough for a large application.

One of the challenges for building better tools is building an interface which can interact with potentially very large amounts of trace data. Fortunately, Google's Chrome team added server-side trace support to SpeedTracer, making it possible for us to focus entirely on the problem of getting that data and simply reuse the work which the Chrome team has already done building the SpeedTracer UI and making it fast. Best of all, it uses only HTTP headers so there's no way for it to interfere with your sites' generated HTML and it works for any type of resource as long as it's served by Django.

Getting Started

1. Download and install Speed Tracer
2. Download and install my django-sugar fork - e.g. pip install git://github.com/acdha/django-sugar.git
3. Add "sugar" to your INSTALLED_APPS
4. Add "sugar.middleware.speedtracer. SpeedTracerMiddleware" to your MIDDLEWARE_CLASSES
5. Load your page inside Chrome with SpeedTracer enabled
6. Open SpeedTracer and expand the "Server Trace" in the page's detailed report which should look something like the picture above

Next steps

This has already proven itself to be somewhat handy for me at work but there's obvious improvement which could be made: this really should be converted into WSGI middleware so any Python-based webapp could benefit.

Inspiration

Thanks to Ilya Grigorik for his blog post about his racks-speedtracer module which inspired a night of tinkering last month. No thanks to my memory for not hitting publish sooner…

If you work on sites with complex JavaScript you've probably wanted a way to know about the errors reported by users' browsers: even with a rigorous test process it's likely that there's some permutation of browser version and settings which you don't test, particularly when you consider external factors like JavaScript from third-party sources or the many ways in which anti-virus software, corporate proxies and policies can interact in darkly obscure ways.

There's now a really easy way to collect JavaScript errors thanks to Andy McKay and Clearwind Consulting: Arecibo. It's available as a commercial service for people who need support but it's also a completely open-source project on Github. Recently I've been working on an improved JavaScript client which has now merged into the official codebase, making it really easy to setup a personal error aggregation service up for all of your projects:

1. Set up an Arecibo service on AppEngine following the installation guide
2. Add this JavaScript fragment to your HTML templates:

   <script src="http://your-arecibo.appspot.com/lib/error.js" type="text/javascript" charset="utf-8"></script>
   <script type="text/javascript" charset="utf-8">
       arecibo.account = 'YOUR PUBLIC API KEY';
       arecibo.registerGlobalHandler();
   </script>
           

   It's often desirable to defer loading Arecibo until after the rest of your page has displayed which can be done using something like this example with jQuery:

   <script type="text/javascript" charset="utf-8">
       jQuery(function($) {
           $.getScript("http://your-arecibo.appspot.com/lib/error.js", function () {
               arecibo.account = 'YOUR PUBLIC API KEY';
               arecibo.registerGlobalHandler();
           });
       });
   </script>
           

The reporting interface looks like this:

There are two general caveats here: this service can't collect data when JavaScript is completely disabled or when the problem is caused by internet connectivity issues. Unfortunately browser error handling is also not standardized and WebKit browsers like Safari and Chrome currently don't have a way to capture unhandled exceptions; similarly, attempts to collect detailed stack traces varies from browser to browser so you'll find richer error reports from Firefox than Internet Explorer but in most cases simply getting the report is enough to start working on a fix or at least a more exhaustive test.

I often need to work with both Git and Mercurial repositories. I've previously used hg-git to work in Hg and push changes to Git but have found Mercurial to be less comfortable than Git (no flames, please: this is the newer vi-Emacs debate. Use the one you like) and was hoping to work in the opposite manner: local changes in Git pushed to an Hg repo on BitBucket or Google Code.

Travis Cline posted some instructions for working in Git and pushing changes to Hg which I've updated with a few stylistic tweaks:

• Install hg-git (e.g. ``pip install hg-git``)
• Make sure you've enabled the Hg bookmark extension in your ``.hgrc``
• Add this to your .hgrc::

      [git]
      intree=1
  

• Clone your Mercurial repo::

      $ hg clone https://acdha@bitbucket.org/ned/coveragepy
  

• Change into the repo::

      $ cd coveragepy
  

• Create a local bookmark tracking your Mercurial default branch - this is what will be exported to Git::

     
      $ hg bookmark hg/default -r default
  

• Export to the git repo::

      $ hg gexport
  

• Configure Hg to ignore the Git repo::

      $ echo ".git" >> .hg/hgignore
  

• Configure Git to ignore the Hg repo::

      $ echo ".hg*" >> .git/info/exclude
  

• Configure Git to ignore the same things as Mercurial::

      $ git config core.excludesfile `pwd`/.hg/hgignore
  

• Have your master branch track the exported Hg default branch::

      $ git branch --track hg/default master
      $ git reset --hard
  

• Do stuff in Git and make commits
• Export your changes to Hg::

      $ hg gimport
      
  

• Push them out to the world::

      $ hg push
  

I see the usual concerns about the government collecting data about citizens are making the rounds again, with a recent Wired post about national ID cards and a concerned editorial in the CS Monitor about Census data. This sort of thing comes up regularly in both the generally left-leaning privacy circles and the right-leaning small-government crowd and I find it interesting that the discussion almost inevitably revolves around strategies for preventing the government from collecting data.

What I find surprising is that the discussion always seems to be based on the misunderstanding that this is a future threat, as opposed to something which has been routine for decades. I'm not referring to just SSNs, although they're closed enough to a national ID to make these discussions obsolete, but simply the fact that there are already many large databases containing various bits of personal data and since the invention of the digital computer (or at least the relational database) there's been no privacy advantage to be gained by avoiding a number since anyone can trivially cluster data on characteristics like name, age, address, etc. Academic research has demonstrated that much of our sense of privacy is illusory - something as simple as the name, zipcode and birthdate or approximate home and work locations suffices to uniquely identify most people. At this point, campaigning against a national ID card devolves into the case that abuse will be stopped because the suspect agencies aren't capable of basic database use.   

Since we know that large scale, effective data-mining is already going on - not every agency shares TSA's reliance on WWI-era technology in excel and not all of the data mining is being done by public agencies - the better question is what a reasonable expectation of privacy is in the post-Google era and how we can deter abuse while taking advantage of the benefits offered by modern computing¹. Since data is so easy to collect and mine, we really should be discussing acceptable use of data and the penalties for abuse, covering both government and corporate use. If we had European-style privacy laws, a national ID card could be discussed as the obvious good idea it is rather than the proxy for a litany of tangentially-related fears.

1. Does anyone really want to make the case that government shouldn't use effective tools? Instead of fighting the Census we should be trying to figure out how it could be updated yearly or better so policy can be adjusted for trends on less than a decadal timescale.

I've been posting post-storm pictures on Flickr. Some favorites: