Jan 28

Reading for the modern web

A coworker recently asked for some JavaScript pointers and I figured I'd pass them along to everyone else

JavaScript the Good Parts is a classic book, if you're into dead trees and want to learn why some people love JavaScript. You can get a lot of the philosophy of modern JavaScript wizardry for free from Douglas Crockford (YUI) and John Resig (jQuery) and there's a ton of good stuff covering everything from philosophy to in-the-trenches coding available from YUI Theater.

DailyJS.com is a good resource for staying abrest of what's going on (not a huge fan of ajaxian.com's style); the 24ways.org web advent calendar was a good survey of where modern web development's going.

Finally, Google's closure library has some interesting tools (Closure Compiler is great) and the articles explaining their philosophy have a lot to learn from even if you don't use the Closure library; there's also a lot of modern performance wisdom summarized in Let's make the web faster, some directly JavaScript related and most of the rest of interest to any front-end developer.

Jan 26

Django: reliably saving URLs to a file field

In the hopes that this saves someone else the hassle, I recently ran into a bit of an inconvenience while importing remote URLs into Django ImageFields. It's easy to work with local files - simply wrap them in django.core.files.File() objects - but File can't handle the file-like objects returned by urlopen (see http://code.djangoproject.com/ticket/8501) which don't implement all of the methods supported by a normal file.

Most of the suggestions you'll find online suggest using urllib.urlretrieve to save the remote URL to a temporary file which you can then wrap normally. Unfortunately, urlretrieve does no error handling by default and it's a bit contorted to add so I found it easier to use Django's NamedTemporaryFile class:
from django.core.files import File
from django.core.files.temp import NamedTemporaryFile

img_temp = NamedTemporaryFile(delete=True)
img_temp.write(urllib2.urlopen(url).read())
img_temp.flush()

im.file.save(img_filename, File(img_temp))
Jan 08

Washington DC 1851

A cool Google Maps combp with an overlay of an 1851 DC map. Check out the areas like Tiber creek which used to be water and were reclaimed.

Visit: http://rumsey.geogarage.com/maps/gct000745b.html

Dec 24

Web Toolbox

I've collected some helpful tools for web developers

I've built collected various website testing tools into a webtoolbox repository on Github. This includes my earlier red_spider work as well as a few other utilities which have come in handy:

red_spider

A spider based on Mark Nottingham's redbot: it will produce a nice HTML report of page cacheability and, optionally, HTML validation; since it uses the same nbhttp library it's pretty fast, too. There are a number of options for filtering and it allows you to save lists of page and media URLs for use with tools like wk-bench or tornado-bench.

log_replay

If you need to replace webserver log files at something approximating realtime, log_replay is your friend. It uses Tornado's non-blocking HTTP client (based on pycurl - at some point it would be good to refactor down to just that) to fetch all of the URLs but will sleep any time it's too far ahead of the simulated virtual time.

tornado-bench

Also uses Tornado's non-blocking HTTP client, this program simply takes a big list of URLs and simply retrieves them as quickly as possible.

wk-bench

Mac OS X-specific tool which measures user-perceived page-load performance. It uses PyObjC to load a full WebKit browser, processes a list of URLs and reports the time taken from beginning the HTTP request until the browser fires the didFinishLoadForFrame event, which includes things like image loading, Flash, JavaScript, etc. This is also useful for reporting JavaScript errors as they are logged to the console and can very easily be extracted for verifying that you don't have on-load errors site-wide.

Dec 23

Mark Lynas: What happened in Copenhagen

Copenhagen was a disaster. That much is agreed. But the truth about what actually happened is in danger of being lost amid the spin and inevitable mutual recriminations. The truth is this: China wrecked the talks, intentionally humiliated Barack Obama, and insisted on an awful "deal" so western leaders would walk away carrying the blame. How do I know this? Because I was in the room and saw it happen.

— Mark Lynas

Dec 18

Django memory profiling middleware

Used in a hurry to profile the memory usage of a Django app

Visit: http://gist.github.com/198154