Chris Adams

Mark Nottingham recently released redbot, a modern replacement for the classic cacheability tester. I've been using it at work to audit website performance before releases since proper HTTP caching makes an enormous difference in perceived site performance.

redbot is a focused tool and provides a great deal of detail about at most one page and, optionally, its resources. I wanted to expand the scope to testing an entire site and performing content validation and with a little work came up with red_spider.py, which produces a consolidated report like this.

I have a few ideas for the future, which should involve splitting the code into a separate project rather than a fork of redbot as it acquires more validation capabilities such as borrowing from something like collective.validator.css to validate CSS, RSS/Atom, etc., using PIL to verify that images don't have things like wasteful embedded thumbnails, and borrowing from my wk-bench experiment to load pages using WebKit and report JavaScript errors.