Chris Adams

The Washington Post is running a classic example of someone's damage control masquerading as actual journalism. In this case the agenda is that of web publishers who have inadvertently published confidential information - rather than admitting such gross incompetence they're claiming that this is a new security problem created by Google, et al.

The entire article is based on the false assumption that the problem is in some way caused by search engines. By definition web search engines can only see public content - Google has neither password crackers nor an army of humans working to bypass access control systems. This is hardly a new concern, either - security has been a feature of HTTP since the very beginning and the concept is something you'll encounter in all but the most cursory introductions to web authoring. The author attempts to create the appearance of controversy by including quotes from Homeland Security and the FBI and invalid assertions ("It is unclear who is at fault when someone digs up a confidential document") but it's really quite simple: if you publish sensitive information you're responsible, not the party who discovers your mistake.

Worse yet, the only advice we get is security by obscurity: edit robots.txt so Google won't index your site. This is worse than doing nothing in many ways because malicious types have been known to spider the excluded directories to see what's being concealed - akin to posting a large neon sign requesting that people don't steal the valuables you left sitting on the sidewalk.