Last.fm has presence in three datacenters in the London area - this is currently more because of necessity than redundancy; datacenter space is at a real premium in the UK and we can’t fulfill our power growth needs at just one site. We operate our own 20 gigabit fibre ring between these, which means we can basically treat all of our London presence as one site from a latency perspective. We don’t yet have enough redundancy to tolerate the complete loss of a site with no effect to the service (but we’re getting there).

Our largest (provisioned) site is at the Level3 facility in Braham Street, near Aldgate. It has fairly standard security, with the appropriate number of security gimmicks; I enter using a proximity card and PIN, take the lift up to our floor, and then get onto the floor using my card and a hand scan.

At 4am on Monday of this week, the Braham Street facility suffered a break-in. Three men managed to batter down an external fire escape door, made their way to our floor, and broke down the door to the data floor. Then then proceeded to try and break into a suite. They failed to get into one suite, leaving the door pretty mangled, and so they moved onto the one next door. Our suite.

The robbers succeeded in breaking our door down. They then made their way to the back of our suite, and picked out a very specific rack: the one which holds our core router for that site, a Cisco 6500-series machine. I think they probably knew what they were looking for; these routers contain several cards which are probably the most lightweight, valuable items we have. These are popular things to steal.

The only thing which stopped the entirety of Last.fm going down on Monday morning was the robbers not spotting that the door to that rack was unlocked. They had started to crowbar the door off when site security and the police apprehended them. They were taken to court the following day and pled guilty, sentencing to follow.

That was a bit too close for comfort.

The scary thing is that this isn’t the first time Braham Street has been burgled. In 2006 the thieves were successful and managed to steal cards out of Level3’s own routers, bringing down part of their London network. Also in 2006, thieves simply walked into the Easynet (formerly Interxion) facility in Brick Lane and loaded up a van with £6m worth of kit. More recently, there have been two burglaries at BT telephone exchanges in the London area, where the thieves also came out with a tidy number of cards from the routers powering their new 21st century network.

I guess the moral of this story is that even if you think you’re resilient against everything, are you resilient against thieves walking in and stealing bits of your network?