Datacenter Security: A Cautionary Tale
Posted on March 12th, 2009 by Russ. Filed under Systems Admin.
Last.fm has presence in three datacenters in the London area – this is currently more because of necessity than redundancy; datacenter space is at a real premium in the UK and we can’t fulfill our power growth needs at just one site. We operate our own 20 gigabit fibre ring between these, which means we can basically treat all of our London presence as one site from a latency perspective. We don’t yet have enough redundancy to tolerate the complete loss of a site with no effect to the service (but we’re getting there).
Our largest (provisioned) site is at the Level3 facility in Braham Street, near Aldgate. It has fairly standard security, with the appropriate number of security gimmicks; I enter using a proximity card and PIN, take the lift up to our floor, and then get onto the floor using my card and a hand scan.
At 4am on Monday of this week, the Braham Street facility suffered a break-in. Three men managed to batter down an external fire escape door, made their way to our floor, and broke down the door to the data floor. Then then proceeded to try and break into a suite. They failed to get into one suite, leaving the door pretty mangled, and so they moved onto the one next door. Our suite.
The robbers succeeded in breaking our door down. They then made their way to the back of our suite, and picked out a very specific rack: the one which holds our core router for that site, a Cisco 6500-series machine. I think they probably knew what they were looking for; these routers contain several cards which are probably the most lightweight, valuable items we have. These are popular things to steal.
The only thing which stopped the entirety of Last.fm going down on Monday morning was the robbers not spotting that the door to that rack was unlocked. They had started to crowbar the door off when site security and the police apprehended them. They were taken to court the following day and pled guilty, sentencing to follow.
That was a bit too close for comfort.
The scary thing is that this isn’t the first time Braham Street has been burgled. In 2006 the thieves were successful and managed to steal cards out of Level3’s own routers, bringing down part of their London network. Also in 2006, thieves simply walked into the Easynet (formerly Interxion) facility in Brick Lane and loaded up a van with £6m worth of kit. More recently, there have been two burglaries at BT telephone exchanges in the London area, where the thieves also came out with a tidy number of cards from the routers powering their new 21st century network.
I guess the moral of this story is that even if you think you’re resilient against everything, are you resilient against thieves walking in and stealing bits of your network?
March 13th, 2009 at 00:13
This has certainly happened before in other parts of the world.
e.g. the multiple break-ins and physical coercions in Chicago. http://www.theregister.co.uk/2007/11/02/chicaco_datacenter_breaches/
There are more secure options available (ok, past the M25, but still – http://www.thebunker.net/). Has Last.fm ever considered these options or is it cheaper to be robbed and made whole again than it is to invest in prevention to begin with?
March 13th, 2009 at 00:29
Wow Russ – this is pretty brazen, even for professional IT thieves. Glad to hear nothing went missing!
March 13th, 2009 at 02:10
hmm next time you get a new space, you’d need to check physical protection as well (outside the main entrances… nobody’s going to go through PINs and biometric checks, they will go through the back door)
March 13th, 2009 at 02:11
Don’t you guys have 24/7 security at those facilities? The cost of the equipment in your racks is enough to warrant it. For some companies 10 minutes of downtime can cost them a huge amount of money. Perhaps Last.fm doesn’t fit that mold, but I’m honestly shocked there isn’t an on-site security guard for a multi-floor building of expensive computer equipment.
March 13th, 2009 at 03:37
I thought Cisco 6500 series were switches not routers? I believe they can also do layer 3 in hybrid mode.
March 13th, 2009 at 06:04
muriithi: These days you can run a Cat6500 in native IOS mode and do all the routing you need so long as you don’t need non-ethernet line cards or service provider features
(that stuff is restricted to the 7600. Same chassis just licensed and branded as a router).
March 13th, 2009 at 09:07
@heri: We were a bit surprised, as the fire door they managed to break down was pretty solid.
@Anon: They do have 24/7 security, but it usually consists of one or two guys. It took them a short while to notice the alarm and respond. (Alarms for all Level3 sites are monitored centrally from their NOC in the US)
@muriithi: As Jon says, you can basically operate them as a slightly cheaper version of the 7600, although “cheap” is very relative…
March 13th, 2009 at 09:52
@A Lol Cat: (Sorry, you got caught in my spam queue)
On the whole, the “extra-secure” datacenters in the UK don’t have the power available for our needs. They also charge an extortionate markup. If we paid for a year’s worth of our hosting there I think it might end up cheaper to get all of our kit stolen ;).
I know we share several of our datacenters with major banks. On the whole, the security provisions of the major London facilities are pretty similar.
March 16th, 2009 at 15:12
[...] Just imagine it: your web site is down, you can’t reach the server, you can’t reach the router, the guys at the datacenter aren’t answering the phone, what the heck is going on? You get in your car and drive down to the datacenter and as you drive up you notice all the police cars in the parking lot. Walking into your datacenter, you learn that thieves have broken in over night and made off with you’re hardware as well as that of half a dozen companies in the same datacenter. Sound too ridiculous to be true? Last.FM found out last week that steel doors aren’t enough to keep determined criminals from breaking into your datacenter. [...]
March 24th, 2009 at 22:46
Cisco 65xx = ridiculously expensive switches, or very expensive routers w/fancy extra functions. Yes.
March 27th, 2009 at 23:40
The datacenter where I work (not London but UK) could be susceptible to this type of robbery – and it’s likely that this and other similar recent thefts have persuaded management to consider it a threat.
The plan – a milatary style defensive wall around the entire complex!
April 6th, 2009 at 19:20
I’ve heard stories of break-ins at the facility I work at, it’s not a data center in the tradditional sense but we have a lot of expenive kit anway. At once point when we moved locations the police required that we only moved during daylight hours and that the truck had a police escort. These days we have large metal doors and all the windows have been covered in amour plating. It’s is pretty worrying how organised (and well informed) computer thieves are.