On Facebook’s Outage…

I am not going to be the asshole tech pundit who trashes others for not knowing things like Border Gateway Protocol, or what have you.

I mean seriously, who at the Facebook’s technical staff even know what BGP stands for? Or any server administration period.

I am talking about the Facebook outage that occurred on Monday morning – the morning after the whistle blower appeared on 60 Minutes. There was little clarity whether it was a Distributed Denial of Service attacks (or DDOS – yes I capitalize the “O” because I find mixed capitals in initials to be tacky) or just a simple routing issue. It turns out, according to their blog post in response to the outage

“Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt.” From Facebook’s engineering blog

Where is the PR to help gel out vague languages such as a “configuration change”… that even though I am not technical, I would certainly test things before applying (but that would violate the Move Fast and Break Things ethos.)

I’ve had a theory that coders were very narrowminded groups of people who don’t know much about networking, much like how networking guys took a literal backwards view on telephony.

Or that coders have basic views of networking, that is no different than a gamer or a YouTube influencer.

This is sad if a large trillion dollar enterprise is acting like a bro startup at a scrappy office building. It’s been confirmed that Facebook’s corporate hierarchy is “flat” according to the whistle blower speaking to Congress this week. It’s not a surprise and coders see the world as flat (maybe they are flat-earthers!)

Facebook’s own computing (of which they do, and not use Google or Microsoft or Amazon), is not just their own farm, but they had built their own bare computers, without standard rack hardware, just sitting out in the open, using open source software along with their wacky hardware designs. Facebook’s software however, should be concerning if the underlying code is basic Linux code, and if Linux servers are acting as routers, they typically are not intended to be built to handle billions of users.

This is really, really bad, if Facebook’s routing is as shallow as a home gamer with consumer grade equipment or consumer grade networking settings. This consumer mindset should alarm enterprises of any size because IP networking and routing is more than just from going in and out.

If you want to be the next Facebook, it’s likely logical you should consider Software Defined Networking, just make sure it’s built for scale and built for serious environments. Facebook’s very casual and reckless approach for managing their systems should also be a wake up call for aspiring web disrupters.