What happened to Facebook, Instagram and WhatsApp? – Krebs on security
Facebook and its sister properties Instagram and WhatsApp suffer from ongoing global blackouts. We don’t yet know why this happened, but the how is clear: Earlier this morning, something inside Facebook prompted the company to revoke key digital records that tell computers and other connected devices on the Internet how to find these destinations online.
Doug Madory is Director of Internet Analytics at Kentik, a San Francisco-based network monitoring company. Madory said that at around 11:39 a.m. ET today (3:39 p.m. UTC), someone at Facebook caused an update to the company’s Border Gateway Protocol (BGP) records. BGP is a mechanism by which the world’s Internet service providers share information about the providers responsible for routing Internet traffic to which specific groups of Internet addresses.
Put simply, this morning Facebook took away the map telling computers around the world how to find its various properties online. As a result, when typing Facebook.com into a web browser, the browser has no idea where to find Facebook.com and therefore returns an error page.
In addition to blocking billions of users, the Facebook outage also prevented its employees from communicating with each other using their internal Facebook tools. This is because Facebook’s emails and tools are all managed in-house and through the same domains that are now blocked.
“Not only are Facebook’s services and applications inaccessible to the public, but its internal tools and communication platforms, including Workplace, are also available,” the New York Times technical reporter said. Ryan mac tweeted. “No one can work. Several people I spoke to told me that it was the equivalent of a “snow day” at the company. “
Breakdowns occur just a few hours after 60 minutes of CBS broadcast a long-awaited interview with Francoise Haugen, the Facebook whistleblower who recently disclosed a number of internal Facebook investigations showing the company knew its products were causing massive damage and was prioritizing profits rather than taking bolder steps to reduce abuse on its platform, including disinformation and hate speech.
We don’t know how or why the outages persist on Facebook and its other properties, but the changes must have come from within the company, as Facebook manages these records internally. Whether the changes were made maliciously or by accident is a puzzle at this point.
Madory said it could be that someone on Facebook screwed up.
“Over the last year or so, we’ve seen a lot of these big blackouts where some sort of global network configuration update has gone wrong,” Madory said. “We obviously can’t rule out that someone might hack them, but they could have done that to themselves as well.”
Update, 4:37 p.m. ET: Sheera Frenkel with the New York Times tweeted that Facebook employees told him they were having trouble accessing Facebook buildings because their employee badges no longer worked. This could be one of the reasons this outage has persisted for so long: Facebook engineers may have difficulty physically accessing the computer servers needed to upload new BGP records to the global internet.
Update, 6:16 p.m. ET: A trusted source who spoke to someone about the recovery effort on Facebook was told the outage was caused by a routine BGP update gone awry. The source explained that the flawed update prevented Facebook employees – the majority of whom work remotely – from rolling back the changes. Meanwhile, those who had physical access to Facebook buildings could not access Facebook’s internal tools as these were all linked to the company’s blocked domains.
Update, 7:46 p.m. ET: Facebook says its domains are slowly coming back online for most users. In a tweet, the company thanked users for their patience, but still provided no explanation for the outage.
In the meantime, several different domain registrars have put the Facebook.com domain up for sale. This has happened through automated systems that look for registered domains that appear to be expired, abandoned, or recently released. There is no reason to believe that Facebook.com will actually be sold as a result, but it’s fun to consider how many billions of dollars it could make in the open market.
This is a developing story and will likely be updated throughout the day.