Google traffic redirected to China (via Nigeria) 'in error'

public://webform/writeforus/profile-pictures/richi-2016-480.jpg
Richi Jennings, Industry analyst and editor, RJAssociates

[ Webinar: Get Started with Seamless App Sec in a Single Day (Jan. 23) ]

Google went off the air for a couple of hours earlier this week. Its traffic seemed to disappear into a black hole, thanks to a bad route announcement.

But the route went to China, which got people wondering. A Nigerian Internet service provider said it was the company’s fat-finger fault, but that explanation hasn’t satisfied everyone.

Could there be a more worrying explanation? In this week’s Security Blogwatch, we take a listen to network monitors ThousandEyes and make like Bobby Vee.

Your humble blogwatcher curated these bloggy bits for your entertainment. Not to mention: Tim Griffin 

Application Security Research Update: The State of App Sec in 2018

BGP SNAFU or PRC SOP?

What’s the craic? Frank Bajak is frank—Internet traffic hijack disrupts Google services:

An internet diversion that rerouted data traffic through Russia and China disrupted several Google services. … In addition to Russian and Chinese telecommunications companies, a Nigerian internet provider was also involved.

Border gateway protocol hijacking can knock essential services offline and facilitate espionage and financial theft. … A recent study by U.S. Naval War College and Tel Aviv University scholars says China systematically hijacks and diverts U.S. internet traffic.

And Dan Goodin gives Google goes down after major BGP mishap:

Google lost control of several million of its IP addresses for more than an hour. [It] appeared suspicious to many, in part because it misdirected traffic to China Telecom.

At 21:13 UTC when MainOne Cable Company, a small ISP in Lagos, Nigeria, suddenly updated tables in the Internet’s global routing system to improperly declare that [it] was the proper path to reach 212 IP prefixes belonging to Google. … China Telecom improperly accepted the route and announced it worldwide [as did] Russia-based Transtelecom. … The domino effect played out over a two-hour span.

The border gateway protocol that routes Internet traffic from autonomous system to autonomous system around the globe is as fragile as it is intricate. [it] underscores the continued inability of major providers to address the performance and security limitations of BGP.

They say Ameet Naik’s a runaround lover: [You’re fired—Ed.]

On November 12th, 2018, between 1:00 PM and 2:23 PM PST, ThousandEyes noticed … traffic to Google was getting dropped at China Telecom. Why would traffic from a San Francisco office traversing to Google go … to China? We also noticed a Russian ISP in the traffic path.

All the traffic slammed into the great firewall, terminating at a China Telecom edge router [causing] a massive denial of service to G Suite and Google Search. However, this also put valuable Google traffic in the hands of ISPs in countries with a long history of Internet surveillance.

MainOne has a peering relationship with Google via IXPN in Lagos and has direct routes to Google, which leaked into China Telecom. … On November 13, MainOne tweeted out that the root cause of the problem was in fact due to a configuration error.

One of the fundamental weaknesses in the fabric of the Internet [is that] BGP was designed to be a chain of trust between well-meaning ISPs and universities. … While verification methods like ROA exist, few ISPs use them. Even corporations like Google with massive resources at their disposal are not immune.

Though Jeremy Kirk and Mathew Schwartz say it isn’t so:

Alex Henthorn-Iwane, vice president of product marketing for the security company ThousandEyes [says it’s] unlikely the routing was simply an error, especially because it involved network providers within Russia and China. … "It's not a mistake. … There's nothing about this that suggests that this was a mistake."

"It's the irony that the very openness, agility and resilience that the internet displays also creates a vulnerability." … BGP is largely an honor system, Henthorn-Iwane says. Network operators hope that other network operators won't make unauthorized BGP announcements on their behalf. But there are no real mechanisms in place to stop that from happening.

But if Emma Woollacott puts me down for another, I’ll know, believe me, I’ll know:

It's the latest of several incidents that have involved the misdirection of traffic via China Telecom. But … Google says it has no reason to believe that the problem was anything other than an accident.

Late last month, researchers … said they had found evidence that China Telecom has been publishing bogus routes, misdirecting large quantities of internet traffic via China before delivering it with a slight delay. … Routes from Canada to South Korean government sites. … Several US routes to the headquarters of an Anglo-America bank in Milan. Traffic between Scandinavia and Japan … targeting a major US news organisation.

They point out that China has ten Points of Presence (PoPs) in North America and Europe, making traffic delays less noticeable and interference harder to detect. … Whether or not China Telecom really is deliberately intercepting traffic, it seems pretty clear that it could if it chose.

And Kevin Beaumont—@GossiTheDogcan’t help but see if you are true to me:

We still use SMTP to send email and largely can’t deploy basic things like SPF records and DKIM to stop spoofing - i.e. the internet is made of string. Basic things like BGP filtering rules often aren’t even done right

BGP needs some serious work - NTT blindly accepting a BGP change in the US from a tiny company in Africa which routed Google Cloud to China (via Russia too) is a continued sign.

I realise this isn't news, but somebody needs to take a look at how BGP filtering is implemented within the US and its allies.

If you can do this by mistake, imagine what you can do on purpose and with budget.

My technical description of BGP tables and peering arrangements: It’s like a spider with 800,000 legs you’re trying to stop eating itself.

There was no way that was hijacking in sense of interception as they sent it to a black hole. However … if you send all traffic to nowhere, you still hijack a legit service and crash it.

With choices two, it’s MatthiasF:

If this is not malicious then it is … incredibly bad configuration. … Too many people use full route tables when they should be using filtered tables. Had every spoke in this scenario done so, this propagation would have never happened.

Only the routers at major hubs should be reconciling full route tables and all routers at international transversal should be using BGP monitoring software to stop bad information from passing them.

And if DarkLogix ruled the world:

The following three should be implemented by any major ISP

SIDR https://www.nccoe.nist.gov/projects/building-blocks/secure-inter-domain-routing

BGPsec https://tools.ietf.org/html/rfc8206

rPKI https://tools.ietf.org/html/rfc8210

I'd rule it a US national security issue and that all ISP's (over some minimum) must implement these as well as DNSsec on their reverse DNS zones ASAP.

But vix86 alleges an allegation:

China is planning to dump billions of dollars into Nigeria in various areas. I wouldn't be surprised to learn that telecoms in Nigeria are also working closely with China and in exchange for money and favors … ie: acting as an arm of the Chinese Intelligence agency.

The West should keep an eye on all the countries that China gets involved with via the One Belt One Road initiative, because those could be countries that might opt to route traffic to China as well.

Meanwhile, there’s one thing jmsbar is certain of:

One thing I’m certain of is that there was a Nigerian prince involved somewhere.

The moral of the story? If a Nigerian prince announced a bogus route for your network, how soon would you find out?

And finally …

Border Gateway Protocol

 


You have been reading Security Blogwatch by Richi Jennings. Richi curates the best bloggy bits, finest forums, and weirdest websites … so you don’t have to. Hate mail may be directed to @RiCHi or sbw@richi.uk. Ask your doctor before reading. Your mileage may vary. E&OE.

Image source: Steve Song (cc:by)

Topics: Security