On Wednesday, February 28th, 2018 at 9:15am Pacific Standard Time, GitHub, the popular web-based hosting service for software development, was a victim of a powerful DDoS attack that impacted its global user base of 20M. It was one of the largest DDoS attacks, with attack traffic peaking at 1.3Tbps. Concerned users, some who were in the midst of a code check-in, took to Twitter to express their feelings while simultaneously looking for answers. Unfortunately for GitHub, their status page was also affected, adding a bit of anxiety to the equation. Minutes after the attack, GitHub’s DDoS defense systems kicked in, keeping the disruption to a minimum and making it a successful DDoS mitigation story. Feel free to follow along through this sharelink as we analyze the GitHub’s DDoS attack and mitigation.
Update (March 1st, 2018): Github was hit with a second DDoS attack on Thursday, March 1st, 2018 at 9:10am Pacific Standard Time. Availability dropped by 61% (2 times more severe than yesterday’s attack), but services were immediately restored within 15 minutes. The impact of today’s attack appears to be more powerful and global in scope than yesterday’s attack.
Initial Signs of an Attack
At 9:15am PST, we were alerted to a dip in availability to GitHub services from a handful of our Cloud Agents globally. HTTP server availability to www.github.com fell by 26% (Figure 2), predominantly due to Connect errors in the TCP signaling phase. Connect errors typically indicate a deeper problem at the network layer. We were able to corroborate our theory by the elevated packet loss within the network layer.
The goal of a DDoS attack is to overwhelm resources to temporarily shut down access to a service. While this can be achieved through a variety of techniques, congesting the network by bombarding it with packets is common practice. In this case, we saw nearly 100% packet loss in some network paths (Figure 3), which strongly indicated a denial of service attack.
Prolexic in BGP Path Confirms DDoS Attack
Confirmation that GitHub was being DDoS’d was further validated through BGP data. ThousandEyes collects multiple data sets across different network layers to cross correlate application performance and user experience to network anomalies. Within minutes of the attack, we noticed Prolexic being introduced in the BGP AS Path for reachability to at least 5 of GitHub’s prefixes. Figure 4 below is a spatial representation of BGP reachability to GitHub prefixes from multiple vantage points also known as BGP route monitors. 100% of our BGP monitors observed path changes. Before the attack, GitHub (AS 36459) peered with 4 different upstream ISPs including Telia, Level 3 and NTT. With DDoS mitigation in effect, GitHub withdrew its BGP routes (indicated by red dotted lines) from its primary upstream ISPs and established new BGP peering with Prolexic (AS 32787). Prolexic is a subsidiary of Akamai and a popular DDoS mitigation platform. This change in BGP path forced all traffic destined to GitHub through Prolexic’s scrubbing centers that were capable of absorbing and mitigating the DDoS attack. Within 15 minutes, at 9:30am PST, GitHub services were restored (Figure 1), keeping the impact of the DDoS attack to a minimum.
establishes new peering with Prolexic (AS 32787), a DDoS mitigation platform.
A Successful and Efficient DDoS Mitigation
GitHub was quite efficient in mitigating the DDoS attack. Within minutes, the attack was identified and DDoS defense mechanisms kicked in. Given how quickly DDoS mitigation started, it is highly probable that the entire detection and mitigation process was automated (which is quite impressive, I must say!). While the impact of the attack did not last for more than 15 minutes, GitHub-destined traffic continued to flow through Prolexic scrubbing centers up until 6 hours after the attack. The two spikes in the BGP path change timeline below (Figure 5) represents the various point in time when Prolexic was introduced in the AS-path and subsequently removed.
Two Attacks in Two Days
Yesterday’s attack was exceptional in the history of DDoS attacks. It was the most powerful DDoS attack recorded, with 1.3 Tbps of attack traffic. However, within 24 hours, GitHub was struck with another DDoS attack. And from the looks of it, it seems to be more severe in its impact. GitHub’s availability dropped by 61% today, as compared to 26% yesterday. See Figure 6 and Figure 7 that compares the two DDoS attacks from the perspective of service availability and global scope. Just like yesterday, Prolexic stepped in very quickly to mitigate the impacts of the attack.
And the impact seems much global in scope compared to yesterdays.
Monitoring DDoS Mitigation
DDoS attacks are becoming more frequent and ever more powerful. While the GitHub attack had minimum service interruption and showcased a well executed mitigation process, not all DDoS attacks are created equally. Meanwhile, it’s always helpful to get a view into how your mitigation service is working and how your user experience is holding up under attack. Without that sort of visibility into all your service dependencies, you can’t effectively manage your online business. If you’d like to experience this type of network intelligence for yourself, request a demo or start a free trial of ThousandEyes.