(A photo I took on a night out with the team while visiting the Cloudflare Austin office)

Dogfooding is when an organization uses its own products. In this case, we dogfed our newest cloud service, Magic Transit, which both protects and accelerates our customers’ entire network infrastructure—not just their web properties or TCP/UDP applications. With Magic Transit, Cloudflare announces your IP prefixes via BGP, attracts (routes) your traffic to our global network edge, blocks bad packets, and delivers good packets to your data centers via Anycast GRE.

We decided to use Austin’s network because we wanted to test the new service on a live network with real traffic from real people and apps. With the target identified, we began onboarding the Austin office in an always-on routing topology.

In an always-on routing mode, Cloudflare data centers constantly advertise Austin’s prefix, resulting in faster, almost immediate mitigation. As opposed to traditional on-demand scrubbing center solutions with limited networks, Cloudflare operates within 100 milliseconds of 99% of the Internet-connected population in the developed world. For our customers, this means that always-on DDoS mitigation doesn’t sacrifice performance due to suboptimal routing. On the contrary, Magic Transit can actually improve your performance due to our network’s reach.

Now that we’ve completed onboarding Austin to Magic Transit, all we needed was a motivated attacker to launch a DDoS attack. Luckily, we found more than a few willing volunteers on our Site Reliability Engineering (SRE) team to execute the attack. While the teams were still assembling in multiple locations around the world, our SRE volunteer started firing packets at our target from an undisclosed location.

Without Magic Transit, the Austin office would’ve been hit directly with the packet flood. Two things could have happened in this case (not mutually exclusive):

1. Austin’s on-premise equipment (routers, firewalls, servers, etc.) would have been overwhelmed and failed

2. Austin’s service providers would have dropped packets that exceeded its bandwidth allowance

Both cases would result in a very bad day for everyone.

Instead, when our SRE attacker launched the flood the packets were automatically routed via BGP to Cloudflare’s network. The packets reached the closest data center via Anycast and encountered multiple defenses in the form of XDP, eBPF and iptables. Those defenses are populated with pre-configured static firewall rules as well as dynamic rules generated by our DDoS mitigation systems.

Static rules can vary from straightforward IP blocking and rate-limiting to more sophisticated expressions that match against specific packet attributes. Dynamic rules, on the other hand, are generated automatically in real-time. To play fair with our attacker, we didn’t pre-configure any special rules against the attack. We wanted to give our attacker a fair opportunity to take Austin down. Although due to our multi-layered protection approach, the odds were never actually in their favor.

Source: https://imgflip.com

As part of our multi-layered protection approach, Dynamic Rules are generated on-the-fly by analyzing the packets that route through our network. While the packets are being routed, flow data is asynchronously sampled, collected, and analyzed by two main detection systems. The first is called Gatebot and runs across the entire Cloudflare network; the second is our newly deployed DoSD (denial of service daemon) which operates locally within each data center. DoSD is an exciting improvement that we’ve just recently rolled out and we look forward to writing more about its technical details here soon. DoSD samples at a much faster rate (1/100 packets) versus Gatebot which samples at a lower rate (~1/8000 packets), allowing it to detect even more attacks and block them faster.

The asynchronous attack detection lifecycle is represented as the dotted lines in the diagram below. Attacks are detected out of path to assure that we don’t add any latency, and mitigation rules are pushed in line and removed as needed.

Multiple packet attributes and correlations are taken into consideration during analysis and detection. Gatebot and DoSD search for both new network anomalies and already known attacks. Once an attack is detected, rules are automatically generated, propagated, and applied in the optimal location within 10 seconds or less. Just to give you an idea of the scale, we’re talking about hundreds of thousands of dynamic rules that are applied and removed every second across the entire Cloudflare network.

One of the beauties of Gatebot and DoSD is that they don’t require a traffic learning period. Once a customer is onboarded, they’re protected immediately. They don’t need to sample traffic for weeks before kicking in. While we can always apply specific firewall rules if requested by the customer, no manual configuration is required by the customer or our teams. It just works.

Let’s look at what happened in Austin when one of our SREs tried to DDoS Austin and failed. During one of the first attempts, before DoSD had rolled out globally, a degradation in audio and video quality was noticed for Austin employees on video calls for a few seconds before Gatebot kicked in. However, as soon as Gatebot kicked in, the quality was immediately restored. If we hadn’t had Magic Transit in-line, the degradation of service would’ve worsened until the point of full denial of service. Austin would have been offline and our Austin colleagues wouldn’t have had a very productive day.

On a subsequent attack attempt which took place after DoSD was deployed, our SRE launched a SYN flood on Austin. The attack targeted multiple IP addresses in Austin’s prefix and peaked just above 250,000 packets per second. DoSD detected the attack and blocked it in approximately 3 seconds. DoSD’s quick response resulted in no degradation of service for the Austin team.

Green line = Attack traffic to Cloudflare edge, Yellow line = clean traffic from Cloudflare to origin over GRE

Dogfooding Magic Transit served as a valuable experiment for us with lots of lessons learned both from the engineering and procedural aspects. From the engineering aspect, we fine-tuned our mitigations and optimized routings. From the procedural aspects, we drilled members of multiple teams including the Security Operations Center and Solution Engineering teams to help refine our run-books. By doing so, we reduced the onboarding duration to hours instead of days in order to assure a quick and smooth onboarding experience for our customers.

Request a demo and learn how you can protect and accelerate your network with Cloudflare.

When Cloudflare launched, three of the original five cities in our network - Chicago, Ashburn and San Jose - were located in the United States. Since then, we have grown the breadth of the global network considerably to span 66 countries, and even expanded the US footprint to twenty five locations. Even as a highly international business, the United States continues to be home to a number of our customers and the majority of Cloudflare employees.

Today, we expand our network in the United States even further by adding five new locations: Houston (Texas), Indianapolis (Indiana), Montgomery (Alabama), Pittsburgh (Pennsylvania) and Sacramento (California) as our 129th, 130th, 131st, 132nd and 133rd data centers respectively. They represent states that collectively span nearly 100 million people. In North America alone, the Cloudflare network now spans 37 cities, including thirty in the US.

In each of these new locations, we connect with at least one major local Internet service provider and also openly peer using at least one major Internet exchange. We are participants at CyrusOne IX Houston, Midwest IX Indianapolis, Montgomery Internet Exchange, Pittsburgh IX, and the upcoming Sacramento IX.

These deployments improves performance, security and reliability for our customers, even while expanding the edge (and the compute capability it enables). In the not too distant future, we’d like to deploy at cell towers across major metro markets (and beyond!) to support the next generation of 5G-enabled applications.

With the launch of our next data center, Cloudflare will have deployments located in all of the ten most populous North American metropolitan areas.

This map reflects the network as of the publish date of this blog post. For the most up to date directory of locations please refer to our Network Map on the Cloudflare site.

Five key facts to know about McAllen, Texas

• McAllen, Texas is on the southern tip of the Rio Grande Valley

• The city is named after John McAllen, who provided land in 1904 to bring the St. Louis, Brownsville & Mexico Railway railway into the area

• McAllen, Texas is named the City of Palms

• The border between Mexico and the USA is less than nine miles away from the data center

• McAllen, Texas is where Cloudflare has placed its 119th data center

While McAllen is close to the Mexican border, its importance goes well beyond that simple fact. The city is halfway between Dallas, Texas (where Cloudflare has an existing datacenter) and Mexico City, the center and capital of Mexico. This means that any Cloudflare traffic delivered into Mexico is better served from McAllen. Removing 500 miles from the latency equation is a good thing. 500 miles equates to around 12 milliseconds of round-trip latency and when a connection operates (as all connections should), as a secure connection, then there can be many round trip communications before the first page starts showing up. Improving latency is key, even if we have a 0-RTT environment.

Image courtesy of gcmap service

However, it gets better! A significant amount of Mexican Cloudflare traffic is delivered to ISPs and telcos that are just south of the Mexican border, hence McAllen improves their performance even more-so. Cloudflare chose the McAllen Data Center in order to provide those ISPs and telcos a local interconnect point.

As astute readers of the Cloudflare blog know, the Cloudflare network interconnects to a large number of Internet Exchanges globally. Why should that be any different in McAllen, Texas. It’s not. As of last week, there was a brand new Internet Exchange (IX) in McAllen, Texas.

Image, with permission, from Joel Pacheco’s Facebook page

MEX-IX is that new IX and it provides a whole new way to interconnect with Mexican carriers, many of which are present in McAllen already. Cloudflare will enable peering on that IX as quickly as we can.

Cloudflare has plenty of datacenter presence in South America, however Panama is the only datacenter we have operating within Central America. That means that we still have to work on Belize, Costa Rica, El Salvador, Guatemala, Honduras, and Nicaragua.

But there’s one more place we need to deploy into in order to move the Mexican story forward and that’s Mexico City. More about that in a later blog.

Cloudflare will continue to build new datacenters, including the ones south of the border, and ones around the globe. If you enjoy the idea of helping build one of the world's largest networks, come join our team!

Want some ideas on what to start with? Check out the idea suggestion list on our Cloudflare Community page. It's a great idea to review our Apps documentation available here.

Want to request a Cloudflare Apps workshop in your city? Please drop a line to community@cloudflare.com

Share your works in progress and compare notes with other developers on the community forum.

As it turns out, it takes a lot of work to stop massive attacks and to help make the web faster. Over the past six months, our entire team has contributed in every way imaginable to more than double the capacity of our global network. Below is a behind-the-scenes look into how we keep our global network running.

Along the way we’ve encountered many surprises—some fun and some cringe worthy—that have taught us about our team, our data centers and overcoming challenges that occasionally seem beyond our control.

• San Jose, US (SJC): Our counsel, Ken, is great at pumpkin carving, and even better at standing up to protect the privacy rights of our users (including against Kanye West’s army of lawyers). What you may not have known is that he is happiest in the data center (not to mention our cabling was much prettier when he finished!).

  

  Ken, our counsel

• Los Angeles, US (LAX): Our engineers monitor our network around the clock. Occasionally this means juggling multiple tasks. During our most recent upgrade, Joshua (Systems Reliability Engineer & super dad) managed to snatch a moment to put his kids to bed while managing simultaneous upgrades in Los Angeles and Stockholm.

• Chicago, US (ORD): Just before our Chicago upgrade we learned that our carrier had misplaced a shipment of memory. Fortunately, Nitin (Special Projects) averted disaster and got the courier to radio the driver (it took some convincing!), find and grab our DIMMs, and get the install done in time. When most people think of Chicago they think of the ‘95-96 Chicago Bulls (arguably the greatest basketball team to step foot on this earth). When we think of Chicago, our minds turn to the bullwhip effect. We precisely plan every data center launch and upgrade throughout the entire supply chain—from cables to servers to shipping schedules—to control against situations where precautions amplify errors.

• Dallas, US (DFW): At CloudFlare, stamina is key. Trey (Solution Engineer) experienced this first hand. Despite working through the night on our Dallas upgrade, he still managed to catch a 6:30 AM flight the following morning to San Antonio, where he ran a workshop for our friends at Rackspace.

• Ashburn, US (IAD): An important customer meeting the following morning didn’t stop Matthew (CEO) and Trey (Solution Engineer) from working through the night to upgrade our Ashburn facility. Trey even realized he could use his toenail clippers to save time cutting zipties and keep the install moving.

Home is where the datacenter is

• Hong Kong, HK (HKG): After speaking at an Internet security conference in China, Joshua (Special Projects Lead) spent three consecutive nights upgrading our Seoul, Tokyo and Hong Kong data centers. In addition to the data center, he found airport lounges and taxis to be equally habitable.

• London, GB (LHR): CloudFlare’s first international office opened in London in 2013. Since then we’ve enlisted an amazing team of engineers to keep our network humming 24x7x365. On more than a few occasions, James (Systems Reliability Engineer), Marty (Support Engineer) and Simon (Support Engineer) have found a warm room full of servers in our London facility a comfort on a bitter winter night.

• Stockholm, SE (ARN): Simon (Support Engineer) managed to navigate to the local Kjell to pick up a few needed power adapters, and then braved the 1° C cold outside of our Stockholm data center for a bit longer than he would have liked while waiting for an access card.

• Miami, US (MIA): We take security seriously, and so do our data center partners. Justin (Systems Reliability Engineer) was at least a little intimidated to find guards armed with machine guns protecting the entrance of our Miami facility.

• Tokyo, JP (NRT): We love our data centers so much that we even name our conference rooms after them. As our San Francisco office expands (we’ve now knocked down two walls!), we’ve named the latest NRT.

New challenges

• Atlanta, US (ATL): Each of our racks around the world are fitted with high tech PDUs (power distribution units) that allow for remote monitoring and power cycling. This allows us to monitor our infrastructure in real-time, and react at a moment’s notice. When our colocation provider in Atlanta told us that our PDU wouldn’t fit, giving up wasn’t an option. Joshua (Special Projects Lead) proposed a rack extender to do the trick!

• Paris, FR (CDG): Imagine being told that equipment you had just shipped 5,000 miles across the globe was about to be sent right back. Jérôme (Network Engineer), one of our resident French speakers, saved the day and made sure our equipment stayed right where it belonged: working hard in support of one of our busiest datacenters. Merci Jérôme!

• Seattle, US (SEA): We install console servers with out-of-band, cellular Internet access in each of our data centers to remotely manage our infrastructure in the case our primary Internet connectivity is lost. While this makes it easier to address connectivity issues, installing the equipment itself can occasionally be more difficult. With a SIM card stubbornly lodged into our console server in Seattle, Jerome (Partner Engineer) used what he had available—namely, dental floss and a pair of forceps—to get the job done in a way that even MacGyver would approve.

• Seoul, KR (ICN): Korea ranks near the top of most challenging locations to import equipment into (right up there with Warsaw). Fortunately, having facilitated hundreds of shipments in dozens of countries, Nitin (Special Projects) was able to break through a two month logjam in which Incheon airport became a temporary home for some of our equipment. Among his other talents, Nitin can now hum the FedEx and DHL songs in nearly any language of your choice.

• Valparaíso, CL (SCL): Sometimes events are just out of one’s control. To launch our newest data center we had to wait through multiple customs strikes before equipment could arrive. Still, the show went on. Tom (Network Engineer) worked through Christmas to shave 170ms off of latency for our users in Latin America.

What’s in store for 2014, you ask? Over the next 12 months we will significantly expand our data center footprint, adding facilities in regions we currently lack coverage: Latin America, the Middle East, Africa, and parts of Asia. China is our second largest market, Brazil is our third: in 2014 we’ll be significantly expanding our network to better serve these customers.

If this sounds like fun, and if you enjoy a few surprises every once in a while, please consider joining us. We’re actively recruiting for someone passionate and talented to assist with our expansion.

The CloudFlare team will be at HostingCon 2013 in Austin next week. This is our third year at the show and we have a lot of things in store for partners.

Here's a sneak peek:

• Complimentary limousine transfers from Austin-Bergstrom International Airport to the Hilton Austin hotel on Sunday, June 16th. Reserve your spot today!

• New CloudFlare tshirts

• Live music to supercharge your day during breakfast each morning

• Charging stations at our booth (#523) to keep your devices supercharged

• Bigger and better Nerf Railguns. There is limited quantity, so be sure to visit us at booth #523 to get your Railgun

CloudFlare Railguns ready for HostingCon 2013

We are looking forward to connecting with our current partners and meeting new partners at the show. If you are already a CloudFlare Certified Partner, be sure to stop by and introduce yourself. If you are not a partner yet, stop by to learn more about how CloudFlare can reduce your server load, improve the performance of your network, block spammers, botnets and other web threats, and provide DDOS protection. More details about the CloudFlare Certified Partner program here.

Here's where the CloudFlare team will be all week:

Limo transfers from Austin-Bergstrom International Airport to the Hilton Austin Hotel.

Registration is still open, reserve your spot now!

• 7:45am-8:45am: CloudFlare sponsored breakfast located in the Level 4, Ballroom D Foyer - Live music by Alternator Jones

• 5:00pm onwards: Come find the CloudFlare team at the welcome reception!

• 7:45am-8:45am: CloudFlare sponsored breakfast located in the Level 4, Ballroom D Foyer - Live music by Jackie Venson

• 12:00pm-4:00pm: CloudFlare is in Exhibit Hall 4 at booth #523

• 4:00pm-6:30pm: Visit our booth during the exhibit hall happy hour for a beverage and to supercharge your mobile phone!

• 8:00am-10:00am: CloudFlare sponsored breakfast located in the Level 4, Ballroom D Foyer - Live music by Sean Evan

• 9:00am-9:45am: Our co-founder and CEO Matthew Prince will be speaking on the IPv6 panel discussion "Now is the Time for IPv6" in room #18D

• 9:00-9:45am: Maria Karaivanova and John Roberts from CloudFlare will be co-hosting a talk on partnerships, "Strategies for Successful Partnerships" in room #16

• 12:00-4:00pm: CloudFlare is in Exhibit Hall 4 at Booth #523

Connect with us on Twitter during the event to find out where we are and what's coming up next:#hostingcon, @hostingcon, @CloudFlare

See you in Austin!

Some people from Texas (hint: Terry, our network engineer, hails from Texas) were confused when we said we brought up our data center in Paris. They thought we meant Paris, Texas, not Paris, France. We wanted to end that confusion as quickly as possible, so following hot on the heels of Paris (France) going live, we're turning on the Dallas (Texas) data center.

Turns out the Dallas, Texas and Paris, Texas aren't that far apartgeographically. What surprised me is that Paris, France and Dallas, Texas aren't that far apart culturally either. (Now I've just offended the French, or maybe the Texans... again.) The French seem to have a fascination with the 80s TV show "Dallas." Check out the following clip from the national news in France discussing the important question of "What Shot JR?"

CloudFlare's goal is to make the Internet faster and safer wherever you are. So now, whichever Paris you're in, sites on CloudFlare will be evenfaster.

P.S. - Turns out it was Kristin who shot JR. Shocker, I know!