In addition, we are working with our existing partners including Microsoft Azure, Digital Ocean and several others to onboard customers and provide them the benefits of the Bandwidth Alliance. Contact us at cloudflare-bandwidth-alliance@cloudflare.com if you are interested.

Over the past year we have seen several customers take advantage of the Bandwidth Alliance and wanted to highlight two examples.

Nodecraft, which allows users an easy way to set up their own game servers, is a perfect example of how the Bandwidth Alliance helped to cut down egress costs. Nodecraft supports games like Minecraft, ARK: Survival Evolved, and Counter-Strike. As Nodecraft’s popularity increased, so did their AWS bill. They were not only being charged for storage they were using, but also for ‘egress’ or data transfer fees out of AWS. They made the decision to move their storage to Backblaze. Now they use Backblaze’s B2 Storage and Cloudflare’s extensive network to deliver content to customers without any egress charges. Read more about their journey here and here.

Pippa.io provides simple and smart solutions for podcasting, including hosting, analytics, and ads. The most important of Pippa’s business is rapid and reliable asset delivery. As Pippa grew, they were pushing millions of large audio files to listeners worldwide. This resulted in significantly increased costs, including excessive data egress fees for retrieving data from a cloud storage service such as AWS S3. DigitalOcean waives egress fees to transfer data to Cloudflare, effectively creating a zero-cost data bridge from DigitalOcean to Cloudflare’s global network. Pippa moved to DigitalOcean storage and Cloudflare’s global cloud security and delivery network. With the combination of lower cost storage and zero egress fees, Pippa saw a 50% savings on their cloud bill.

Nodecraft and Pippa are just two examples of small businesses who are seeing significant cost savings from the Bandwidth Alliance. They both chose the best storage and cloud solution for their use case and a global cloud network by Cloudflare without any taxation of transferring data between these two products. With our newly added partners we expect many more customers to benefit.

You may be asking - ‘How much can I save?’ To help you get a sense of the scale of your potential savings, by moving to the Bandwidth Alliance, we have put together a calculator. Fill in your details on egress to figure out how much you could be saving. We hope this is a helpful resource as you evaluate your cloud platform choices and spend.

We are tremendously excited about the value our Bandwidth Alliance partner ecosystem adds to our customers. We’re on a mission to help make the internet a better place; and ensuring everyone can access cloud resources at zero-egress rates supports that mission in many ways. It’s an easy way for our clients to build modern, cloud-centric applications without the design constraint and financial burden of egress fees.

The cloudflare bandwidth alliance partner landscape continues to grow, and incorporate a diverse group of partners, with today’s second wave announcement.  With over a dozen different partners, the range of choices can quickly become overwhelming. And, while these are all high-quality platforms which we are happy to recommend to our clients - their important differences will help determine the best fit for you, the customer.

In this post, I’ll lay out some of Cloudflare’s approach to this solution design question through the lens of a large client we recently worked with. We apply this approach across our full range of products and services, including many use cases far different from the Storage need we’ll dig into in this post. I hope that this can help all of our clients, or anyone else interested, mirror a similar approach.

The first step to solution design, whether on a technical issue or a business process, is a clear understanding of the needs. In this case, we identified a few key needs from our new client:

• Zero-egress storage option: required to manage costs

• Costs: low cost storage given likelihood of high volume growth

• Read requests: ability to support thousands of read requests per second

• Write requests: less concern about rate of write requests

• Volume: fairly high volume of storage; 500 TB +

• Size: 10’s of millions of small objects in storage

• API: compatible with the familiar S3 API

• Security: authentication between the storage and Cloudflare

These needs were specific to this large client but the factors of consideration are likely similar for any customer looking to store data on a host and deliver it through Cloudflare. The relative weight to each of these factors will depend on your particular application.

With the client’s needs in mind, we were able to start filtering out some providers which did not align well to these needs. I generally find it useful to sort the options into three buckets:

1. Checks all the boxes

2. Soft no (fails to check a few boxes, but we may be able to find a middle ground)

3. Hard no (fails crucial boxes)

First, several providers use a custom API instead of S3. This can have many advantages including cost and performance in some cases; but was not aligned with this client’s request given their development plans. We put all of those into ‘Soft No’ right away.

Then, we dug into each provider’s performance and economic model around read vs write requests; storage volume; and read object size. A few had economics or rate limits which were very challenging for the client’s use case, which put them into ‘Hard No’ category. For example; some providers charge a fee based on the number of Reads from storage. This client wanted to perform 10’s of millions of reads per day on average, across their many stored objects, so any pricing based on this would quickly break their economics. For other use cases, when a low number of large objects are stored, this would not be as much of a factor.

At this point, we had identified a partner which was a very good fit for our client. We introduced the teams, and began implementation. This customer is currently ramping up storage and delivery of their content based on this joint solution and we expect to be serving 100TB’s of their stored data over the next year or so.

In any technology implementation; and especially a complex engagement in Cloudflare’s ever-expanding ecosystem, it is important for us to keep the customer’s goals and use case first in mind. By building close partnerships with our clients, we are able to arrive at a clear understanding of these needs and design the best solution.

We’re excited to work with clients of any scale on Storage, Edge Compute, Security, and many technologies; and leverage our ever growing network to help them succeed.

At Cloudflare, our mission is to help build a better Internet. That means making the Internet faster, smarter, safer, but also more cost efficient with the help of our partners. We are always on the lookout for ways to help save customers money. With that goal we announced the Bandwidth Alliance with our founding partners during our Birthday week.

The key concept of the Bandwidth Alliance is to help reduce and in many cases waive data transfer charges, sometimes known as "bandwidth” or “egress” charges, for our mutual customers. We achieve this in partnership with the founding partners through strongly interconnected networks over peered connections. These connections typically occur within the same facility with no middleman. So, neither Cloudflare nor the cloud provider bears incremental costs. Further, we will also use our smart routing system (read details in this technical blog post) to ensure that all our customers’ traffic on participating cloud providers, once their systems are set up, qualify for this offer.

We are proud to announce the following cloud providers and hosting companies have joined the Bandwidth Alliance in committing to zero data transfer fees for mutual customers.  Below is the list of companies from whom you’ll be able to get zero data transfer fees as a Cloudflare customer. If you're hosted with any member of the Bandwidth Alliance, your data transfer fees should decrease as soon as the required technical and accounting systems are activated. Click on each partner to learn more.

We are especially excited to note the global nature of these partners, expands the Alliance and allows customers in many parts of the world to take advantage of the Alliance. The requests to join the Alliance have been significant and we are working to onboard further partners from both a technical and business perspective.  

This is an open alliance of like-minded companies and if you are a cloud provider or networking company and aren't listed, we encourage you to email cloudflare-bandwidth-alliance@cloudflare.com with a request for inclusion.

The Bandwidth Alliance is providing this benefit to all Cloudflare customers at no additional cost. Our partners provide an a wide variety of choices of cloud services and products.

We are working with our founding partners to make it even easier to use our services jointly. We are helping an initial set of customers use cloud storage with their Cloudflare service and zero data transfer fees. These customers were looking for a best of breed solution with specific storage and delivery needs. Please read this technical architecture post by Kirk Schwenkler describing the considerations in enabling such a solution.

This is a starting point and we expect to be developing many reference architectures with our partners to allow our customers to have a truly best of breed option without any concerns on transfer fees.

Here is what the joining members say:

“Open, affordable and free internet is what we strive for and what we give to our clients. Everyone and every company should have access to it. We are thrilled that the values shared by the Bandwidth Alliance can also be our values.”

Marcin Kowalski, Senior Product Manager at Data Space

"DNSnetworks is proud to become a partner in Cloudflare’s initiative for a global bandwidth alliance.  Reducing the cost of bandwidth for partners and not-for-profit businesses fits into our mission of giving back to the community that has provided us with their business and trust."

Kevin Conroy, VP Cloud Operations at DNSnetworks

“At Host1Plus we believe in long-term connections. Therefore, we simplify and personalize the customer experience for our clients to entrust their infrastructure to us. By joining this alliance, we empower our customers to scale their businesses easily across the globe. We look forward to working together with the Bandwidth Alliance and offer the fastest connections to Host1Plus and Cloudflare customers”

Vincentas Grinius, CEO & Founder at Host1Plus

"At Vultr, we strive to provide the most predictable cloud platform to all of our customers. Joining the Bandwidth Alliance, in partnership with Cloudflare, furthers our vision by being able to provide zero egress to our mutual customers."

David Aninowsky, Founder/CEO at Vultr

"In March of 2018, Wasabi pioneered the cloud storage industry’s first ‘unlimited free egress’ pricing plan. We are excited to join forces with Cloudflare and the Bandwidth Alliance, validating that the early business models of Cloud 1.0 vendors no longer applies in a world that expects unlimited everything. This partnership is another way of slingshotting our mutual customers into a future that is already here today."

David Friend, CEO and co-founder at Wasabi

As an organization, we could choose to celebrate Cloudflare’s birthday in lots of different ways (a press release, a company party, or fun gifts for all our employees). But at Cloudflare, we have a unique birthday tradition: we roll up our sleeves and give our customers and the Internet community a new capability (i.e. a gift) every day of our birthday week.

Some of this past week’s launches have been entirely new offerings, like providing key-value storage across Cloudflare’s global cloud network with Cloudflare Workers KV.  Other birthday week launches help improve the overall Internet ecosystem: the Bandwidth Alliance reduces data transfer charges from major cloud hosts and Cloudflare Registrar reduces the hidden fees typical of many domain registration providers. Other new offerings are focused on improving the Internet’s security and performance and are completely free to use. For example, Encrypted SNI helps fix one of the security holes in the Internet, and our support of the QUIC protocol promises to help make mobile browsing faster.  

We believe the only real way to help build a better Internet is to keep innovating, keep building, and keep launching -- every single day.  In fact, our prelude to this year’s Birthday Week was Crypto Week, a full week dedicated to announcing new technologies that use cryptography to make the Internet better.  No promises, but it is entirely imaginable that in coming years Cloudflare won’t just be celebrating a Birthday Week, but we’ll be launching new capabilities every day of a Birthday Month!

Below is a wrap-up of the capabilities launched this past week.

Cloudflare is fixing one of the core Internet bugs by keeping hostnames private using Encrypted Server Name Indication (SNI). All domains on Cloudflare using our authoritative name servers get Encrypted SNI enabled by default. Explore the protection of Encrypted SNI.

Cloudflare is looking forward to the standardization of the new QUIC protocol being developed by the IETF. Applications are being accepted for early access to our test implementation that allows developers to validate their QUIC deployments before supported web browsers become available. Learn more about QUIC.

The Bandwidth Alliance is a group of forward-thinking cloud and networking companies that are committed to discounting or waiving data transfer (also known as bandwidth) fees for shared customers.  Learn how the Bandwidth Alliance reduces costs.

Cloudflare Registrar lets you securely register and manage your domain name with transparent, no-markup pricing that eliminates surprise renewal fees and hidden add-on charges. Be one of the first to buy a domain or transfer your domain to Cloudflare. Register for early access to the Cloudflare Registrar.

Cloudflare Workers KV provides access to a secure low latency key-value store at all 153 Cloudflare data centers. Developers can use Cloudflare Workers and Workers KV to augment existing applications or to build entirely new applications on top of Cloudflare's global cloud network. Workers KV scales seamlessly to support applications serving dozens or millions of users. Explore how Workers KV allows for serverless key-value storage.

Another way we’ll soon be serving the Internet community is by hosting the fourth annual Cloudflare Internet Summit next week at our San Francisco office on Thursday, October 4th. We don’t spend any time talking about Cloudflare at the Internet Summit. Instead we facilitate discussions with the people who inspire and challenge us. The Internet Summit focuses on the future of the Internet and will feature a series of fireside chats, intimate panel discussions, and lively conversations from some of the brightest thought leaders, executives, entrepreneurs, researchers, and operators. Tickets are almost sold out, so register for the Cloudflare Internet Summit now or plan to tune in to our live stream!

Three things were required to make the Bandwidth Alliance a reality:

1. An ecosystem of like-minded companies who want to provide reduced data transfer fees to their customers.

2. A large global and well-connected network (Cloudflare has 150+ points of presence around the world and multiple peered and paid links at each location). Our network is connected to thousands of partners through transit providers, Internet exchanges, peering interconnects, and private network interconnects. Having a large network footprint allows us to meet our partners where their infrastructure is and exchange traffic with them over low-cost or free connections, instead of expensive paid transit.

3. Argo, our sophisticated traffic routing engine. Argo allows us to make decisions on how to carry traffic across our network in ways that optimize for a number of factors: latency, throughput, jitter, or in the case of the Bandwidth Alliance, cost to our partners to exchange traffic. This routing engine is the technical underpinning of the Bandwidth Alliance.

Typically, as traffic moves across the Internet, packets are exchanged between multiple networks as they move from origin to destination. The specific path taken is determined by routers along the way using Border Gateway Protocol (BGP). BGP allows routers to learn which networks have connectivity to which other networks and to pick optimal paths across those. Notably, “optimal path” in this case refers to the path with the fewest number of network hops to the destination, not the best performance or lowest monetary cost for any party involved.

Cloudflare is directly connected and peered with thousands of different networks, including all of our Bandwidth Alliance partners. Moving traffic over one of these links is very fast and very cheap, especially compared to moving the same traffic over public transit. However, because BGP and standard path discovery techniques are not built to directly understand either real world performance or monetary cost, moving traffic from Cloudflare to our Bandwidth Alliance partners and back may not actually use these fast, cheap links between us and them.

This is where Argo comes in. We can plug different routing functions into Argo, optimizing for connection parameters other than the default heuristics BGP users.

Argo consists of two logical components:

1. A control plane, that mines our network performance data and understands our network topology in order to generate good routes across the globe.

2. A data plane, that takes that routing information and makes sure data flowing across our network actually takes those paths.

Argo’s control plane, in its standard configuration, optimizes connections for minimum latency. It does this by examining subnet-to-subnet timing data collected from requests that pass between those subnets, determining which paths are fastest. The list of subnets examined is determined by looking at global routing tables; each entry in the table is a logically and physically grouped set of hosts. Argo then determines where on our network traffic destined for a given subnet should exit our network.

Full Argo is a paid service that Cloudflare customers can subscribe to. For the customers hosted on providers that are part of the Bandwidth Alliance, Cloudflare automatically enables a limited, routing-only version of Argo at no additional charge. While this does not get all the benefits of Argo's full performance feature set, it does ensure that traffic is routed to the Cloudflare location nearest where our customer is hosted so traffic can pass across a Private Network Interface (PNI) and therefore enjoy substantially lowered bandwidth costs. Customers who choose to subscribe to full-featured Argo get additional performance benefits from optimized route selection, data compression, tiered caching, and protocol optimization.

The end result of all this: low latency, highly available transit across our network from your host to your user, and all at little to no cost to Cloudflare or your cloud provider — and, in turn, a substantially reduced cost to you. We expect that as the Bandwidth Alliance comes online, Cloudflare customers could save more than $50 million per year in cloud bandwidth fees.

Our commitment through the Bandwidth Alliance is to pass those cost savings on to you, our mutual customers. Today, we’re excited to put the benefits in your hands: smaller bills and better performance.

Subscribe to the blog for daily updates on all our Birthday Week announcements.

At Cloudflare, our mission is to help build a better Internet. That means making the Internet faster, safer and smarter, but also more efficient alongside our cloud partners. As such, wherever we can, we're on the lookout for ways to help save our common customers money. That got us looking into why and how much cloud customers pay for bandwidth.

If you're hosting on most cloud providers, data transfer charges, sometimes known as "bandwidth” or “egress” charges, can be an integral part of your bill. These fees cover the cost of delivering traffic from the cloud all the way to the consumer. However, if you’re using a CDN such as Cloudflare, the cost of data transfer comes in addition to the cost of content delivery.

In some cases, charging makes sense. If you're hosted in a facility in Ashburn, Virginia and someone visits your service from Sydney, Australia there are real costs to moving traffic between the two places. The cloud provider likely hands off traffic to a transit provider or uses its own global backbone to carry the traffic across the United States and then across the Pacific, potentially handing off to other transit providers along the way, until eventually handing it off to the visitor's ISP. Someone has to maintain the expensive infrastructure that hauls the traffic the 9,739 miles from Ashburn to Sydney. It makes sense for the cloud provider to charge a customer to cover the cost of that transit or their own backbone.  For example, Google Cloud and Microsoft Azure both send traffic over their highly available, secure, performant backbones of terrestrial fiber, subsea cable and more.  

For nearly all major cloud providers, traffic that is delivered to users via Cloudflare, passes across a private network interface (PNI) or private interconnection. That PNI typically occurs within the same facility through a fiber optic cable between routers for the two networks. Unlike when there's a transit provider in between, there's no middleman so neither Cloudflare nor the cloud provider bears incremental costs for transferring the data over this PNI. Cloud providers use these PNI’s to extensively interconnect with third party networks including Cloudflare’s.

Cloudflare automatically carries traffic from the user’s location to the Cloudflare data center nearest your cloud provider and then over such PNIs. Cloudflare is one of the most peered networks in the world, allowing traffic to be carried over such free interconnected links. We at Cloudflare acknowledge that specific customers with highly distributed infrastructures and bandwidth requirements can find it daunting to orchestrate workloads across multiple providers, and so not all traffic passes over such PNIs.

This got us to wonder: could we potentially create a new model and provide our mutual customers with lower costs? We started talking to the most customer-friendly cloud providers that we exchange traffic with and proposed that we look at making our highly efficient and growing interconnects further benefit our mutual customers.

So today — on the eve of Cloudflare's 8th birthday — we're excited to announce the Bandwidth Alliance: a group of forward-thinking cloud and networking companies that are committed to providing the best and most cost-efficient experience for our mutual customers.

Separately from our newly announced Bandwidth Alliance, we have been working with Google Cloud over the past three years as part of their CDN Interconnect program. That program, which Cloudflare has been a part of since its launch, discounts data transfer fees for mutual Cloudflare/Google Cloud customers by up to 75%. We have worked with Google Cloud to ensure that all our mutual customers now automatically get the discount on their bills without having to do anything. Thus, Google Cloud provides a reduced data transfer fee that will help customers save money in comparison to the standard data transfer fee and this is enabled by the high degree of settlement free peering with Cloudflare.

Microsoft Azure is working on its own CDN interconnect program. Cloudflare is excited to be part of this program and pass on the benefits to our mutual customers.

Thus, Google, and soon Microsoft Azure, provide a highly discounted data transfer fee that will help our mutual customers do more for less.

With the Bandwidth Alliance we wanted to go even further. First, beyond Google Cloud and Microsoft Azure, we worked with several cloud and hosting providers including IBM Cloud to create a group of companies with whom you could get reduced data transfer fees. Second, we implemented a new smart routing system (to learn more read this technical blog post) to ensure that all our customers’ traffic on participating cloud providers could qualify for this offer. And, finally, we agreed with many other cloud providers to not just discount, but to waive data transfer fees entirely for mutual customers.

We are proud to announce the following cloud providers and hosting companies who have joined together with us in committing to reduced or zero bandwidth rates for mutual customers.  

Below is the list of companies from whom you’ll be able to get discounted or eliminated bandwidth fees as a Cloudflare customer. Click on each partner to learn more.

This is an open alliance of like-minded companies and we welcome other members to join whether they be cloud providers, hosting companies, or CDNs. If you are a cloud provider or networking company and aren't listed, we encourage you to email cloudflare-bandwidth-alliance@cloudflare.com with a request for inclusion.

The Bandwidth Alliance is providing this benefit to all Cloudflare customers at no additional cost. If you're hosted with a member of the Bandwidth Alliance, your data transfer fees should decrease when the required technical and accounting systems are activated by the member. Some of those systems are live today. Some are planning to go live over the months ahead.

To give you some sense, we estimate that current Cloudflare customers could save nearly $50 million in data transfer fees per year from hosting with a Bandwidth Alliance member as these programs come online.

Happy eighth birthday to all the Internet from the Bandwidth Alliance and us at Cloudflare. Enjoy your lower data transfer fees!

Here is what the founding members and customers say:

“At JPMorgan Chase, portability of workloads is essential as we drive a hybrid, multi-cloud strategy. Minimizing the friction of secure app and data mobility between clouds enables companies like ours to be more efficient, dynamic, and truly harness the full potential of cloud technologies,” said Larry Feinsmith, Managing Director and Head of Technology Strategy, Innovation and Partnerships.

“At Automattic we believe in making the web a better place, and part of that is enabling our customers to build and scale their web properties with freedom. We look forward to partnering with the Bandwidth Alliance to further this mission in allowing a freer flow of data on the internet,” said James Grierson, Global Partnerships, Automattic.

“We are excited to partner with Cloudflare to make our high-quality cloud storage more affordable and available than ever. For more than 10 years Backblaze has delivered the most cost effective cloud storage on the planet and being a founding member of the Bandwidth Alliance reinforces our continuing commitment of being transparent and trustworthy to our customers,” said Gleb Budman, CEO and co-founder, Backblaze.

“One of the reasons businesses use DigitalOcean is the significant amounts of bandwidth that we include with our compute and storage products at no additional cost,” said Shiven Ramji, VP, Product, DigitalOcean. “Our partnership with Cloudflare will improve on this by making it even more predictable for businesses to manage their cloud costs, and gain the benefits of the DigitalOcean Cloud and Cloudflare’s CDN.”

“IBM Cloud enables our clients to achieve greater business value by accelerating their journey to cloud to help them build, modernize and migrate their applications,” said Faiyaz Shahpurwala, general manager, IBM Cloud. "To further support our clients, we will expand our collaboration with Cloudflare through the Bandwidth Alliance to waive data transfer fees between IBM and Cloudflare servers for customers using the IBM Cloud Internet Services enterprise plan. This will help our clients enhance security and performance from the cloud to the edge without data transfer fees."

“At Linode, we’re focused on service, value, and simplifying the way developers consume our services. Our partnership with the Bandwidth Alliance will further our goal of providing affordable cloud computing services through predictable, pay as you go pricing," said Thomas Asaro, Chief Operations Officer, Linode.

"At Packet, we're driven to make infrastructure a competitive advantage for our customers," said Ihab Tarazi, CTO at Packet. "Partnering with Cloudflare to wipe out bandwidth transfer fees between our networks was a no-brainer, as both Cloudflare and Packet share a common goal: to build a better, stronger and faster Internet."

"Scaleway, since 2005, provides ultra easy, high performance infrastructure and services to allow our customers to build and scale appliances. We were the first hosting company to offer unlimited and unmetered connectivity, it’s a part of our DNA. We are excited to join the Bandwidth Alliance to further our mission to meet our customers demand to scale on the cloud." Said Arnaud de Bermingham, CEO, Scaleway.

“Vapor IO is delivering the next generation internet by enabling high performance applications at the edge of the wireless network. Our unique edge colocation business allows CDNs, cloud providers and web scale companies to place IT equipment in close proximity to cell towers where they can leverage software-defined interconnections to cross connect or peer with virtually any carrier, private data center, or cloud provider. We are excited to be part of the Bandwidth Alliance to accelerate edge deployments by not charging our mutual customers egress fees.” said  Matthew Trifiro, CMO, Vapor IO.

Subscribe to the blog for daily updates on all our Birthday Week announcements.

Cuba is currently poorly served by the Internet with a small number of public WiFi hotspots. There are currently 175 public WiFi hotspots in the country, many in public parks. In addition, many large hotels also have public WiFi. Since this is the primary way Cubans get Internet access it’s not uncommon to see situations like this:

Getting on the WiFi means buying a card that gives you access for 2 CUC ($2) per hour. These cards have a login number and a password (hidden behind a scratch off panel). The hour can be used in chunks by logging off and on.

There’s also mobile phone access to the Internet (I saw 3G, EDGE and GPRS as I traveled across Cuba), but at 1 CUC ($1) per MB it’s very expensive. The phone company does provide email access (to their own email service) and so some Cubans I met used their phones to get email, but I didn’t come across anyone who used the web on their phone.

I was able to use 3G Internet access from my phone (via my home carrier) but curiously got a certificate error trying to access my CloudFlare email and decided not to proceed and discover why!

The overall story is that currently there is limited access to the Internet but that it is expensive (especially for Cubans).

To work around this problem everyone I spoke to had access to Cuba’s private “CDN”: El Paquete Semanal. El Paquete is a weekly service where someone (typically found through word of mouth) comes to your home with a disk (usually a 1TB external USB drive) containing a weekly download of the most recent films, soap operas, documentaries, sport, music, mobile apps, magazines, and even web sites. For 2 CUC a week Cubans have access to a huge repository of media while turning a blind eye to copyright.

Cubans told me of children waiting anxiously for “El Paquete Day” when they’d get the next set of cartoons, music and shows.

Vox wrote a nice article that describes El Paquete and includes this short film from Cuba and its creation.

. . . . . . . .

Being a nerd I talked for a long time with Cubans about El Paquete and was intrigued when one of them mentioned that “it even includes this week’s antivirus update”. Makes sense when you realize people can’t get the latest update downloaded across the Internet. And more than that it piqued my interest in examining a copy of El Paquete and seeing what was really included.

I managed to get access to El Paquete Semanal for the week of March 21, 2016[1]. This edition of El Paquete is 815.25 GB and contained 14,208 files. It’s an immense effort to update that on a weekly basis.

Here’s the top level directory of El Paquete showing the major categories of media available.

Sure enough, right there is Actualizacion Antivirus which contains antivirus updates for Avast, AVG, Avira, BitDefender, ClamAV, Kaspersky, McAfee, Microsoft Security Essentials, ESET and Norton.

The McAfee update was 8107xdat.exe which was released by McAfee on March 17, 2016. So, just four days later that update was in the hands of Cubans. The ClamAV was from the same date.

I scanned the entire disk with the latest Sophos release and it showed up three infected files: all in the Aplicaciones Para Pc directory (i.e within PC apps). Luckily, all these would have been picked up by the antivirus programs include in El Paquete.

Most stories written about El Paquete talk about the availability of films and TV. I was intrigued by the Revistas directory. It contains magazines and newspapers and is split into two sections (one for international magazines and one for those in Spanish).

The international magazines directory contained 53 high-quality PDFs of recent magazines from the English-speaking world. As can be seen from the sample below the PDFs are up to date for the week in question and include edition 511 of The Economist (cover date March 5). There’s also Cosmopolitan (April), Bloomberg Businessweek (March 14), Maxim (March), New Scientist (March 5), The Week (March 11) and a number of other specialist magazines. Somewhat incongruously there’s the NSFW British Sunday tabloid Sunday Sport.

The Spanish language magazine collection is so large that it is divided into folders and contains 296 magazines. It covers everything from cars to cinema and cooking, health and sports, news and economics, photography, humor, fashion, technology and travel.

I saw many mobile phones based on Android around Cuba (many of them had made their way to Cuba from the US and had US branding; Verizon seemed especially popular). Because most Cubans can't get directly to the Google Play Store to download apps, El Paquete contains a weekly update of Android Apps as apk files.

The Aplicaciones Para Moviles directory contains apps for Android and iOS. There are 59 apk files (for Android) and 19 ipa (for iOS).

The iOS files contain updates for Instagram and WhatsApp as well as Plants vs. Zombies, Subway Surf and Temple Run 2.

For Android there's a wider selection (a lot of games) plus Facebook Messenger, WhatsApp, Yahoo Messenger, and a number of apps for handling office documents.

There are a huge quantity of media files in the copy of El Paquete that I looked at: 4,909 JPEGs, 2,334 MP3 files, 1,804 MP4 files, 219 AVI files, and 74 MKV files covering music, TV (series, documentaries, soaps, sports), films (English-language and Spanish) and an entire section of essentially "Best Of" videos selected from sites like YouTube.

The "Best Of" (called Interesantes Variados) contains things like unboxing videos of new products, popular vloggers, and product reviews.

To get a sense of how up to date the information is I took a look at the English-language films available. One of the films, London Has Fallen, was released on March 4, 2016 in the US and was available in El Paquete on March 21, 2016 with hard-coded Spanish subtitles (it looked like it was filmed in a cinema with a hand-held camera).

Some Cubans mentioned to me that El Paquete was important in daily life for things like finding services, or a place to live, or a job. This is done through the Cuban equivalent of Craigslist or Gumtree called Revolico.

Revolico is a web site with categories for cars, jobs, services, computer equipment and buying and selling pretty much anything. It's available online (just click above) but it's also available in El Paquete.

In a folder called ! Revolico + Odisea + Lay + Highvista there's an ISO file containing an archive of the entire Revolico web site so that it can be browsed without an Internet connection. It's a web site without the web; just open index.html to start browsing.

The ISO file is dated March 18, 2016, making it only three days out of date.

Finally, there's a folder called ! Lo Ultimo [Red de Redes] which contains screenshots of web pages from the Spanish web site melty. This is divided into sections for Celebrities, Cinema, Fashion, Music, Series, Technology and Video Games and each web page is a long JPG.

Of course, none of the links are clickable but the news is readable.

Over time, it's probably inevitable that Cubans end up with direct access to the Internet for a reasonable price and El Paquete will become a historical artifact. Although at 851GB per week you'd need to be downloading at over 11Mbps 24 hours a day, 7 days a week to get the equivalent of El Paquete.

Until then, El Paquete contains a fascinating slice of the Internet without having direct access.

1. I am very grateful to Larry Press, who has written a great deal about Internet access in Cuba and about El Paquete in particular, for helping me review the contents of El Paquete. ↩︎

CC BY-SA 2.0 image by Quinn Dombrowski

Two years ago, we previewed the relative cost of bandwidth that we see in different parts of the world. Bandwidth is the largest recurring cost of providing our service. Compared with Europe and North America, there were considerably higher Internet costs in Australia, Asia and Latin America. Even while bandwidth costs tend to trend down over time, driven by competition and decreases in the costs of underlying hardware, we thought it might be interesting to provide an update.

Since August 2014, we have tripled the number of our data centers from 28 to 86, with more to come. CloudFlare hardware is also deployed in new regions such as the Middle East and Africa. Our network spans multiple countries in each continent, and, sometimes, multiple cities in each country.

Traffic across 86 data centers in the CloudFlare network

There are approximately thirteen networks called “Tier 1 networks” (e.g., Telia, GTT, Tata, Cogent) who sell “transit” to access any of thousands of other networks on the Internet using their global backbones, including networks who are not their customers. We connect to networks by either purchasing transit from a global "Tier 1 network" (or major regional network), or by exchanging traffic directly with a carrier or ISP using “peering”. Typically, peered traffic is exchanged without settlement between the peered parties.

We try to make it as easy as possible for networks to interconnect with us. CloudFlare has an “open peering” policy, and participates at nearly 150 internet exchanges, more than any other company.

As a benchmark, let's assume the cost of transit in Europe and North America is 10 units (per Mbps). With that benchmark in place, without disclosing exact pricing, we can compare regions by transit cost, percentage of peering, and their effective blended cost (transit + peering).

Europe Transit vs Peering (Last 30 Days)

Based on our benchmark, the transit cost is 10 units. The region has a large number of Internet exchanges, typically non-profit, where we peer around 60% of our traffic. This makes for an effective regional cost of 4 units.

With perhaps the notable exception of the incumbent in Germany, many networks are supportive of open interconnection. CloudFlare already participates at 40 European internet exchanges, and is in the process of joining at least five more.

North America Transit vs Peering (Last 30 Days)

The cost of transit in North America is equal to the cost in Europe, or 10 units. We peer around 40% of our traffic, resulting in an effective regional cost of 6 units.

The level of peering in North America is less than in Europe, but a significant improvement over two years ago. The share of peered traffic is expected to grow. Some material changes have occurred and are occurring in the North American market, such as Frontier acquiring Verizon FiOS customers in three U.S. States and Charter preparing to merge with Time Warner Cable. We can see these changes making an impact to the regional interconnection landscape.

Notably, our peering has particularly grown in smaller regional locations, closer to the end visitor, leading to an improvement in performance. This could be through private peering, or via an interconnection point such as the Midwest Internet Cooperative Exchange (MICE) in Minneapolis.

Africa Transit vs Peering (Last 30 Days)

Transit prices in Africa are amongst the highest in the world at 14 times the benchmark or 140 units, with notable variance across the continent, from Cairo to Mombasa to Johannesburg. Fortunately, of the traffic that we are currently able to serve locally in Africa, we manage to peer about 90% (with a mix of carriers and ISPs), making for an effective cost of 14 units.

Our African deployments help us avoid the significant latency of serving websites from London, Paris or Marseille. A particularly promising but challenging region where we hope to deploy a CloudFlare data center is West Africa - specifically Nigeria, which is already at just under 100 million Internet users.

Middle East Transit vs Peering (Last 30 Days)

CloudFlare currently has four data centers in the Middle East, each of which are cache deployments with strategic ISP partners to serve their respective customers. We are able to peer all the traffic currently served from these data centers. While these collectively provide significant coverage, there is additional traffic (reaching Europe) that we would like to localize in the region. We hope that the remaining ISPs, such as Saudi Telecom Company, deploy similar caches, and enhance the performance of their customers.

Because we can peer 100% of our traffic in the Middle East, our effective pricing for bandwidth in the region is 0 units. There are, of course, other costs to delivering our service beyond bandwidth. However, by driving up peering rates in the Middle East we’ve been able to make our service in the Middle East extremely cost competitive.

Asia Transit vs Peering (Last 30 Days)

In Asia (excluding the Middle East), transit costs 7 times times the benchmark, or 70 units. However, we peer about 60% of our traffic, resulting in an effective cost of 28 units.

Beyond the major meeting points in Hong Kong, Singapore and Tokyo, a significant portion of our interconnection is localized to take place closer to visitors in cities such as Bangkok, Chennai, Kuala Lumpur, Mumbai, Osaka, New Delhi, Seoul, and Taipei. These statistics do not include our network of strategically located data centers inside of mainland China, where the dynamics of interconnection are entirely unique.

Two Asian locations stand out as being especially expensive: Seoul and Taipei. In these markets, with powerful incumbents (Korea Telecom and HiNet), transit costs 15x as much as in Europe or North America, or 150 units.

South Korea is perhaps the only country in the world where bandwidth costs are going up. This may be driven by new regulations from the Ministry of Science, ICT and Future Planning, which mandate the commercial terms of domestic interconnection, based on predetermined “Tiers” of participating networks. This is contrary to the model in most parts of the world, where networks self-regulate, and often peer without settlement. The government even prescribes the rate at which prices should decrease per year (-7.5%), which is significantly slower than the annual drop in unit bandwidth costs elsewhere in the world. We are only able to peer 2% of our traffic in South Korea.

If you include HiNet and Korea Telecom in our blended bandwidth pricing, and take into account peering, our effective price is 28 units. If you exclude HiNet and Korea Telecom, our effective price is 14 units.

South America Transit vs Peering (Last 30 Days)

Transit prices in South America are very high, costing 17 times the benchmark, or 170 units. We peer about 60% of traffic in South America, making for an effective cost of 68 units.

One of the reasons that transit prices are high is that the Tier 1 networks which are newer entrants to this region have yet to pick up significant market share. While markets such as Brazil are less expensive and have greater peering, costs are highest in countries such as Peru and Argentina where, in each, a single incumbent provider, respectively Telefonica and Telecom Argentina, controls access for the last mile delivery of content to the majority of Internet users.

As we try to increase our share of peered traffic, one of the challenges we face is that many Internet exchanges (e.g., NAP Colombia) only permit domestically incorporated and licensed networks to publicly peer, or in another case, require a unanimous vote of all members on an IX to permit a new participant, effectively creating a separation between “international content” and “domestic content”.

If you include Telecom Argentina and Telefonica, our blended cost of bandwidth in South America is 68 units. If you exclude these two providers then our blended cost is 17 units.

Oceania Transit vs Peering (Last 30 Days)

Transit prices in Oceania (Australia and New Zealand) are lower than they used to be, but continue to be extremely high in relative terms, costing 17 times the benchmark from Europe, or 170 units. We peer 50% of our traffic, resulting in an effective cost of 85 units.

If you exclude Optus and Telstra, then the price falls to 17 units — because we peer with nearly everyone else.

Relative Cost of Bandwidth

CloudFlare has always optimized where we serve customers to take into account our effective costs. If you are a free customer using an excessive amount of expensive transit, we would serve you from fewer regions. The good news is that, over the last five years, we’ve been able to negotiate reasonable transit pricing or settlement-free peering with the vast majority of the world’s networks. That allows us to continue to provide the free version of our service as well as to keep prices low for all our paid services.

Today, however, there are six expensive networks (HiNet, Korea Telecom, Optus, Telecom Argentina, Telefonica, Telstra) that are more than an order of magnitude more expensive than other bandwidth providers around the globe and refuse to discuss local peering relationships. To give you a sense, these six networks represent less than 6% of the traffic but nearly 50% of our bandwidth costs.

While we’ve tried to engage all these providers to reduce their extremely high costs and ensure that even our Free customers can be served across their networks, we’ve hit an impasse. To that end, unfortunately, we’ve made the decision that the only thing that will change these providers’ pricing is to make it clear how out of step they are with the rest of the world. To demonstrate this, we’ve moved our Free customers off these six transit providers. Free customers will still be accessible across our network and served from another regional cache with more reasonable bandwidth pricing.

Ironically, this actually increases the cost to several of these providers because they now need to backhaul traffic to another CloudFlare data center and pay more in the process. For instance, if Telstra were to peer with CloudFlare then they would only have to move traffic over about 30 meters of fiber optic cable between our adjoining cages in the same data center. Now Telstra will need to backhaul traffic to Free customers to Los Angeles or Singapore over expensive undersea cables. Their behavior is irrational in any competitive market and so it is not a surprise that each of these providers is a relative monopolist in their home market.

If you’re a Free CloudFlare customer who cares about optimizing the best possible performance from one of these six providers then we encourage you to reach out to them and encourage them to follow a core principle of a free and open Internet and not abuse their monopoly position. We are committed to serving all our customers across every network that peers with us. To that end, help us convince these six networks to be on the right side of a free and open Internet by reaching out to your ISP.

• Ask HiNet to peer with CloudFlare in Taipei

• Ask Korea Telecom to peer with CloudFlare in Seoul

• Ask Optus to peer with CloudFlare across Australia

• Ask Telecom Argentina to peer with CloudFlare in Buenos Aires

• Ask Telefonica to peer with CloudFlare across South America

• Ask Telstra to peer with CloudFlare across Australia

We’ll post updates as we negotiate with these six networks and are hopeful that we’ll soon be able to serve all our customers across all the networks we interconnect with.

The life of a request to CloudFlare begins and ends at the edge. But the afterlife! Like Catullus to Bithynia, the log generated by an HTTP request or a DNS query has much, much further to go.

This post comes from CloudFlare's Data Team. It reports the state of processing these sort of edge logs, including what's worked well for us and what remains a challenge in the time since our last post from April 2015.

In an edge network, where HTTP and DNS clients connect to thousands of servers distributed across the world, the key is to distribute those servers across many carefully picked points of presence—and with over 85 PoPs, no network has better representation than CloudFlare. The reverse of this distribution, however, has to happen for our network's logs. After anycast has scattered requests (and queries) to thousands of nodes at the edge, it's the Data Team's job to gather the resulting logs to a small number of central points and consolidate them for easy use by our customers.

The charts above depict (with some artifacts due to counter resets) the total structured logs sent from the edge to one of these central points yesterday, July 11th. Yesterday we saw:

• An average of 3.6M HTTP logs per second, with peaks over 4.5M logs/s

• An average of 750K DNS logs per second, with peaks over 1M logs/s

This is a typical, ordinary day, with the edge serving hundreds of Gbps in any given minute and transiting for >128M distinct IP addresses in any given hour.

Such a day results in nearly 360TB of raw, Cap’n Proto event logs. Brokering this data requires two Kafka clusters, comprising:

• 1196 cores,

• 170 10G NICs,

• 10.6TB RAM, and

• 4.3PB disk.

Downstream from Kafka sits roughly the same amount of hardware, some shared between Mesos + Docker and HDFS, some wholly dedicated to services like CitusDB.

We expect to see a significant increase in these numbers by the end of 2016.

What's worked well in this system, and why?

There are more, but space only permits naming five things.

1. The log forwarder. The graphs above use metrics from all instances of our log forwarding service running on the edge. This internal software, written in go, handles structured logs both at the edge (from nginx and from our DNS service) and on the data center side of the pipeline. Its plugin architecture has made it remarkably easy to add new log types and Kafka endpoints, and we've had great visibility into its operation thanks to the metrics we collect from it on ingress, drops, and buffer sizes.

2. Kafka. CloudFlare runs several Kafka clusters, including one with just under 80 brokers. Although some failure modes remain hard to automate, there's nothing else on the market that can do what we need it to do, and it does that every day remarkably well.

3. Persistence, and to a greatly improved extent, HTTP retrieval, for log sharing. CloudFlare offers log storage and retrieval to enterprise customers (known as Log Share or ELS), using a dedicated Kafka consumer, HDFS, and several go services. Thanks to more monitoring, improved resiliency to many kinds of network failures, better runbooks, and a lot of hard work, Log Share today offers significantly better availability than it did at the end of 2015. (The story of this work, by everyone in the team and a number of others, too, is worth its own post to detail.)

4. CitusDB. Few things in a data center may seem less glamorous than sharded PostgreSQL, but CitusDB continues to work great as a performant and easily managed service for persisting aggregated data. Support has been quite good, and the fact that it's "just PostgreSQL" under the hood greatly simplified a zero-downtime CitusDB major version upgrade and migration to a completely new cluster.

5. The platform and SRE! None of these parts could work well without the hard work done by Data's sister team, the Platform Team. We're exceptionally fortunate to have some of the best tools at our disposal, including OpenTSDB + Grafana for metrics, Prometheus for alerting, Sentry for exception reports, ElasticSearch + Kibana for logs, Marathon + Mesos + Docker for orchestration, and nginx + zoidberg for load balancing. In the same way, we also remain most grateful to CloudFlare's fantastic SREs, who continue to work with us in making the data center, different from the edge though it is, a reliable and efficient place to work in.

What do we want to improve or expect to add in the near future?

Time, not space, restricts these. In rough order:

1. Make each service more reliable for the customer. This work includes dozens of small things, from better runbooks and automation to capacity planning and wholesale updates to existing architecture. And it runs a continuum from external services like Log Share, which are squarely about providing data to customers, to internal ones, which are squarely about improving automation and visibility at the edge.

2. New analytics. Customers rely on CloudFlare to know what happens at the edge. We're working today to build analytics systems for all of CloudFlare's offerings and to extend our existing analytics so they tell customers more of what they need. Much of the technology we use in these new systems remains up for grabs, although one piece we have up (and handling a million DNS event logs per second) is Spark Streaming.

3. New data pipelines. Both customers and CloudFlare itself need systems that get request and query logs from one point to another. Such pipelines include external ones that push raw logs to the customer, as well as internal ones that let us push logs between data centers for disaster recovery and the like.

4. Better support for complex analysis. Finally, there will always be customers and internal users who need stronger tooling for analyzing high-dimension data at scale. Making it easy for people write such analyses, whether from a UI or as jobs run against a cluster, is a huge challenge and one we look forward to proving out.

For all these services, which must be built so they can work for everyone, a key challenge is making sure that the design (1) works well for the customer, and (2) can be implemented in an economic number of nodes. As much as we can, we work to build solutions that perform well on dozens to hundreds of nodes, not thousands.

In this post from the Data Team, we talked about what's working and what we hope to work on next. We'd also love to talk with you! If you have thoughts on this post, or on what you'd like the Data Team to write about next, please do tell.

In addition: data at CloudFlare is a small team, so if these problems interest you, you stand to have some great problems to work on. Check out the Systems Software Engineer for Data and Data Analyst roles, and let us know what you think.

Finally, if you're headed to GopherCon, CloudFlare has three Gophers attending this week, including one from the Data Team, Alan Braithwaite @Caust1c. (Many, many services in data, including all production services today, are built with go.) Look us up!

1. "through many peoples and across many seas," the beginning of Catullus 101. ↩︎

At CloudFlare we are dedicated to peering. So much so that we just joined our 100th Internet Exchange point!

Image courtesy of Martin Levy

According to Wikipedia:

“In computer networking, peering is a voluntary interconnection of administratively separate Internet networks for the purpose of exchanging traffic between the users of each network”

In reality this normally means a physical place where two different networks (they could be backbones, CDNs, mobile networks or broadband ISPs) connect their respective networks together to exchange traffic. Over the last fifteen years, there has been a major expansion in network interconnections, running parallel to the enormous expansion of the global Internet. This expansion includes new data centre facilities being developed to house network equipment. Some of those data centres have attracted massive numbers of networks, in no small part due to the thriving Internet Exchanges Points (both new and existing) that operate within them. London with the LINX and LONAP exchanges, Amsterdam with AMS-IX and NL-IX exchanges, Frankfurt with DE-CIX and ECIX exchanges are good examples for Europe. In North America there are plenty of IXPs run by Equinix, TelX, Coresite and a myriad of independent operators. The same can be said for Asia Pacific, South America, Oceania and more recently the African continent. Over the last fifteen years IXPs have been deployed in nearly every corner of the globe.

Peering has become a core requirement of the global Internet. With around 54,000 unique Autonomous Networks (ASNs) making up the global Internet, nearly 6,000 of those networks participate in the peering ecosystem; some big, some small.

A volunteer-based organisation runs PeeringDB, a database of peering networks, peering points, peering locations and more. It’s 100% user-driven data and provides networks with a directory of IXPs and contact information for networks willing to peer. As of January 2016 there’s around 5,700 networks and around 630 IXPs listed in the database. Keeping your PeeringDB information maintained is a must for anybody wanting to peer; here are some tips.

There’s almost nothing stopping any network from peering, especially if there’s another network within its physical reach. Once you have two, three (or more) networks in the same locale, then it makes perfect sense to start an IXP.

CloudFlare operates 75+ PoPs globally and nearly all of them are located close to where a local Internet Exchange exists. By connecting to the local IXP (or multiple local IXPs) we provide our customers an unprecedented level of web and DNS service delivery.

At their core, IXPs are large Layer 2 LANs. Sometimes built with one Ethernet switch, sometimes many switches interconnected together across one or more physical buildings. With either configuration the goal is to connect networks together physically. An IXP is no different in basic concept to your home network, with the only real difference being scale. IXPs can range from 100s of Megabits/second to many Terabits/second of exchanged traffic. Independent of size, their primary goal is to make sure that many networks’ routers are connected together cleanly and efficiently. In comparison, at home you would normally only have one router and many computers or mobile devices.

Across the IXP's local network those different ISPs are able to create one-to-one connections using the BGP protocol. This protocol was created to allow disparate networks to announce to each other their own IP addresses plus those that they have provided connectivity to downstream (i.e. their customers). Once two networks set up a BGP session, their respective routes are exchanged and hence traffic can flow directly between them. Any previous intermediary network or networks are removed from the equation.

BGP allows groups of IP addresses to be announced as a single entity (called a CIDR block). An IP address is a 32 bit number for IPv4 and 128 bit number for IPv6. As the global Internet is made up of billions and billions of IP addresses, the BGP protocol consolidates IP addresses into CIDR network addresses. Within the IPv4 Internet these networks could contain anywhere from 256 contiguous IP addresses (called a /24) up to 16,777,216 contiguous addresses (a /8). A /24 means 2^(32-24) addresses. In IPv6, the smallest network block is either a /64 (18,446,744,073,709,551,616 IP addresses - or 2^(128-64) addresses) or a /48 (a staggering 1,208,925,819,614,629,174,706,176 individual addresses). It’s best to think of a /48 as just 65,536 /64s - the basic building block of IPv6 networking.

Getting back to BGP and the global Internet, the reason it works is that cooperative networks announce their IP address blocks (CIDR blocks) to peering neighbours. Here’s an example of one of those BGP announcements:

8.8.8.0/24         *[BGP/170] 15:29:05, MED 0, localpref 200, from 80.249.208.247
                     AS path: 15169 I, validation-state: unverified
                     to 80.249.208.247 via ae3.0
                   > to 80.249.209.100 via ae3.0

CloudFlare can see this route within its network because our network peers directly with the source of that specific CIDR block. What’s going on here is the other ISP’s router (aka our peering partner) is telling our router that our network can send traffic towards 8.8.8.0/24 (the network block that encompasses Google’s 8.8.8.8 DNS resolver service) to the IXP fabric. Our router sends traffic to the IP addresses of 80.249.208.247 or 80.249.209.100 as the next hop on the exchange fabric. (Fabric is a fancy word for Ethernet switch). The AS path shows that it’s a direct hop to AS15169/Google, meaning we have direct connectivity to Google over this specific Internet Exchange Point without an intermediary network. In reality, CloudFlare and Google interconnect via peering points in every continent (nearly 65+ locations globally).

This single example is repeated again-and-again with 1,000s of other networks across all of CloudFlare’s locations. Each peering session and peering point are dedicated to improve CloudFlare’s network offering towards its customers.

Without IXPs, traffic going from one ISP to another would rely on an intermediary backbone ISP to carry the traffic from source to destination. In some situations there’s no problem with doing this: it’s how a large portion of international Internet traffic flows, as it’s cost prohibitive to maintain direct connections to each-and-every ISP in the world. However, relying on a backbone ISP to carry local traffic can be adverse to performance, sometimes due to the backbone carrier being connected to another backbone whom it hands the traffic to in a completely different city. This situation can lead to what’s known as tromboning, where traffic from one city destined to another ISP in the same city can travel vast distances to be exchanged and then return again. In many developing countries this is a very common.

By connecting to a local Internet Exchange Point and peering with other local ISPs, traffic becomes finely tuned and traffic is exchanged locally and directly. This gives the best performance for our web and DNS services towards those destination networks that peer with us. At some of our PoPs we serve more than 90% of outgoing traffic over Internet Exchange Points, helping to keep latency for network traffic as low a possible.

As well as a performance benefit, exchanging traffic directly over IXPs improves redundancy. Because CloudFlare can setup peering with those other ISPs over more than one exchange, this means both networks have possibly diverse paths in which to exchange traffic.

If for example an ISP only has connectivity to one or two backbones and one of those backbones suffers an outage, traffic destined for ISPs on the Internet Exchange would not be affected as it is exchanged directly.

CloudFlare operates a network that protects its customers from DDoS attacks and alas that DDoS traffic (created somewhere on the global Internet) ends up flowing towards CloudFlare’s network. As we have written about before here and here, we have built a massive network to address DDoS traffic and filter it prior to reaching our customers’ web services. Peering plays a big part in that story. In the same way as we want to optimise the path out of our network towards the end user (providing the best experience when viewing content delivered by our network), we also want to optimise the path of nefarious DDoS traffic inbound to our network.

Why? Removing an intermediary network that’s in the path of a DDoS attack can actually help keep the Internet operating cleanly. We've built a network that will handle DDoS attack traffic, if that traffic reaches us via direct peering sessions then experience shows we have the ability to handle the traffic levels (and packet-per-second levels) associated with it.

If there’s a intermediary network in that path we sometimes observe collateral congestion or packet loss within that network. Not a good thing. A direct path (over a peering connection) means there’s the shortest path between the nefarious source and our filtering and sinking of that DDoS traffic. Without peering, the Internet could not handle the types of DDoS attacks that are much too common these days.

Just about anyone! There’s 100s and 100s of IXPs operating globally. Some countries only have one IXP, some (such as Brazil) have nearly 50 individual IXPs. Alas there are quite a few countries where no IXP exists at all.

Many operating models exist for IXPs. CloudFlare connects to IXPs with a variety of these models.

The membership model (preferred in Europe) centres around a group of networks that are members of a non-profit organisation that operates an IXP in one or more cities. Membership payments along with a per-connection fee keep the IXP operating. With a membership based IXP, the fee structure is publicly published and all members pay equally. This model does exist in some North American cities, though to a much lesser extent. It also exists in most other continents.

Next up is a purely commercial model where the operator of an IXP is a commercial for-profit entity. With the commercial model, pricing can be negotiated and in many cases the commercial entity also operates a data centre facility where costs for colocation (space, power, security, remote-hands, cross-connects et al.) eclipse the cost of the IXP itself, Negotiating downwards towards no cost is quite common. After all in the commercial world, a loss-leader mentality can be common.

Sometimes there are 100% volunteer-based IXPs with very little structure. These exist in many cities as a secondary IXP to a primary (read: larger) IXP. Sometimes a friendly group of ISPs get together (over a few beers?) and just decide it would be fun to operate an IXP. We love these people. You’ve read many times about our love of innovation, entrepreneurship, random hacking and experimenting for the good of the Internet. That also happens in the IXP space. Sometimes these IXPs get big and have to convert to a paid-for model. That’s success in its rawest form!

Not be outdone by commercial, membership or even volunteer models, some governments insist on getting into the IXP game. In most cases this is in countries where an excessive number of telecommunication regulations exist. As you can imagine (if you've read this blog before), we aren’t big fans of telecommunication regulations. That said, some of these IXPs surprisingly work quite well. Egypt’s regulatory body (NTRA – National Telecommunications Regulatory Authority) operates CAIX (the Cairo Internet eXchange). It’s a success story for the Egyptian Internet. India’s various Internet Exchanges have not been as successful. These seven IXP locations are operated by the Indian government’s NIXI (National Internet Exchange of India) company. When reading the statistics, from a distance, it shows bandwidth levels very low when compared with the size of the Indian Internet itself. CloudFlare operates three PoPs in India and by design they are not connected to the NIXI exchanges.

Finally, there’s the Pseudo-IXP. This is an exchange operated by a telecom operator (sometimes the incumbent) whose sole purpose is to provide connectivity back to its own network and charge accordingly for the pleasure. Thailand has nearly a dozen of these. Each telecom company or mobile operation has its own IXP, which misses the point of interconnects! We aren't a fan of this model, and most forward-thinking networks aren't fans of this model either.

When a network wants to expand its footprint and connect to an IXP in another city or country (or even continent), it was normal practice to extend the Layer 3 network by placing a router in that remote location. There was expense associated with those implementations. A router has to be purchased and shipped to the remote site. The remote site has to be acquired (renting space, power, etc.) and some form of remote-hands are needed to rack and stack the network equipment. Finally, there needed to be a contract with the IXP and a connection engineered and paid for.

Then along came the concept of remote peering, which extends the Layer 2 switch fabric (in a very controlled manner) via a third party telecom operator to another city, country or even continent, allowing remote networks to become true members of an IXP. Europe took this concept and went wild with it. The main IXPs in London, Amsterdam & Frankfurt got telecom partners signed up to provide remote peering. Their number of connected networks (and traffic levels) increased instantly. Other parts of the world got the remote peering bug and in this day and age you can virtually (pun intended) connect to IXPs all over the world via remote Layer 2 connections.

Remote peering is an obvious idea and to be honest it helps many ISPs reach peers in neighbouring cities or countries via a direct peering relationship. For CloudFlare our aim is to serve traffic as locally as possible and hence we don’t use remote peering. We try to not peer with remotely connected networks if we have a PoP closer to them. We are relentless in this desire to keep traffic local.

Similar to the concept of remote peering, some IXPs expand their footprint across a country or in some cases across multiple countries, this means an ISP can peer with another ISP in a different country via a local connection to the IXP. This is for practical reasons the same as remote peering, with the difference that the long-range link is between the IXP switches, instead of the connection between the ISP and the IXP. There are a few examples of where CloudFlare peers on distributed exchanges, in order to reach both local participants and ISPs in other surrounding countries that we’ve not yet finished building PoPs in.

The world isn’t perfect and in some markets local peering cannot be achieved (sometimes due to a lack of any local IXPs). In situations where we cannot peer locally on an IXP we aim to connect to ISPs privately. This is the same idea as peering over an IXP except we connect directly to the ISP’s router from our router.

In some cases CloudFlare will help foster the creation of a new IXP or IXPs, such as the many new MegaIX exchange points coming to North America. After all, when you improve local Internet connectivity, you actually improve global connectivity. The Internet is a global entity and every part plays an important role.

While some marketplaces have only one IXP (for example Dublin in Ireland with INEX), some markets have seen redundancy, duplication or fragmentation between IXPs. This is in most cases a good thing. Redundancy or even competition between commercial entities always makes the Internet stronger. The Internet was designed to handle disruptions to its underlying infrastructure. IXPs are just one more part of that infrastructure. We will sometimes connect to three (or four) IXPs in a single city. Four may seem excessive, but for the sake of good redundancy, four is a great number.

When there is more than one IXP in a city there’s a tendency for some ISPs to prefer one IXP over another, or in some cases even running their own. By CloudFlare being present at many exchanges, it gives us a unique and redundant view of the local Internet, it also allows us to have the best coverage of that local market. It brings the added advantage of being able to peer with certain regional networks over multiple exchanges and hence allowing for the best routing redundancy.

After five years of operating a network that's grown to over 75 PoPs globally we have joined our 100th Internet Exchange Point. We have a network that’s growing each and every day. Reaching 100 may be just a number; however, only one or two networks have had the ability to grow to that size. Our customers reap the benefits of that global reach. We won’t stop at 100. Far from it! Want to help us grow our network and increase our global peering? Check out available roles here.

Are you an ISP? Do you want to peer with us? Visit as13335.peeringdb.com or talk to us about an on-net cache.

CC BY 2.0 by Kendrick Erickson

Over the last few months, there’s been increased attention on networks and how they interconnect. CloudFlare runs a large network that interconnects with many others around the world. From our vantage point, we have incredible visibility into global network operations. Given our unique situation, we thought it might be useful to explain how networks operate, and the relative costs of Internet connectivity in different parts of the world.

The Internet is a vast network made up of a collection of smaller networks. The networks that make up the Internet are connected in two main ways. Networks can connect with each other directly, in which case they are said to be “peered”, or they can connect via an intermediary network known as a “transit provider”.

At the core of the Internet are a handful of very large transit providers that all peer with one another. This group of approximately twelve companies are known as Tier 1 network providers. Whether directly or indirectly, every ISP (Internet Service Provider) around the world connects with one of these Tier 1 providers. And, since the Tier 1 providers are all interconnected themselves, from any point on the network you should be able to reach any other point. That's what makes the Internet the Internet: it’s a huge group of networks that are all interconnected.

To be a part of the Internet, CloudFlare buys bandwidth, known as transit, from a number of different providers. The rate we pay for this bandwidth varies from region to region around the world. In some cases we buy from a Tier 1 provider. In other cases, we buy from regional transit providers that either peer with the networks we need to reach directly (bypassing any Tier 1), or interconnect themselves with other transit providers.

CloudFlare buys transit wholesale and on the basis of the capacity we use in any given month. Unlike some cloud services like Amazon Web Services (AWS) or traditional CDNs that bill for individual bits delivered across a network (called "stock"), we pay for a maximum utilization for a period of time (called "flow"). Typically, we pay based on the maximum number of megabits per second we use during a month on any given provider.

Traffic levels across CloudFlare's global network over the last 3 months. Each color represents one of our 28 data centers.

Most transit agreements bill the 95th percentile of utilization in any given month. That means you throw out approximately 36 not-necessarily-contiguous hours worth of peak utilization when calculating usage for the month. Legend has it that in its early days, Google used to take advantage of these contracts by using very little bandwidth for most of the month and then ship its indexes between data centers, a very high bandwidth operation, during one 24-hour period. A clever, if undoubtedly short-lived, strategy to avoid high bandwidth bills.

Another subtlety is that when you buy transit wholesale you typically only pay for traffic coming in (“ingress") or traffic going out (“egress”) of your network, not both. Generally you pay which ever one is greater.

CloudFlare is a caching proxy so egress (out) typically exceeds ingress (in), usually by around 4-5x. Our bandwidth bill is therefore calculated on egress so we don't pay for ingress. This is part of the reason we don't charge extra when a site on our network comes under a DDoS attack. An attack increases our ingress but, unless the attack is very large, our ingress traffic will still not exceed egress, and therefore doesn’t increase our bandwidth bill.

While we pay for transit, peering directly with other providers is typically free — with some notable exceptions recently highlighted by Netflix. In CloudFlare's case, unlike Netflix, at this time, all our peering is currently "settlement free," meaning we don't pay for it. Therefore, the more we peer the less we pay for bandwidth. Peering also typically increases performance by cutting out intermediaries that may add latency. In general, peering is a good thing.

The chart above shows how CloudFlare has increased the number of networks we peer with over the last three months (both over IPv4 and IPv6). Currently, we peer around 45% of our total traffic globally (depending on the time of day), across nearly 3,000 different peering sessions. The chart below shows the split between peering and transit and how it's improved over the last three months as we’ve added more peers.

We don't disclose exactly what we pay for transit, but I can give you a relative sense of regional differences. To start, let's assume as a benchmark in North America you'd pay a blended average across all the transit providers of \$10/Mbps (megabit per second per month). In reality, we pay less than that, but it can serve as a benchmark, and keep the numbers round as we compare regions. If you assume that benchmark, for every 1,000Mbps (1Gbps) you'd pay $10,000/month (again, acknowledge that’s higher than reality, it’s just an illustrative benchmark and keeps the numbers round, bear with me).

While that benchmark establishes the transit price, the effective price for bandwidth in the region is the blended price of transit (\$10/Mbps) and peering (\$0/Mbps). Every byte delivered over peering is a would-be transit byte that doesn't need to be paid for. While North America has some of the lowest transit pricing in the world, it also has below average rates of peering. The chart below shows the split between peering and transit in the region. While it's gotten better over the last three months, North America still lags behind every other region in the world in terms of peering.

While we peer nearly 40% of traffic globally, we only peer around 20-25% in North America. Assuming the price of transit is the benchmark \$10/Mbps in North America without peering, with peering it is effectively \$8/Mbps. Based only on bandwidth costs, that makes it the second least expensive region in the world to provide an Internet service like CloudFlare. So what's the least expensive?

Europe's transit pricing roughly mirrors North America's so, again, assume a benchmark of \$10/Mbps. While transit is priced similarly to North America, in Europe there is a significantly higher rate of peering. CloudFlare peers 50-55% of traffic in the region, making the effective bandwidth price \$5/Mbps. Because of the high rate of peering and the low transit costs, Europe is the least expensive region in the world for bandwidth.

The higher rate of peering is due in part to the organization of the region's “peering exchanges”. A peering exchange is a service where networks can pay a fee to join, and then easily exchange traffic between each other without having to run individual cables between each others' routers. Networks connect to a peering exchange, run a single cable, and then can connect to many other networks. Since using a port on a router has a cost (routers cost money, have a finite number of ports, and a port used for one network cannot be used for another), and since data centers typically charge a monthly fee for running a cable between two different customers (known as a "cross connect"), connecting to one service, using one port and one cable, and then being able to connect to many networks can be very cost effective.

The value of an exchange depends on the number of networks that are a part of it. The Amsterdam Internet Exchange (AMS-IX), Frankfurt Internet Exchange (DE-CIX), and the London Internet Exchange (LINX) are three of the largest exchanges in the world. (Note: these links point to PeeringDB.com which provides information on peering between networks. You'll need to use the username/password guest/guest in order to login.)

In Europe, and most other regions outside North America, these and other exchanges are generally run as non-profit collectives set up to benefit their member networks. In North America, while there are Internet exchanges, they are typically run by for-profit companies. The largest of these for-profit exchanges in North America are run by Equinix, a data center company, which uses exchanges in its facilities to increase the value of locating equipment there. Since they are run with a profit motive, pricing to join North American exchanges is typically higher than exchanges in the rest of the world.

CloudFlare is a member of many of Equinix's exchanges, but, overall, fewer networks connect with Equinix compared with Europe's exchanges (compare, for instance, Equinix Ashburn, which is their most popular exchange with about 400 networks connected, versus 1,200 networks connected to AMS-IX). In North America the combination of relatively cheap transit, and relatively expensive exchanges lowers the value of joining an exchange. With less networks joining exchanges, there are fewer opportunities for networks to easily peer. The corollary is that in Europe transit is also cheap but peering is very easy, making the effective price of bandwidth in the region the lowest in the world.

Asia’s peering rates are similar to Europe. Like in Europe, CloudFlare peers 50-55% of traffic in Asia. However, transit pricing is significantly more expensive. Compared with the benchmark of \$10/Mbps in North America and Europe, Asia's transit pricing is approximately 7x as expensive (\$70/Mbps, based on the benchmark). When peering is taken into account, however, the effective price of bandwidth in the region is \$32/Mbps.

There are three primary reasons transit is so much more expensive in Asia. First, there is less competition, and a greater number of large monopoly providers. Second, the market for Internet services is less mature. And finally, if you look at a map of Asia you’ll see a lot of one thing: water. Running undersea cabling is more expensive than running fiber optic cable across land so transit pricing offsets the cost of the infrastructure to move bytes.

Latin America is CloudFlare's newest region. When we opened our first data center in Valparaíso, Chile, we delivered 100 percent of our traffic over transit, which you can see from the graph above. To peer traffic in Latin America you need to either be in a "carrier neutral" data center — which means multiple network operators come together in a single building where they can directly plug into each other's routers — or you need to be able to reach an Internet exchange. Both are in short supply in much of Latin America.

The country with the most robust peering ecosystem is Brazil, which also happens to be the largest country and largest source of traffic in the region. You can see that as we brought our São Paulo, Brazil data center online about two months ago we increased our peering in the region significantly. We've also worked out special arrangements with ISPs in Latin America to set up facilities directly in their data centers and peer with their networks, which is what we did in Medellín, Colombia.

While today our peering ratio in Latin America is the best of anywhere in the world at approximately 60 percent, the region's transit pricing is 8x (\$80/Mbps) the benchmark of North America and Europe. That means the effective bandwidth pricing in the region is \$32/Mbps, or approximately the same as Asia.

Australia is the most expensive region in which we operate, but for an interesting reason. We peer with virtually every ISP in the region except one: Telstra. Telstra, which controls approximately 50% of the market, and was traditionally the monopoly telecom provider, charges some of the highest transit pricing in the world — 20x the benchmark (\$200/Mbps). Given that we are able to peer approximately half of our traffic, the effective bandwidth benchmark price is \$100/Mbps.

To give you some sense of how out-of-whack Australia is, at CloudFlare we pay about as much every month for bandwidth to serve all of Europe as we do to for Australia. That’s in spite of the fact that approximately 33x the number of people live in Europe (750 million) versus Australia (22 million).

If Australians wonder why Internet and many other services are more expensive in their country than anywhere else in the world they need only look to Telstra. What's interesting is that Telstra maintains their high pricing even if only delivering traffic inside the country. Given that Australia is one large land mass with relatively concentrated population centers, it's difficult to justify the pricing based on anything other than Telstra's market power. In regions like North America where there is increasing consolidation of networks, Australia's experience with Telstra provides a cautionary tale.

The chart above shows the relative cost of bandwidth assuming a benchmark transit cost of \$10/Megabits per second (Mbps) per month (which we know is higher than actual pricing, it’s just a benchmark) in North America and Europe.

While we keep our pricing at CloudFlare straight forward, charging a flat rate regardless of where traffic is delivered around the world, actual bandwidth prices vary dramatically between regions. We’ll continue to work to decrease our transit pricing, and increasing our peering in order to offer the best possible service at the lowest possible price. In the meantime, if you’re an ISP who wants to offer better connectivity to the increasing portion of the Internet behind CloudFlare’s network, we have an open policy and are always happy to peer.