Cloudflare Waiting Room protects sites from overwhelming traffic surges by placing excess visitors in a fully customizable virtual waiting room, admitting them dynamically as spots become available. Instead of throwing error pages or delivering poorly-performing site pages, Waiting Room empowers customers to take control of their end-user experience during unmanageable traffic surges.

A key decision customers make when setting up a waiting room is what pages it will protect. Before now, customers could select one hostname and path combination to determine what pages would be covered by a waiting room. Today, we are thrilled to announce that Waiting Room now supports coverage of multiple hostname and path combinations with a single waiting room, giving customers more flexibility and offering broader site coverage without interruptions to end-user flows. This new capability is available to all Enterprise customers with an Advanced Purchase of Waiting Room.

As part of the simple, no-coding-necessary process for deploying a waiting room, customers specify a hostname and path combination to indicate which pages are covered by a particular waiting room. When a site visitor makes a preliminary request to that hostname and path or any of its subpaths, they will be issued a waiting room cookie and either be admitted to the site or placed in a waiting room if the site is at capacity.

Last year, we added Waiting Room Bypass Rules, giving customers many options for creating exceptions to this hostname and path coverage. This unlocked capabilities such as User Agent Bypassing, geo-targeting, URL exclusions, and administrative IP bypassing. It also allowed for some added flexibility regarding which pages a waiting room would apply to on a customer’s site by adding the ability to exclude URLs, paths, and query strings. While this update allowed for greater specificity regarding which traffic should be gated by Waiting Room, it didn’t offer the broader coverage that many customers still needed to protect greater portions of their sites with a single waiting room.

Let’s review some simple yet impactful examples of why this broader coverage option was important for our customers. Imagine you have an online store, example.com, and you want to ensure that a single waiting room covers the entire customer journey — from the homepage, to product browsing, to checkout. Many sites would delineate these steps in the flow using paths: example.com/, example.com/shop/product1, example.com/checkout. Because Waiting Room assumes an implied wildcard at the end of a waiting room’s configured path, this use case was already satisfied for these sites. Thus, placing a waiting room at example.com/ would cover all the URLs associated with every step of this customer journey. In this setup, once a site visitor has passed the waiting room, they would not be re-queued at any step in their user flow because they are still using the same waiting room’s cookie, which indicates to Waiting Room that they are the same user between URLs.

However, many sites delineate different steps of this type of shopping flow using subdomains instead of or as well as paths. For example, many sites will have their checkout page on a different subdomain, such as checkout.example.com. Before now, if a customer had this site structure and wanted to protect their entire site with a single waiting room, they would have needed to deploy a waiting room at example.com/ and another at checkout.example.com/. This option was not ideal for many customers because a site visitor could be queued at two different parts of the same customer journey. This is because the waiting room at checkout.example.com/ would count the same visitor as a different user than the waiting room covering example.com/.

That said, there are cases where it is wise to separate waiting rooms on a single site. For example, a ticketing website could place a waiting room at its apex domain (example.com) and distinct waiting rooms with pre-queues on pages for specific events (example.com/popular_artist_tour). The waiting room set at example.com/ ensures that the main point of entry to the site does not get overwhelmed and crash when ticket sales for any one event open. The waiting room placed on the specific event page ensures that traffic for a single event can start queuing ahead of the event without impacting traffic going to other parts of the site.

Ultimately, whether a customer wants one or multiple waiting rooms to protect their site, we wanted to give our customers the flexibility to deploy Waiting Room however was best for their use cases and site structure. We’re thrilled to announce that Waiting Room now supports multi-hostname and path coverage with a single waiting room!

Customers can now configure a waiting room on multiple hostname and path combinations — or routes — belonging to the same zone. To do so, navigate to Traffic > Waiting Room and select Create. The name of your domain will already be populated. To add additional routes to your waiting room configuration, select Add Hostname and Path. You can then enter another hostname and path to be covered by the same waiting room. Remember, there is an implied wildcard after each path. Therefore, creating a waiting room for each URL you want your waiting room to cover is unnecessary. Only create additional routes for URLs that wouldn’t be covered by the other hostname and path combinations you have already entered.

When deploying a waiting room that covers multiple hostname and path combinations, you must create a unique cookie name for this waiting room (more on that later!). Then, proceed with the same workflow as usual to deploy your waiting room.

A frequent customer request was the ability to cover a multi-language site with a single waiting room — offering different text per language while counting all site traffic toward the same waiting room limits. There are various ways a site can be structured to delineate between different language options, but the two most common are either by subdomain or path. For a site that uses path delineation, this could look like example.com/en and example.com/es for English and Spanish, respectively. For a site using subdomain delineation, this could look like en.example.com/ and es.example.com/. Before multihost Waiting Room coverage, the subdomain variation would not have been able to be covered by a single waiting room.

Waiting Room’s existing configuration options already supported the path variation. However, this was only true if the customer wanted to gate the entire site, which they could do by placing the waiting room at example.com/. Many e-commerce customers asked for the ability to gate different high-demand product pages selling the same product but with different language options. For instance, consider example.com/en/product_123 and example.com/es/product_123, where the customer wants the same waiting room and traffic limits to cover both URLs. Before now, this would not have been possible without some complex bypass rule logic.

Now, customers can deploy a waiting room that accommodates either the subdomain or path approach to structuring a multi-language site. The only remaining step is setting up your waiting room to serve different languages when a user is queued in a waiting room. This can be achieved by constructing a template that reads the URL to determine the locale and define the appropriate translations for each of the locales within the template.

Here is an example of a template that determines the locale from the URL path, and renders the translated text:

<!DOCTYPE html>
<html>
  <head>
    <title>Waiting Room powered by Cloudflare</title>
  </head>
  <body>
    <section>
      <h1 id="inline-msg">
        You are now in line.
      </h1>
      <h1 id="patience-msg">
        Thank you for your patience.
      </h1>
    </section>
    <h2 id="waitTime"></h2>

    <script>
      var locale = location.pathname.split("/")[1] || "en";
      var translations = {
        "en": {
          "waittime_60_less": "Your estimated wait time is {{waitTime}} minute.",
          "waittime_60_greater": "Your estimated wait time is {{waitTimeHours}} hours and {{waitTimeHourMinutes}} minutes.",
          "inline-msg": "You are now in line.",
          "patience-msg": "Thank you for your patience.",
        },
        "es": {
          "waittime_60_less": "El tiempo de espera estimado es {{waitTime}} minuto.",
          "waittime_60_greater": "El tiempo de espera estimado es {{waitTimeHours}} de horas y {{waitTimeHourMinutes}} minutos.",
          "inline-msg": "Ahora se encuentra en la fila de espera previa.",
          "patience-msg": "Gracias por su paciencia.",
        }
      };

      {{#waitTimeKnown}}
      var minutes = parseInt( {{waitTime}} , 10);
      var time = document.getElementById('waitTime');

      if ( minutes < 61) {
        time.innerText = translations[locale]["waittime_60_less"]
      } else {
        time.innerText = translations[locale]["waittime_60_greater"]
      }
      {{/waitTimeKnown}}

      // translate template text for each of the elements with “id” suffixed with “msg”
      for (const msg of ["inline-msg", "patience-msg"]) {
        const el = document.getElementById(msg);
        if (el == null) continue;
        el.innerText = translations[locale][msg];
      }
    </script>
  </body>
</html>

Here’s how the template works: given a site distinguishes different locales with various paths such as /en/product_123 or /es/product_123 in the <script /> body after the rendering the page, the locale is extracted from the location.pathname via locale = location.pathname.split(“/”)[1]. If there isn’t a locale specified within the translations object we default it to “en”. For example, if a user visits example.com/product_123, then by default render the English text template.

Similarly, in order to support a multi-language waiting room for sites that delineate site structure via subdomain, it would only require you to update how you extract the locale from the URL. Instead of looking at the pathname we look at the hostname property of the window.location object like locale = location.hostname.split(“.”)[0], given, we have site structure as following: en.example.com, es.example.com.

The above code is a pared down example of starter templates we have added to our developer documentation, which we have included to make it easier for you to get started with a multi-language waiting room. You can download these templates and edit them to have the look and feel aligned with your site and with the language options your site offers.

This is an example of the starter template provided in dev docs. This waiting room is in Queue-all mode and displays the Spanish text when the user visits example.com/es/product_123.

These templates can further be expanded to include other languages by adding translations to the `translations` object for each of the locales. Thus, a single template is able to serve multiple languages depending on whatever the locale the site is being served as either via subdomain or path.

The waiting room assigns a __cfwaitingroom cookie to each user to manage the state of the user that determines the position of the user in line among other properties needed to make the queueing decisions for the user. Traditionally, for a single hostname and path configuration it was trivial to just set the cookie as __cfwaitingroom=[cookie-value]; Domain=example.com; Path=/es/product_123. This ensured that when the page refreshed it would send us the same Waiting Room cookie for us to examine and update. However, this became non-trivial when we wanted to share the same cookie across different subdomain or path combinations, for example, on example.com/en/product_123.

There were two approaches that we brainstormed and evaluated to allow the sharing of the cookie for the same waiting room.

The first approach we considered was to issue multiple Set-Cookie headers in the HTTP response. For example, in the multi-language example above, we could issue the following in the response header:

Set-Cookie: __cfwaitingroom=[cookie-value]…Domain=example.com; Path=/en/product_123 …
Set-Cookie: __cfwaitingroom=[cookie-value]...Domain=example.com; Path=/es/product_123 …

This approach would allow us to handle the multiple hostnames and paths for the same waiting room. However, it does not present itself as a very scalable solution. Per RFC6265, there isn’t a strict limit defined in the specification, browsers have their own implementation-specific limits. For example, Chrome allows the response header to grow up to 256K bytes before throwing the transaction with ERR_RESPONSE_HEADERS_TOO_BIG. Additionally, in this case, the response header would grow proportionally to the number of unique hostname and path combinations, and we would be redundantly repeating the same cookie value N (where N is the number of additional routes) number of times. While this approach would have allowed us to effectively handle a bounded list of hostname and path combinations that need to be set, it was not ideal for cases where we can expect more than a handful routes for a particular site.

The second approach that we considered and chose to move forward with was to set the cookie on the apex domain (or the most common subdomain). In other words, we would figure out the most common subdomain from the list of routes and use that as the domain. Similarly, for the paths, this would entail determining the least common prefix from the list of paths and use that as the value to be set on the path attribute. For example, consider a waiting room with the following two routes configured, a.example.com/shoes/product_123 and b.example.com/shoes/product_456.

To determine the domain that would be set for the cookie, we can see that example.com is the most common subdomain among the list of domains. Applying the most common subdomain algorithm, we would get example.com as the domain. Applying the least common prefix algorithm on the set of paths, /shoes/product_123 and /shoes/product_456, we would get /shoes as the path. Thus, the set-cookie header would be the following:

Set-Cookie: … __cfwaitingroom=[cookie-value]; Domain=example.com; Path=/shoes …

Now, when a visitor accesses any of the pages covered by this waiting room, we can guarantee Waiting Room receives the right cookie and there will only be Set-Cookie included in the response header.

However, we are still not there yet. Although we are able to identify the common subdomain (or apex domain) and common path prefix, there would still be a problem if routes from one waiting room would overlap with another waiting room. For example, say we configure two waiting rooms for the same site where we are selling our special shoes:

Waiting Room A
    a.example.com/shoes/product_123
    b.example.com/shoes/product_456

Waiting Room B
    c.example.com/shoes/product_789
    d.example.com/shoes/product_012

If we apply the same algorithm as described above, we would get the same domain and path properties set for the two cookies. For Waiting Room A, the domain would be example.com and the path would be /shoes. Similarly, for Waiting Room B, the most common subdomain would also be example.com and least common path prefix would be /shoes. This would effectively prevent us from distinguishing the two cookies and route the visitor to the right waiting room. In order to solve the issue of overlapping cookie names, we introduced the requirement of setting a custom suffix that is unique to the customer’s zone. This is why it is required to provide a custom cookie suffix when configuring a waiting room that covers multiple hostnames or paths. Therefore, if Waiting Room A was assigned cookie suffix “a” and Waiting Room B was assigned cookie suffix “b”, the two Set-Cookie definitions would look like the following:

Waiting Room A

Set-Cookie: __cfwaitingroom_a=[cookie-value]; Domain=example.com; Path=/shoes

Waiting Room B

Set-Cookie: __cfwaitingroom_b=[cookie-value]; Domain=example.com; Path=/shoes

When a visitor makes a request to any pages where the two different waiting rooms are configured, Waiting Room can distinguish and select which cookie corresponds to the given request and use this to determine which waiting room’s settings apply to that user depending on where they are on the site.

With the addition of multihost and multipath Waiting Room coverage, we’re thrilled to offer more flexible options for incorporating Waiting Room into your site, giving your visitors the best experience possible while protecting your site during times of high traffic. Make sure your site is always protected from traffic surges. Try out Waiting Room today or reach out to us to learn more about the Advanced Waiting Room!

Every day, all across the Internet, something bad but entirely normal happens: thousands of origin servers go down, resulting in connection errors and frustrated users. Cloudflare’s users collectively spend over four and a half years each day waiting for unreachable origin servers to respond with error messages. But visitors don’t want to see error pages, they want to see content!

Today is exciting for all those who want the Internet to be stronger, more resilient, and have important redundancies: Cloudflare is pleased to announce a partnership with the Internet Archive to bring new functionality to our Always Online service.

Always Online serves as insurance for our customers’ websites. Should a customer’s origin go offline, timeout, or otherwise break, Always Online is there to step in and serve archived copies of webpages to visitors. The Internet Archive is a nonprofit organization that runs the Wayback Machine, a service which saves snapshots of billions of websites across the Internet. By partnering with the Internet Archive, Cloudflare is able to seamlessly deliver responses for unreachable websites from the Internet Archive, while the Internet Archive can continue their mission of archiving the web to provide access to all knowledge.

Enabling Always Online in the Cloudflare dashboard allows us to share your hostname with the Wayback Machine so that they can archive your website. When a website’s origin is down, Cloudflare will go to the Internet Archive to retrieve the most recently archived version of the site, so that visitors will still be able to view the site’s content.

When a person visits a Cloudflare website, a request is made from their laptop/phone/tablet/smart fridge to Cloudflare’s edge. Our edge first looks to see if we can respond with cached content; if the requested content is not in cache, or is determined to be expired, we then obtain a fresh copy from the origin. As part of fulfilling an uncached/expired origin fetch, we also update our cache to allow subsequent requests to be served to visitors faster and more securely. If we are unable to reach the origin, our edge tries a few more times to connect before marking the origin as being down and serving an error page to the visitor. Receiving an error page is not ideal for anyone, so we try really hard to ensure that visitors to websites using Cloudflare can get some content, even if an origin is struggling.

When Cloudflare started 10 years ago, most of our customers were small and running on hosts that were subject to frequent downtime. These early customers feared that their host may go down at the same time a search engine was indexing their site. The search engine’s crawler would report the downed site as non-responsive and the site would drop in their search ranking. Always Online was born from that concern.

Through operating Always Online over the past 10 years, we’ve learned that fighting Internet downtime with simple, unobtrusive tools was something that our customers and their users deeply value. Though some features have undergone rewrite upon rewrite, other parts of our code have remained relatively untouched by the sands of time, a testament to their robustness. For example, Always Online clearly shows a banner indicating that it is serving an archived version of the page due to the origin being unreachable, and this transparency is well-received by both website owners and visitors.

We recently set out to make Always Online even better. We wanted to preserve what customers loved — as seamless an experience as possible for their users when their origin servers are down — while increasing the amount of content available through Always Online, ensuring it is as fresh as possible, and performing this archiving in a way that helps make the Internet a better place.

What a visitor will see with Always Online. 

Partnering with the Internet Archive’s Wayback Machine to power the next generation of Always Online accomplishes all of these goals. The Internet Archive’s mission is to provide universal access to all knowledge. Since 1996, the Internet Archive’s Wayback Machine has been archiving much of the public Web: preserving and making available millions of websites and pages that would otherwise be lost. In pursuit of that mission, they have archived more than 468 billion web pages, amounting to more than 45 petabytes of information.

Always Online’s integration with the Internet Archive will help the Archive expand their record of the Internet; many of the domains that opt-in to Always Online functionality may not have been otherwise discovered by the Archive’s crawler. And for Cloudflare customers, the Archive will seamlessly provide visitors access to content that would otherwise be errors.

In other words, Cloudflare partnering with the Internet Archive makes the Internet better, stronger, and more available to everyone.

“Through our partnership with Cloudflare, we are learning about, and archiving, webpages we might not have otherwise known about, and by integrating with Cloudflare’s Always Online service, archives of those pages are available to people trying to access them if they become unavailable via the live web”—Mark Graham, Director of the Internet Archive’s Wayback Machine

“We are excited to work with Cloudflare and expect this partnership to bring important redundancy to the Internet and allow for us to advance our ongoing efforts to make the Internet more useful and reliable.”—Brewster Kahle, Founder and Digital Librarian of the Internet Archive

Upgrading to the new Always Online in the Cloudflare dashboard allows us to share some basic information about your website with the Internet Archive (like hostname and popular URLs), so they can begin to crawl and archive your website at regular intervals. This information sharing and crawling ensures content is available to Always Online and also serves to deepen the library of content available directly through the Archive.

If your origin goes down or is unreachable, Cloudflare’s edge will return a status code in the 520 to 527 range, indicating an issue connecting to the origin. When this happens, Cloudflare will first look to the local edge datacenter to see if there is a stale or expired version of content we can serve to the website visitor. If there isn’t a version in the local cache, Cloudflare will then go to the Internet Archive and fetch the most recently archived version of the site to serve to your visitors. When that happens, Always Online serves the archived content with a banner to let your visitors know that your origin is having problems. The banner allows for your visitors to check and see if your origin is back online with a single click. While dynamic content that requires communication with an origin server will still show an error to visitors (e.g. web applications or shopping carts), basic content will often be available with Always Online.

For now, the old Always Online service will still be available, but we plan to fully transition to the Internet Archive-backed version soon.

Cloudflare customers can enable Always Online in the dashboard:

• For more about Always Online, and how it works, please check out our documentation.

• To get started using Always Online, please log into your Cloudflare dashboard and toggle it on.

• Please see the Internet Archive’s announcement of our partnership here.

• To help improve Always Online, or other parts of our slice of the Internet, drop us a line.

When installing an app, a site owner must select where - what URLs on their site - they want what apps installed. Our original plan for selecting the URLs an app would be installed on took a few twists and turns. Our end decision was to utilize our Always Online crawler to pre-populate a tree of the user’s site. Always Online is a feature that crawls Cloudflare sites and serves pages from our cache if the site goes down.

The benefits to this original setup are:1. Only valid pages appearAn app only allows installations on html pages. For example, since injecting Javascript into a JPEG image isn’t possible, we would prevent the installer from trying it by not showing that path. Preventing the user from that type of phony installation prevents the user from being confused later when it doesn’t work.2. The user was not required to know any URL of their siteThe URLs are available right there in the UI. With the click of a check mark, the user would not have to type a thing.

The disadvantage of this setup is the dependency of the Always Online crawler.

First off, some users do not wish to have Always Online turned on. Without the consent of the site owner to crawl the site via Always Online, the page loader tree will not load and the user had no options of pages to install an app on.

When a user does have Always Online enabled properly, the crawler might not crawl every page the site owner wishes to install an app on.

The duty of Always Online is to make sure in the most catastrophic event for a site owner - their site being down - users can still see a version of the site via cached static HTML. Once upon a time before Always Online v2, we actually used the Google bot and other search engine crawlers’ activity to decide what to cache for the Always Online feature. We found that implementing our own crawler made more sense. Our goal is to make sure the most vital pages of a site are crawled and stored on our cache, contrasting with search engine crawler’s priority of get the most information possible from the site, thus going “deep” into the depths of a site map.

The duty of an app install on Cloudflare’s Apps platform is to seamlessly enable users to select pages in which to inject Javascript, HTML, CSS, and in the near future, Cloudflare Service Workers into. Since the objectives of the Always Online crawler differ from that of the Cloudflare Apps platform, there were inevitable consequences. Here are some examples where a page would not be crawled:

• The page’s subdomain was not "orange-clouded".

• The page was not be accessible from the site's homepage via links.

• The site’s homepage had too many links for us to follow.

• The page was password-protected, preventing us from accessing it and adding it to your site map.

• The page was added before we had a chance to crawl the site.

Although our custom crawler works well for the Always Online feature, this limited control for our customers who are installing apps. We decided to do something about it. Combining the advantages of the crawler data we already had implemented with the ability to enter any URL in an install, we created the best of both worlds.

Now, site owners can type in whatever URL they wish to install an app. There is also an option for selecting an entire directory or strictly that page. For simplicity, no regex patterns are supported.

As the apps on the Cloudflare Apps platform advance, it is vital that the platform itself advance. In the near future, the App’s platform will have the power of Cloudflare Workers, local testing, and much more to come.

There were two talks: I was preceded by designer Laura Kalbag who talked about designing icons for WordPress sites. This is something that she made look incredibly easy using a tool called Sketch. I suspect that however good Sketch is, I'd end up drawing icons that looked awful!

My talk was about using WordPress and CloudFlare together. CloudFlare has a ton of features and I highlighted some that are of great interest to WordPress users including the CloudFlare WordPress Plugin and our integration with W3TC.

The other features that people found most interesting were:

1. Always Online: CloudFlare crawls the WordPress site and keeps a copy in a special cache. If the original site goes down CloudFlare serves up the most recent version from the crawler cache with a banner indicating that it is old content. This helps keep sites online when things go badly wrong.

2. Auto-minify: many WordPress sites have large amount of HTML, CSS, and JavaScript (especially if they use lots of plugins). Auto-minify helps shrink those resources so that sites are delivered faster to web browsers.

3. Rocket Loader: a tool that reorganizes the loading of resources such as CSS and JavaScript to that they are downloaded to web browsers quickly by bundling them.

4. A new, unannounced feature that I'm calling "Help, I've gone viral!" which allows any web site owner to instantly tell CloudFlare to start completely caching a URL (overriding any caching headers set by the site) to cope with load. With this if a URL goes viral and is overloading a WordPress site it's possible to just paste in its URL and ask CloudFlare to take the load of that page. We'll be writing more about that feature when it's released.

And, of course, other CloudFlare features like Easy SSL, SPDY, and IPv6 help everyone get the latest technology onto their site quickly.

Photo credit: Hy Chalmé/Instagram

Now that Hurricane Sandy has passed and the flood waters have begun to recede, we wanted to recap what we saw over the last 24 hours across the CloudFlare network.

Our network is designed to survive hurricanes and other natural disasters, so we were confident even if some of our data centers that were in the hurricane's path failed, traffic would immediately be transferred to the next closest facility. That said, our preference is always that all our data centers remain online and able to continue to serve traffic.

Yesterday morning our ops team met to plan for the potential loss of our facilities in Newark, NJ, which we refer to by the airport code EWR, and potentially Ashburn, VA, which we refer to by the airport code IAD. Our equipment is located in an Equinix facility in both locations and we confirmed that they had taken steps to ensure their systems were tested and as hurricane-ready as they could be.

Data centers are setup so that, if power from the grid is disrupted, they switch to stored backup power until generators can kick in. In EWR, power is stored in what are, effectively, a series of car batteries. Enough power is stored in the batteries that the data center can continue to run without a new source of power for several minutes. The diesel generators are setup to kick in within that time period, usually less than a minute after a power failure is detected. The generators are intended to be able to power the facilities indefinitely so long as there is sufficient fuel. Most of the data centers from which we operate worldwide are considered "critical infrastructure" and, during an emergency, they are second in line, behind only hospitals, for delivery of diesel fuel.

The generators at our EWR facility would get a test as the storm passed overhead. At 01:31 EDT, several hours after the storm had made landfall, we received notice EWR had lost grid power. As designed, power was immediately transferred to the batteries and then to the generators. The incident description read: "Equinix IBX reported a utility power disturbance and transferred customer loads to generator power. No customers have been impacted and Site Staff reports that sufficient fuel supplies are available. Next update will be when a significant change to the situation occurs." Our systems continued to run and we did not detect any power surge or interruption.

At 08:32 EDT, we received notice that one of the EWR generators had failed: "Equinix IBX reports customer loads are on generator power, however they have a loss of redundancy do to the failure of generator 4. Engineers are investigating the issue. Next update will be when a significant change to the situation occurs." Data centers are designed for redundancy, so losing a single generator would not cause a power loss. Our systems continued to function as normal and the functional generators continued to power our equipment throughout the day. At 19:09 EDT, 11 and a half hours after the generator originally kicked in, we received notice that grid power had been restored: "Equinix IBX AMFO reports that utility power has been restored."

While we were fortunate that all of CloudFlare's facilities stayed online, other data centers and networks experienced issues. Around 02:10 EDT, our network ops team noticed a change in routing from traffic that usually transited via Level(3)'s Yellow/Atlantic Crossing-2 (AC-2) undersea cable. The cable runs from Bude, United Kingdom to Bellport, New York. While routing changed, it did not impact our customers and our network routed around the problem. We later confirmed with other network operators that AC-2 had experienced a failure.

Several regional data centers experienced outages which caused interruption to their customers' sites. In some cases, our customers had their origin data centers knocked offline. When this happens, CloudFlare's Always Online functionality kicks in and continues to serve a static version of the site until the origin is restored. The graph below illustrates the deviation from normal of websites that have triggered Always Online. At the height of the storm, beginning around 22:30 EDT and lasting until 00:30 EDT, we were 2.5 standard deviations above normal in terms of the sites on our networkwhose origin servers were offline but we were serving static copies of their sites.

One thing that was somewhat surprising is that traffic to our EWR and IAD data centers dropped less than 1% versus normal operations on a regular Monday night. We had speculated that with power outaged affecting a large number of homes and businesses throughout the Northeastern United States, traffic to the data centers would have been more impacted. Our speculation is that while fewer people may have been online, those that still had connectivity were glued to their computers and surfing more than usual.

Everyone at CloudFlare's thoughts are with the people of the Northeastern United States as they begin the process of cleaning up from this extremely destructive storm. Thanks to the police, fire fighters, rescue workers, and the teams on the ground in the region that kept the lights on and allowed us to continue to operate from the region uninterrupted.

There is a large hurricane in the Atlantic Ocean which is likely to come on-shore in the Eastern United States late tomorrow. Dubbed Sandy, the current track of the hurricane has it making landfall around Philadelphia, Pennsylvania. High winds and rain from the hurricane will likely impact two of CloudFlare's data centers: Ashburn, VA and Newark, NJ.

In both locations, the vendors that run the buildings have taken steps to ensure that service is not interrupted. Specifically:

• Backup generators have been tested, fuel tanks are full, levels verified, and backup fuel vendors have been placed on standby in the event of extended power interruption.

• Special arrangements have been made to make sure staff is available on site as necessary to maintain operability standards.

• Hotel rooms near the sites have been secured, cots are in the facilities, MREs, and other emergency supplies, are available should the situation become extreme.

That is comforting, but what is more comforting is that the CloudFlare network is designed to have no single points of failure. If connectivity is lost in any of our data centers -- due to a hurricane, earthquake, or someone just tripping over a power cable -- traffic automatically fails over to the next closest data center.

If your website is hosted in a region that is likely to be impacted by Sandy, signing up for CloudFlare can help ensure that your site stays online. Even if the effects of the hurricane are significant for the region, CloudFlare will continue to serve the static portions of your site via our Always Online feature.

We've been through hurricanes before with Irene about a year ago.We're keeping a close eye on the hurricane and will post updates to @CloudFlareSys and our System Status page if there is any impact to our Ashburn or Newark facilities. However, you can rest assured that even if there is a regional interruption, it won't impact the availability of CloudFlare's service.

The video on CloudFlare's home page promises that we will keep your web page online "even if your server goes down." It's a feature we dubbed "Always Online" and, when it works, it's magical. The problem is, Always Online doesn't always work.

This blog post is to announce that we've just released a new version of Always Online which we believe will make the feature significantly better. But, before I get to that, let me tell you a bit about the history of Always Online, how it has worked up until recently, and why it didn't always work. Then I'll turn to what we've done to create Always Online v.2.

Prior to starting CloudFlare, Lee and I ran Project Honey Pot. The Project Honey Pot website is database driven and contains a virtually infinite number of pages. One of the biggest challenges we had wasn't human traffic, which followed a predictable browsing pattern and could therefore reliably be cached, but instead dealing with traffic from automated crawlers.

These crawlers, whether legitimate (e.g., Google's bot) or illegitimate (e.g., spam harvesters), tend to crawl very "deep" into sites. As a result, they hit pages that are unlikely to have been crawled in a while and, in doing so, can impose significant load on a database. I've previously written about the hidden tax web crawlers impose on web performance.

At Project Honey Pot, Lee built a number of sophisticated caching strategies in order to help lessen the load of automated crawlers on the site's database. At CloudFlare, he realized that we could provide the same type of caching in order to cut the burden bots placed on backends. In essence, we automatically cache content for a short amount of time and, if it hasn't changed since the last request from a bot, deliver it without having to burden your web application. It works great.

In the process of building the bot content cache, Lee realized he could implement something else: a system to serve static versions of pages if an origin server fails. Using human traffic to build such a cache is dangerous because you don't want to expose one user's private information to another user (e.g., we can't cache when one user visits their bank's website to view their statement and then show that statement to another user). However, search engine crawlers are the perfect anonymous user to build a site's cache. The logic was: if it's in Google, then it's already effectively cached.

The approach of using known search engine bot traffic to build CloudFlare's cache was clever, but it had some problems. The first was that CloudFlare runs multiple data centers around the world and the cache in each is different. The solution was to find the data center with the most search engine crawler traffic and, if a copy of the page didn't exist in the local data center's cache, fall back on the "primary" data center. In our case, our Ashburn, Virginia data center received the most crawl traffic so we added a lot more disks there and used it to build up the Always Online cache.

That worked great for some sites, but for others we still would not have content in our cache when the server went offline. Seemingly bizarrely, the more static the page the less likely it was to be in our cache. The explanation was the source of the cache data: search engine crawlers. These crawlers are generally setup to visit pages that change regularly more often, and for pages that rarely change only occasionally. If the page returned a 304 "Not Modified" response then the content didn't get recached. We didn't help things by automatically expiring items in ourcache after a period of time.

The net result was, far too often, when someone's site would go offline their visitors wouldn't see a cached version of the page but, instead, a CloudFlare error page telling them that the site was offline and no cached version was available. This became one of the top complaints from our users and the visitors to their sites. When our support team dubbed the feature "Always Offline" we knew it was time to make it better.

We made a number of improvements in how we cache pages in order to improve Always Online, but the biggest change we made was to begin to actively crawl pages ourselves. CloudFlare now runs a crawler which periodically crawls our customers' pages if they have the Always Online feature enabled. The crawler's useragent is:

Mozilla/5.0 (compatible; CloudFlare-AlwaysOnline/1.0; +http://www.cloudflare.com/always-online )

You can learn more about the crawler's behavior by visiting: www.cloudflare.com/always-online. The frequency that we refresh pages in the Always Online depends on your plan. We crawl free customers once every 9 days, Pro customers onceevery 3 days, and Business and Enterprise customers daily. We are tinkering with the amount of time we spend crawling each site as well as tuning the crawler to ensure it doesn't visit sites when they're under load or otherwise impose any additional burden.

Given that we can now control exactly what is in our Always Onlinecache, our next iteration will be to turn that control over to our usersand allow you to both "pin" the pages you want to ensure are always available and "exclude" any pages you never want cached. In the meantime, we're using data we have about the most popular portions of each site in order to choose what pages to prioritize in the cache.

Our goal is to make the Site Offline error a thing of the past. We started building the new cache a couple days ago and expect everyone with Always Online to have a more robust cache available within the next few days. While everyone hopes their origin server will never go down, with Always Online v.2 we're happy to provide better peace of mind in case it ever does.

• Backup generators have been tested, fuel tanks are full, levels verified, and backup fuel vendors have been placed on standby in the event of extended power interruption;

• Special arrangements have been made to make sure staff are available on site as necessary to maintain operability standards;

• Hotel rooms near the data center have been secured, and have brought in cots, MREs, and other emergency supplies, should the situation become extreme;

• Leak diversion kits, sandbags, tarps, vinyl sheeting, are on hand with which to secure doors and windows; and

• All drains and sump pumps have been verified as free from debris and in working order.

So that's reassuring, but, you know, it's still a freakin' hurricane!

Here's what I find more reassuring: CloudFlare is designed so that whole data centers can go down and the service will keep right on running. From the beginning, we architected our system to minimize single points of failure. We regularly take whole data centers offline without anyone noticing. Traffic flows immediately to the next closest data center and things keep right on rolling.

So bring on Irene. Our Ashburn, VA data center is better prepared than most, but even if it goes down, CloudFlare will keep traffic flowing. And, if you're hosted at a provider that is knocked offline by Irene (or anything else), we've got your back and will do our best to keep you Always Online™ even if your hosting provider goes down.

Finally, for CloudFlare users who are personally in Irene's path: heed the warnings, stay dry, stay safe... we'll watch your site.

Each year, the SXSW Music, Film and Interactive Conferences (sxsw.com) solicit session ideas from the community for the upcoming event using Panel Picker. After submissions are collected, PanelPicker launches a voting interface that allows the world to pick what sessions they'd like to see at SXSW. As soon as the polls open, there is a flood of social media-fueled traffic and all sorts of other challenges for a small IT team to manage. Earlier this week the PanelPicker for the 2012 SXSW Festival began taking votes and traffic to it was intense. That's where CloudFlare came in.

While the site continues to hum along powering millions of page views, CloudFlare has helped SXSW cut bandwidth use by almost half and substantially decrease the load on their servers. SXSW manages their own system security, so they turned CloudFlare's security settings to a low and just used us for the performance benefits.

You can see the effects of enabling CloudFlare on the SXSW servers for yourself:

The SXSW conferences are being held in Austin, TX from March 9 - 18. Members of the CloudFlare team are represented on four proposed panels, so we recommend you test to make sure the PanelPicker is running fast and smooth yourself by heading over there and voting up the following proposed panels:

• Scaling to Infinity: Dealing with Rocket Growth

• Anything You Can Do, I Can Do Backwards in Heels

• JavaScript Performance MythBusters™

• Surviving Lulz: Behind the Scenes of LulzSec

Finally, let us know if you're planning on attending SXSW 2012. We plan to be there in force and would love to celebrate in Austin with everyone at SXSW from the CloudFlare community!

Today is Groundhog Day: the day on which a groundhog (named Phil) crawls out of his hole in Punxsutawney, PA and either sees his shadow, meaning we're in for six more weeks of winter, or does not, in which case it will be an early spring. Phil's official keeper is the Punxsutawney Groundhog Club. As you'd expect, the Club has a website: www.groundhog.org.

The site gets a consistent stream of visitors during the year, but every February 2 it gets slammed.

This year, the 125th Anniversary of the Groundhog Day tradition, the Punxsutawney Groundhog Club turned to CloudFlare's free service to help keep its site online during their spike in traffic. To get a sense of the volume, below is the Analytics page for the site which shows over 2.2 million page views over the last 24 hours, spiking at nearly 300,000 per hour this morning.

We're happy to report that CloudFlare did not disappoint Phil or his loyal following. In spite of the Club's backend server crashing at one point, CloudFlare's Always Online feature kept the site online throughout the day and helped substantially decrease the server load. As Glenn Kelly, who helps administer the groundhog.org site just wrote us:

Yesterday afternoon our server experienced an outage due to increased traffic. If CloudFlare were not in place, the entire website would have been offline.

CloudFlare even kept the contents of the site available when the Club took it down for scheduled maintenance late last night. In the end, even with an enormous flood of traffic, according to Glenn there was "zero downtime." And, all the while, CloudFlare's global network helped make the site about 40% faster for visitors worldwide.

Best of all, since Phil did not see his shadow, it looks like it will be an early Spring. The entire team of CloudFlare is proud to have been a little part of bringing that news to the world.

As a fellow startup, it's been really painful to watch the challenges Tumblr has gone though over the last 24 hours. With any sufficiently complex system, it is inevitable that unexpected problems will arise. It looks like Tumblr is getting back on their feet, but nothing feels worse than downtime.

That's why one of my favorite CloudFlare features is "Always Online." Think of it like insurance for your website. We use the natural pattern of search engine crawlers to build a cached copy of your site. If, for any reason, your web host or service provider goes down, we kick in to keep your site online. We can't perform magic and make a dynamic site fully functional when its backend is down, so things like shopping carts and other web applications will still cause an error. However, your basic content will stay up so your visitors can still find you and search engines will keep indexing you.

Some of my favorite messages to get from customers read like the following:

Just wanted to let your team know how Cloudflare has saved our sales ... Our site was offline most of the afternoon due too many connections trying to connect at once ... but since our site was cached through CloudFlare's Always Online service it stayed available and customers still made it to us. If it was not for CloudFlare we would have missed sales. Thank you CloudFlare for keeping our business online and our sales coming when even our site is down!

That's pretty cool.

While anyone with their own domain can sign up for CloudFlare today, we're talking with several platform providers to bring the insurance of Always Online, and the other benefits CloudFlare offers, to even more sites. Our goal is simple: to make outages like the one Tumblr just lived through a thing of the past.