15 ways your website loads from Google Search and how to measure each one
How to improve page load time for Google visitors using Signed Exchanges (part 7 of 10)
When you find a page on Google, you probably don't think much about what happens before you click it. Perhaps you've heard about prefetching, but did you know that Google employs 5 or more methods (depending on how you classify them) for loading pages? Each technique has distinct performance characteristics.
This post is a part of a series about Signed Exchanges (SXG). In an attempt to measure how SXG impacts page loading speed I needed to distinguish between different page load types. This article is based on my research and summarizes my findings.
SXG is a technology to make your website load faster for Google-referred users. If you want to implement it on your website, start here.
Main page load type categories
(Plain old) on-demand document loading
Let’s start with the obvious way of loading the page. It was there from the beginning of the web.
When the user clicks a link, the browser fetches the HTML. Then the browser fetches all the assets required for the document to display.
It’s simple and it works. However, if the server hosting the page is slow or overloaded, the user will experience a delay, which could lead to a poor experience.
To improve the situation, Google rewards websites that load quickly with better search results positions. However, Google knows that not every website in the world can become fast.
That is probably one of the reasons Google implemented prefetching.
Prefetching the document (Speculation Rules)
The idea is to download the page before the user decides to click on it. When they do, the page HTML is ready.
For the vast majority of Google results, prefetching is implemented using Speculation Rules and a private prefetch proxy.
This feature is supported on Chromium-based browsers, but support from other browsers may follow.
HTML prefetching greatly improves user experience and works with most websites out of the box, but it comes with a limitation. It doesn’t prefetch assets such as CSS styles, images, and fonts.
Prefetching the complete page (Signed Exchanges)
It is possible to have the whole page (including assets) prefetched on Google results. When the user clicks the result, the browser starts rendering the page immediately without the need to download assets first.
To achieve this, the website owner has to implement Signed Exchanges (SXG). This technology was integrated into Google search even before the HTML prefetching I described above.
It’s worth noting that only Chromium-based browsers implement SXG.
Prerendering (Accelerated Mobile Pages)
Probably the fastest way to display a page is to use Accelerated Mobile Pages (AMP). That’s because Google prerenders websites built using this technology, so when the user clicks, the page is not only prefetched but also fully rendered. It’s hard to imagine a better user experience.
On the downside, AMP has severe restrictions. Developers are forced to use a subset of HTML, JavaScript is constrained, and many other limitations apply.
It works only on mobile devices using Chromium-based browsers.
The other problem with AMP is a centralized cache fully controlled by Google. It means Google effectively became a hosting company for every possible AMP page on the Internet. Quite dystopian, if you ask me.
As AMP pages are hosted on Google, your website becomes a subpage of google.com. Your URL will look like below:
https://www.google.com/amp/s/www-yourdomain-com/yourpageThere is a way to deal with the last two problems by introducing SXG to the mix. Cloudflare even has a switch for that. But when I was writing this post, I tried hard but failed to find any example of an AMP website using it.
On-demand loading with server-side redirection (ads)
The last category describes how a page is loaded when the user clicks a Google ad. The document is loaded on demand, but with an additional HTTP/302 redirect for registering the click, which adds latency.
No prefetching. Ironically, Google’s paying customers get the worst possible page load method. If you use Google Ads, make sure your page is optimized to load fast to neutralize the latency added by Google. Another potential solution is to use AMP on your ad landing page for mobile users; however, I was unable to find an example in the wild.
Conditions, edge cases, and quirks
The above categories are just scratching the surface of a full list of possible ways the page can load when referred from Google. That’s because various factors can improve or degrade page load efficiency, and some page load types impact others.
Signed Exchanges
When mentioning prefetching the entire page using SXG above, I described the best possible scenario: the HTML and all the required assets (or subresources) are being prefetched. That’s the goal, but in the real world, things don’t always go smoothly.
Various factors can influence how the page is loaded, and they greatly impact performance. Here are the possible SXG page load types I identified:
Page prefetched with subresources
If most of your SXG-enabled page views are prefetched with subresources, you did a great job optimizing your website!
However, despite your efforts, you will notice other, less efficient page loads.
Page prefetched without subresources
The browser will use subresources only if all of them were successfully prefetched. The all-or-nothing principle states that even if one subresource fails to prefetch, when the user visits the page, all of the subresources will need to be downloaded again.
Here are the causes of missing subresources I identified:
SXG implementation errors (should not happen if you followed my previous SXG posts carefully)
Global Google SXG cache issues (very rare)
Google SXG cache housekeeping and temporary errors (happens regularly, but impacts only a small portion of page loads)
The user not spending enough time on the search results page for the prefetching to complete
User opening the page in a new tab (more on this later)
As you can see, apart from the first, the remaining factors are beyond your control.
But still, at least the HTML was prefetched, and it’s a win for performance when compared to vanilla, on-demand page load.
Page loaded on-demand using client-side redirect fallback
Sometimes the target page is unavailable in Google SXG cache. The browser will still try to prefetch it (it signals Google to populate the SXG cache with this page when possible), but will fail.
When the user finally decides to click the result, the browser will once again try to load the cached page from the SXG cache. The cache response will include a simple HTML document with a client-side JavaScript redirect. The browser will follow this redirect, loading the target page.
It’s worth noting, the redirect document introduces additional latency caused by another HTTP request, HTML parsing, and JavaScript execution.
Page loaded on-demand from Google SXG cache
Sometimes the browser doesn’t prefetch the page, but loads its SXG version on demand when the user navigates to it. I found 2 reasons for that:
Google prefetches only 1 SXG result on a page. I don’t know how Google determines which SXG result to prefetch, but it’s not always the first result for sure. If the user chooses to click the one that was not prefetched, it is loaded on demand, but still via SXG.
Google tried to prefetch the SXG page and failed. However, the user stayed long enough for the Google SXG cache to become populated. Now, when the user clicks the result, it’s loaded—on demand—from the SXG cache instead of the fallback document mentioned above.
Loading the SXG version of the page on demand introduces cryptography overhead. I suspect it comes mostly from additional requests required to download certificates for signature verification. I don’t think the CPU overhead plays a big role, because cryptography operations are cheap nowadays.
Speculation Rules
Currently, Google prefetches the page with Speculation Rules if all of the following conditions apply:
The target page is in the top 2 results, or the user hovered over the result (on desktop only).
The browser doesn’t hold any cookies for the target website. In most cases, this means the user hasn't visited the site before.
The device has enough capacity in terms of memory, network, and battery. For example, using battery-saving mode on a mobile will deactivate prefetching.
Prefetching is not disabled by browser extensions; for example, Privacy Badger by default disables prefetching.
The target website doesn’t disallow prefetching (by default, prefetching is allowed).
It’s worth noting that none of the above conditions apply to SXG prefetching.
AMP prerendering
Similarly to SXG, only one of the AMP results is prerendered. Others need to be loaded on demand from the AMP cache.
The AMP viewer is shared between the prerendered page and the others; therefore, it loads instantly every time. However, the user needs to wait for the non-prerendered pages to load in the viewer.
There are likely other edge cases when loading AMP pages, such as missing AMP cache entries during prerendering or on-demand loading.
Replicating them manually would require creating test pages, waiting for Google to index them, searching for them in Google, and hoping that edge cases manifest themselves.
I couldn’t find any tools that would make it less difficult and time-consuming. Therefore, I chose not to research these cases. If you have additional information on this topic, please leave a comment below.
Opening the page in a new tab
Most of the time, when the user opens the page in a new tab, it is loaded in the background. The user will likely switch to the tab after some time, giving it a chance to render fully, while they are still interacting with the referrer website. Therefore, I believe the page load speed is far less critical in these cases. Your performance optimizations are aimed mostly at people opening the page in an existing tab.
However, it’s good to know that opening the page in a new tab degrades performance:
Already prefetched SXG subresources are not used as mentioned previously.
The prefetched SXG HTML is also discarded, unless the CTRL+click method is used (only on desktop). I explained why in the previous part.
AMP pages have to be loaded on demand. No prerendering or prefetching.
On the other hand, Speculation Rules prefetching handles new tabs extremely reliably. It just works, regardless of how the tab is opened, on both desktop and mobile.
Duplicate prefetching
If the given page implements SXG, Google prefetches its SXG version, but at the same time uses Speculation Rules prefetching for the normal version. In my tests, I observed this phenomenon on the desktop only.
Seems like a waste of the user’s bandwidth (it is!), but it has a bright side too. If the user decides to open the page in a new tab (using a right-click menu, not CTRL+click), then the prefetched SXG version is discarded, but the normal version prefetched with Speculation Rules is used instead. Subresources have to be loaded normally, but at least the document is ready quickly.
Generic techniques for improving page load speed
If you want to compare various methods used to load a page when referred from Google, you should also consider techniques that improve performance and are not specific to Google.
You should segment your results by the technique applied for a given page load. That’s because each page load may use a different set of techniques. Mixing high- and low-performance loads in measurements increases the variance of the results, thus making it harder to analyze.
Early Hints
Early Hints allow the browser to begin fetching subresources even before the main document starts to load. This can improve the performance, especially for the pages that take time to render on the server.
Caching at the edge
If you cache HTML at the edge, such as when using Cloudflare cache, then it should be measured as a different category of page loads for the same reason as above.
If the page is delivered from the edge cache, it results in:
Subresources start loading immediately, especially if they were listed in the Link header of the response. The benefits are similar to Early Hints.
HTML becomes available much earlier than if it had to be delivered from the origin.
The cached page can use Early Hints, but I don’t see any performance benefits.
No impact on prefetching
The above page load types affect only the on-demand category of page loads. If the page is prefetched, it doesn’t matter if it was served with Early Hints or using edge cache.
Browser caching
If the user has visited the given website in the past, when they visit it again, the browser cache may contain some subresources, such as the website logo, ready to be used. If the website uses client-side HTML-caching, even the document could be saved in the browser cache, making subsequent visits instant.
When measuring the performance of various page loads, cached visits should be separated into a different category. Later, it may be included or excluded from the analysis.
Personally, I exclude it because returning users behave differently and may be less sensitive to page load speed because they know the site already.
Probably an incomplete list of page load types
Combining all the scenarios described above results in the following list of page load types. The list excludes scenarios involving a returning user and opening the page in a new tab, as those are not very useful in performance analysis.
Server Load
Server Load with Early Hints
Edge Cache Load
Speculation Rules Prefetch
SXG Prefetch with Subresources
SXG Prefetch without Subresources
SXG On-Demand Load
Server Load via SXG Redirect
Server Load with Early Hints via SXG Redirect
Edge Cache Load via SXG Redirect
AMP Prerender
AMP On-Demand Load
Server Load via Ad Redirect
Server Load with Early Hints via Ad Redirect
Edge Cache Load via Ad Redirect
That’s a lot of possibilities! I started preparing a flowchart describing the conditions needed for each page load type. However, it quickly became too complex, so I ditched this idea.
Comparison of various page load types
At first, I thought to sort all the load types by the speed, measured by Largest Contentful Paint (LCP), that they should (hypothetically) offer. It’s easy for top performers:
AMP Prerender (fastest)
SXG Prefetch with Subresources
Speculation Rules Prefetch
SXG Prefetch without Subresources (slowest)
The AMP Load is very hard to grade because it’s totally different and potentially leaner than a full page. Therefore, it may, in some conditions, load even faster than the full page with a prefetched HTML. On the other hand, if the prefetched page is optimized, AMP Load will load more slowly.
In the on-demand category, there are various ways to fetch data. I sorted them by speed:
Edge Cache Load (fastest)
Server Load with Early Hints
Server Load (slowest)
But there is one more - SXG On-Demand Load. It’s challenging to rank due to the SXG overhead and other individual factors, such as your server speed and connectivity.
The other dimension is the redirection method used. It may be ordered like this:
No redirection (fastest).
Server-side redirection (Google Ads).
Client-side redirection (SXG fallback, slowest).
How to measure different page load types in real life
Now, when you understand the difference between various page load types, it’s time to measure how they compare in terms of performance under real-world conditions. I will show you how I did it on my website.
The first thing that’s needed is a method to differentiate page load types. I created a JavaScript library page-load-type just for that.
It uses a variety of techniques to determine how the page was loaded and supports most of the load types described in this post, except AMP and Google Ads (both were not used on my website during the measurement).
Requirements
Measuring every page load doesn’t make sense. The visit should be collected only when all of the following requirements are met:
The user comes from Google. It’s easy to overlook a special case involving the SXG cache. Read on.
The browser cache doesn’t contain entries for the measured website. Otherwise, some assets may be fetched from the cache, which improves load time, but pollutes the measurement data, making it overly optimistic.
The page isn’t opened in a new tab. As discussed earlier in this post, it breaks some prefetching/prerendering methods, and at the same time, decreases the performance sensitivity of the user.
If the website, such as mine, implements SXG, additional requirements follow:
The current page supports SXG. It is perfectly fine if some of your pages, particularly non-performance-critical ones, don’t support SXG. Examples include Terms and Conditions and Privacy Policy. Make sure to insert the JavaScript measurement code only on pages with SXG implemented.
The browser supports SXG. It doesn’t make sense to include Firefox visits, for example, because Mozilla doesn’t like SXG.
Detecting Google-referred visits
To determine if a user is Google-referred, the easiest method is to check the referrer for Google domains. However, as I wrote earlier, when the SXG fallback is used, the referrer is set to the Google SXG cache (your-domain-com.webpkgcache.com). Taking everything into account, the final JavaScript function could look like this:
function isFromGoogle() {
if (!document.referrer) return false;
const regex = /(^|\.)((google\.[a-z]{2,3}(?:\.[a-z]{2})?)|(webpkgcache\.com))$/i;
try {
const referrer = new URL(document.referrer);
return regex.test(referrer.hostname);
} catch (e) {
return false;
}
}Checking the browser cache
The simplest way to ensure the cache is empty is to check whether the visitor is accessing the page for the first time.
In most cases, this can be done by checking for the presence of a cookie or local storage entry and then immediately setting it for future visits1:
function isFirstVisit() {
const returning = localStorage.getItem('returning');
localStorage.setItem('returning', 'true');
return !returning;
}Detecting a new tab
When navigating between pages, the browser maintains the history of visited pages. However, when the page is opened in a new tab (or new window), the history is not preserved. This browser behavior can be used to detect how the page was opened.
function isOpenedInNewTab() {
return(!window.history || window.history.length === 1);
}SXG support in the browser
I couldn’t find a direct way to query the browser for SXG support from JavaScript. As the SXG is implemented only in Chromium, the naive approach is to parse the User Agent.
However, on iOS devices, Chromium uses WebKit, which doesn’t support SXG. Also, some Chromium-based browsers, such as Brave, intentionally disable SXG. In some cases, the browser may spoof the User Agent to look like Google Chrome. All of this makes it impossible to reliably detect SXG support by parsing the User Agent.
Accept header to the rescue
When loading a page, the browser includes the Accept header in the request. It contains the application/signed-exchange substring if the browser supports SXG.
However, the Accept header is accessible only on the server. Therefore, the detection of SXG support should be implemented in the app or some form of middleware. Probably one of the simplest methods is to include a server-side generated <script> tag near the top of the HTML, looking like this:
<script id="sxgSupportScript">window.sxgSupport = false</script>The global sxgSupport constant can be accessed later in the frontend code.
HTML caching introduces a challenge
Things become more complex when HTML caching on the edge becomes involved. Let’s say your server generated an HTML with the sxgSupport constant set to true, which was correct for the Chrome browser requesting the page at this moment. But the page has been cached, the sxgSupport is frozen to true, and when Firefox requests the page, it gets the incorrect value of sxgSupport.
The solution is to set the sxgSupport constant after it’s retrieved from the cache. It’s a perfect task for a Cloudflare worker, which could look like this:
export default {
async fetch(request) {
// Get the original response
const response = await fetch(request);
// Check the SXG support
const acceptHeader = request.headers.get('accept') || '';
const supportsSXG = acceptHeader.includes('application/signed-exchange');
// Only process HTML responses
const contentType = response.headers.get('content-type') || '';
if (!contentType.includes('text/html')) return response;
// Create HTMLRewriter to modify the script element
const rewriter = new HTMLRewriter()
.on('script#sxgSupportScript', {
text(text) {
// Replace the entire content of the script tag
text.replace(`window.sxgSupport = ${supportsSXG}`);
}
});
// Apply the rewriter and return the modified response
return rewriter.transform(response);
// For more information, see the following blog post:
// https://www.pawelpokrywka.com/p/different-methods-of-prefetching
}
};For the instructions on how to set up and deploy Cloudflare workers, see my earlier blog post.
Including the page load type in the measurements
If you implemented all of the above and properly installed the page-load-type library in your project, the measurement code should look like this:
// If you use SXG, this code should be run only on SXG-enabled pages.
// If you don't use SXG you can skip checking for window.sxgSupport.
import getPageLoadType from "page-load-type";
if (isFromGoogle() &&
isFirstVisit() &&
!isOpenedInNewTab() &&
window.sxgSupport) {
const loadType = await getPageLoadType();
// Run measurement code when loadType becomes available
}For LCP measurement, I used DebugBear, a frontend performance monitoring tool. I added the DebugBear snippet to the <head> section and implemented page load type tracking. Here is the resulting code, with boring fragments replaced by [...] marks.
[...]
if (isFromGoogle() && [...]) {
const loadType = await getPageLoadType();
// Record page load type
window.dbbRum.push(["tag1", loadType])
}Shortly after deployment, I began receiving performance reports for each qualifying Google-referred visit, along with the page load type.
I measured user engagement metrics in Google Analytics by firing a custom event within the same if condition:
gtag('event', 'google_visit', {page_load_type: loadType});After creating a custom, event-based dimension and waiting 24 hours, I was able to use the page load type in my reports.
Results
You learned about the various methods Google uses to load your page, how to differentiate between them, and how to set up performance measurement.
Performance measurements will vary across websites, but seeing a real-life case study can help you decide whether and how SXG could improve your site's performance.
Therefore, I'll present the results for my website in the upcoming part of the series.
If the user clears browser storage and returns to the page, this method will falsely indicate that it's their first visit. The same will happen if the browser clears its storage by itself.