Google has added a new section to its crawler and fetcher documentation for HTTP caching, which clarifies how Google's crawlers handle cache control headers. With that, Gary Illyes from Google also wrote a blog post named HTTP caching asking site owners to allow Google to cache its pages.
What is new. The new documentation now says:
"Google's crawling infrastructure supports heuristic HTTP caching as defined by the HTTP caching standard, specifically through the ETag response- and If-None-Match request header, and the Last-Modified response- and If-Modified-Since request header."
Google does not support other HTTP caching directives and if both ETag and Last-Modified response header fields are present in the HTTP response, Google's crawlers use the ETag value.
Caching decreased. Gary Illyes from Google said in the blog post that the "number of requests that can be returned from local caches has decreased." He said, "10 years ago about 0.026% of the total fetches were cacheable, which is already not that impressive; today that number is 0.017%."
"If you're in the business of making your users happy and perhaps also want to potentially save a few bucks on your hosting bill, talk to your hosting or CMS provider, or your developers about how to enable HTTP caching for your site. If nothing else, your users will like you a bit more," Gary Illyes added.
Why we care. Caching may help Google crawl your site more efficiently, which may make for a happier Googlebot. There is no mention of any SEO or ranking benefit to using caching, there is also no mention of crawl budget benefit.
Add Search Engine Land to your Google News feed.
Related stories
New on Search Engine Land
SEO for ChatGPT search: 4 key observations
How to measure YouTube ad success with KPIs for every marketing goal
How to find emerging audience needs using Google Trends
In 2025, the AI-infused world will require humans bring strategy and judgement
Google adds FAQs on site reputation abuse policy
About the author
Staff
Barry Schwartz
Barry Schwartz is a technologist and a Contributing Editor to Search Engine Land and a member of the programming team for SMX events. He owns
RustyBrick, a NY based web consulting firm. He also runs Search Engine Roundtable, a popular search blog on very advanced SEM topics.
In 2019, Barry was awarded the Outstanding Community Services Award from Search Engine Land, in 2018 he was awarded the US Search Awards the "US Search Personality Of The Year," you can learn more over here and in 2023 he was listed as a top 50 most influential PPCer by Marketing O'Clock.
Barry can be followed on X here and you can learn more about Barry Schwartz over here or on his personal site.