In April 2024, Vercel and MERJ tracked how Googlebot handled nextjs.org, a JavaScript-heavy site that mixes static pre-rendering, server-side rendering, and client-side rendering. Every indexable HTML page they tracked was fully rendered, including content loaded through API calls. For anyone worried that JavaScript sites can’t rank, that was good news.
Eight months later, the same team looked at AI crawlers. None of the major ones they studied, including OpenAI’s crawlers and Anthropic’s ClaudeBot, were running JavaScript at all. A page that Google reads perfectly can still be close to empty for other bots.
That gap matters most on headless sites. A headless CMS stores content and delivers it through an API, and a separate front end decides what HTML each visitor and crawler receives. This article looks at how leading SEO companies approach SEO on headless CMS platforms, and why most of the important decisions are architectural ones.
Headless Removes The SEO Plugin, Not The SEO Work
In a headless setup, an SEO field only exists if someone adds it to the content model. Contentful’s developer documentation spells this out. Its pages come with a title and a slug. If you want SEO descriptions, Open Graph tags, or other metadata, you define those fields yourself and write front-end code to read them.
So the content model is where SEO starts. A sensible approach is to decide up front what every indexable page type needs, such as a title, a description, a canonical override, a robots setting, and a social image. It’s just as important to decide what the front end does when one of those fields is empty, so a missing description falls back to something useful instead of nothing.
Rendering Is The First SEO Decision
Google processes JavaScript in three phases: crawling, rendering, and indexing. Pages that return a 200 status wait in a rendering queue. That wait might be a few seconds, and it might be longer. Google’s documentation also describes the “app shell” model, where the first HTML response contains no real content and Google has to run JavaScript before it can see anything.
Google’s position on this hasn’t wavered. It still calls server-side rendering or pre-rendering a great idea, because it makes sites faster for people and crawlers, and because not every bot can run JavaScript. The old shortcut is gone, too. Dynamic rendering, which served crawlers a pre-built page while users got the JavaScript version, is now described by Google as a workaround rather than a long-term solution. Google recommends server-side rendering, static rendering, or hydration instead.
AI Crawlers Only Read What The Server Sends
The Vercel and MERJ AI crawler study gives a sense of scale. GPTBot made 569 million requests across Vercel’s network in a single month. Some AI crawlers downloaded JavaScript files, but none of the major ones executed them. AppleBot was the exception, rendering pages much the way Googlebot does.
Crawler behavior changes, so treat this as a snapshot from late 2024 rather than a permanent rule. The practical lesson still holds. If your main content only appears after JavaScript runs, Google may see it while many other crawlers don’t.
Metadata Has To Exist Before The Page Loads
Google allows JavaScript to set a canonical URL, but it warns against using JavaScript to change the canonical to something different from what’s in the original HTML. Multiple or conflicting canonical tags can lead to unexpected results.
The robots tag is even stricter. If Google finds noindex in the original HTML, it may skip rendering entirely. A script written to remove noindex later may never run. Put simply, the first HTML response has to tell the truth.
Status codes need the same care. Google’s documentation notes that in client-side routed single-page apps, returning meaningful status codes can be impractical, which leads to soft 404s. Its fix is to redirect to a URL that returns a real 404, or to add noindex to error views. Google also says it can only reliably discover links that are standard anchor tags with an href, and it recommends the History API over URL fragments for routing.
Sitemaps And Previews Become Your Responsibility
With no CMS generating a sitemap, the front end has to build one from the content API. The dates in it matter. Google says it uses the lastmod value only when it’s consistently and verifiably accurate. That means lastmod should come from when the content actually changed, not from when the site was last deployed.
Drafts need a plan as well. GOV.UK handles this through its architecture. When a publisher saves a draft, it goes to a draft content store. Only published content goes to the live content store, which the public frontend apps request as JSON and render. Drafts and live pages stay apart because they’re stored in different places, not because of a setting someone has to remember.
For preview environments that are reachable on the web, Google’s robots.txt documentation is worth knowing. Blocking crawling with robots.txt doesn’t reliably keep a page out of search results. Password protection or a noindex directive is the right tool.
What To Ask Before Hiring An SEO Agency For A Headless Build
Start with rendering. Ask which approach the agency recommends for each page type, and why. Then ask how it checks what crawlers actually receive. A solid answer involves comparing the raw HTML response with the rendered HTML in Google’s URL Inspection Tool, and accounting for crawlers that never render at all.
Then get specific. Ask to see the SEO fields in a content model the agency has built. Ask what happens to redirects when an editor changes a slug, where the sitemap’s lastmod dates come from, and how preview URLs stay out of the index. Each of these has a concrete answer, and an experienced team should be able to give it without hedging.
Listen for honesty about tradeoffs, too. Server-side rendering adds infrastructure. Static generation can delay how quickly content updates go live. A team willing to walk you through those costs is giving you the information you need to make the decision.
A headless CMS doesn’t make SEO harder so much as it makes every SEO decision explicit. The research shows Google can handle JavaScript well, but other crawlers can’t yet, which is why the HTML your server sends first is still the most important thing to get right.
Browse our vetted list of SEO agencies to find a team that knows how to handle SEO on headless CMS platforms.