Your website is probably invisible to AI, and how I fixed mine

SEO & Growth · 2026-08-10

The empty room nobody told you about

When I checked what an AI crawler actually gets back from my own site, I had one of those moments where you go "hang on, that's not right." Turns out a huge chunk of the internet, including sites that look totally normal to you and me, are basically invisible to AI. Not metaphorically. Actually invisible.

Here's the thing. Most modern websites are JavaScript single-page apps. You open the site, your browser runs the JS, and content appears like magic. Feels instant, feels great. But GPTBot, ClaudeBot, PerplexityBot, and a fair few search crawlers too, don't run JavaScript. They just fetch the raw HTML and leave. And what's in that raw HTML? Often nothing. A loading spinner. An empty div waiting for a script that never runs, because the crawler's already gone.

So when someone asks ChatGPT "what does Jack do" or "tell me about this company", the assistant can only repeat what its crawler actually managed to read. If your site was an empty shell to that crawler, the assistant has nothing. It might guess. It might make something up. It might just say it doesn't know. None of those are good.

Why this actually matters now

A year or two ago I'd have shrugged at this. Who cares if a bot can't read your site, humans still can, right? But that's not really how people search any more. People are asking Perplexity questions instead of Googling. They're asking ChatGPT to summarise a company before a meeting. The assistant becomes the layer between you and the person who wants to know about you, and if that layer got fed an empty HTML shell, you basically don't exist to it.

That's the bit that got me moving. Not some abstract SEO principle, just the very real chance that someone asks an AI about my work and it comes back with nothing, or worse, something wrong.

What I actually did

First thing: I built a Cloudflare worker that detects crawler user-agents and serves them real, server-rendered HTML of the same content a human would see. So instead of the empty shell, GPTBot gets the actual page, properly rendered, no JS required. Same content, different route to get there.

Then I added an llms.txt file. It's basically a plain-text guide to the site, written for language models rather than humans, telling them what's here and where to look. Simple idea, barely any effort to set up, and it means a model reading my site gets a clear map instead of guessing.

I also added JSON-LD structured data across the site, Person, Blog, and BlogPosting schemas, so there's machine-readable metadata sitting alongside the human-readable stuff. Doesn't change what a visitor sees. Changes everything about what a crawler understands.

The sitemap was the next fix, and this one's less exciting but genuinely important: it's generated live from the database now, so when I publish a new blog post it just appears in the sitemap automatically. No manual updating, no forgetting, no stale sitemap sitting there pointing at content from six months ago. I added an RSS feed too, same logic, keep it live, keep it current.

Last piece: canonical identity. My product, Hilyt.it, builds AI-readable profile pages, basically a proper canonical source for who someone is and what they do. My personal site now points to my Hilyt profile as that canonical source. One clear answer to "who is this person", rather than scattered fragments across five different pages that might contradict each other.

The bit that surprised me

Here's a genuinely useful thing I learned along the way. If you look at raw "unique visitor" numbers from something like a CDN dashboard, they are wildly inflated. I compared mine against first-party analytics and the difference was dozens of times over. Dozens.

Turns out most of that traffic isn't people at all, it's bots and crawlers hitting the site constantly. Which is fine, that's their job, but it means if you're proudly looking at your CDN's visitor count and thinking "wow, look at all these people," you're probably looking at mostly bot traffic dressed up as humans. Measure them separately. Your real audience is smaller than the dashboard says, and that's actually fine, you just need to know the real number.

A quick checklist, if you want to check yours

View your own site with JavaScript switched off in the browser. If you see nothing, that's roughly what a crawler sees too.

Check what a crawler user-agent actually gets back, there are tools that let you fetch a page pretending to be GPTBot, it's eye-opening.

Add structured data if you don't have any.

Add an llms.txt file, it takes minutes.

Keep your sitemap live and generated automatically rather than hand-updated.

And treat bot traffic and human traffic as completely different metrics, because they are.

None of this is complicated, which is honestly the annoying part. I'd assumed there'd be some clever technical trick I was missing, and instead it was mostly just making sure the front door was actually open. Anyway, worth a look at your own site tonight, a few minutes with JavaScript off will tell you more than any traffic dashboard will.