Product design on Vibecodr.Space is the work of deciding what kind of public place runnable software becomes. It connects creator workflow, remix culture, attribution, runtime safety, and the small product choices that make people willing to come back.

A crawler hits a Vibecodr page and finds almost nothing. Not a 404, not an error. Just an empty box, <div id="root"></div>, waiting for JavaScript it will never run. The app is in there. The words are in there. But to something that reads the web without executing it, a single-page app is a dark room with the light switch wired to a script it can't reach.

For a long time I told myself that was fine. Browsers run the JavaScript. People use browsers. Done.

Then I looked at who actually reads the web now.

A lot of it is machines. Search crawlers, link unfurlers, feed readers, and the new and growing share: the agents and training crawlers behind the models people now ask instead of searching. None of them are guaranteed to run your code. Many of them don't run it at all. And if you believe, like I do, that being on the open web means being readable on the open web, then "readable only after you execute my app" is a quiet way of being closed.

So I owed them a real page. Two of them, it turned out.

The empty #root problem

A Vibecodr page, like most modern apps, ships as a shell. The HTML the server sends is mostly an empty container and a script tag. The browser downloads the app, runs it, and only then does the page fill with content. A person never notices. A crawler that doesn't run scripts sees the shell and nothing else.

"Just put the content in the HTML" sounds like the obvious fix, right up until you remember the same URL still has to be a fast, hydrating app for the person in a browser. Serve everyone the heavy server-rendered document and you throw away the thing that makes the app feel instant. So the page has to answer two very different readers without making either one pay for the other.

Give them the document, not the shell

When a request comes from a verified crawler, the server fills #root with the real, rendered document and strips out the scripts that would have booted the app, keeping the content and its structured data. The crawler gets a complete page and is never asked to run anything.

When the request is a person's browser, the shell stays exactly as it was, so the app hydrates with no flash, and the same rendered document is tucked inside a <noscript> block, so a visitor with JavaScript disabled gets the readable page instead of a spinner that never resolves.

const rewritten = rewriteHtmlDocument(response, {
  seo: model.seo,
  ...(isBot
    ? {
        rootHtml: renderPublicVibeDocumentHtml(model),
        disableClientBootstrap: true,
        stripNonContentScripts: true,
      }
    : {
        noscriptHtml: renderPublicVibeDocumentHtml(model),
      }),
});

Two lanes, one document, rendered from the same model. A browser never paints <noscript>, so there is no flash and no double render. A crawler never gets the bootstrap, so there is no empty #root.

A verified crawler must never receive an empty #root. That is not a nice-to-have. It is the one rule that, if it breaks, makes a page silently invisible to every search engine and model at once. It is the kind of bug you don't notice until the traffic is already gone. So it is pinned by a test that walks every public, indexable route and fails the build if any of them would hand a crawler a stripped shell with no body. Most routes get their crawler body automatically; a few own it by hand, and those are the ones with no fallback, so the test makes sure every one of them actually has a body to give.

A plain-text twin for anything that asks

Server-rendered HTML is the right answer for a crawler that wants the page as a page. But a lot of agents don't want your page. They want your content. Just the content, without the navigation and the layout. So Vibecodr also serves a plain-Markdown version of its public pages to anything that asks for one.

Asking is the whole trigger. If a request says it accepts text/markdown, it gets Markdown. Otherwise it gets the app. No User-Agent guessing. Just standard, explicit content negotiation:

export function isMarkdownForAgentsRequest(request: Request): boolean {
  const accept = request.headers.get("accept");
  if (!accept) return false;
  if (!accept.toLowerCase().includes("text/markdown")) return false;
  return acceptsTextMarkdown(accept);
}

There's a second door to the same room: most pages with a twin also answer at /its-path/index.md. Ask with an Accept header or just fetch the .md URL, and either way you get identical bytes, because both routes call the same builder. There's no second copy of the content to drift. The site root is the one deliberate exception: / asked for as Markdown returns the homepage's own twin, while /index.md is mapped to the docs overview instead.

That "no second copy" is the part I care about most. The Markdown isn't scraped back out of the rendered HTML, and it isn't hand-written next to it. The page and its twin are generated from one source: the same content definition that produces the human page produces the Markdown. Add a new page and its twin, its /index.md, and its discovery link all come into being from that one definition. We learned that the hard way. An earlier version kept a hand-maintained list of which routes had twins, the content-policy page quietly fell off it, and nobody noticed until we went looking. Deriving the list from the single source closed the gap and made that whole class of mistake impossible. Agents find all of it through /llms.txt, a plain index of what's available.

Not every page gets the full treatment, and that is deliberate. The authored public pages, the docs, the marketing and editorial pages, the blog, get a real Markdown twin. Private and user-generated routes get nothing: ask for /messages or a profile in Markdown and you get a 406, not content.

User-Agent strings lie, so I stopped asking

The Markdown lane is easy to gate: it fires only when something explicitly asks for text/markdown. The document lane is harder. A crawler fetching a normal page sends a normal request. It doesn't announce itself in any way I should trust. So for that lane, the server has to decide whether the visitor is a crawler at all.

The obvious way is to read the User-Agent string. It says Googlebot or ClaudeBot right there. The obvious way is wrong.

A User-Agent is just a header. Anything can put anything in it; "I'm Googlebot" is a claim, not a credential. And it fails the other direction too: real browsers sometimes carry strings that look automated. The code even names the incident it guards against: real iOS Safari traffic that looked bot-shaped enough that a naive check would have served actual people the stripped, crawler-only page.

So Vibecodr doesn't route on what a visitor calls itself. It routes on what Cloudflare can verify about it:

const botManagement = readCloudflareBotManagement(req);
if (botManagement?.verifiedBot === true) {
  const category = readCloudflareVerifiedBotCategory(req);
  if (!category) {
    return true;
  }
  return PUBLIC_HTML_CONTENT_CONSUMING_BOT_CATEGORIES.has(category);
}
if (botManagement?.signedAgent === true) {
  return true;
}
if (isLikelyAutomatedLowScoreRequest(req)) {
  return true;
}
return false;

Read top to bottom, that's a ladder of trust. First: has Cloudflare verified this is a real crawler, checked it against the operator's known identity rather than its self-description? If so, what kind is it? Cloudflare tags each verified bot with a category, and only the ones that actually read content get the rendered document: search crawlers, AI crawlers and assistants, link previews, feed readers, archivers. The operational bots, the ones doing uptime monitoring and security scanning and webhook delivery, stay on the normal app path. They don't read the page, so there is no reason to spend a render on them, even though "bot" is right there in their name.

Below verified bots sits Web Bot Auth: an agent that signs its requests with a cryptographic signature Cloudflare validates. Below that, a last-resort heuristic for clearly-automated traffic that carries no browser fingerprint at all. And the User-Agent allowlist, the list with ClaudeBot and GPTBot and the rest, does exist, but only for logging and for keeping our robots.txt honest. It never decides what a visitor sees.

What I gave up to get there

This trades cleanly in one direction. If Cloudflare can't verify a bot, that visitor is treated as a person and gets the app shell, not the document. Maybe it's a real crawler that lost its verified status, maybe it's traffic somewhere bot verification isn't available. Either way, I would rather under-serve an honest crawler than fingerprint-guess and hand a real person a page with its scripts torn out. It is degraded, but it is safe.

And the honest crawler isn't stranded even then. It can still ask for text/markdown and read the page that way. The two systems hold each other up: the lane that decides who gets a rendered page is conservative on purpose, and the lane anyone can opt into just by asking is the relief valve.

It is the same instinct I had when I stopped fighting the browser. Stop guessing. Align with what the platform can actually tell you. Give honest readers an honest page.

I used to think of crawlers as something between a nuisance and a threat. Traffic to survive, not readers to serve. I don't anymore. A public page is a promise that anything able to read it, can. A lot of what reads the web now doesn't have eyes, won't run your code, and didn't ask for a layout. It just wants the words. Handing them over, cleanly, at a stable address, is most of what it means to be public.

The web is reading. I'd like us to be legible to it.

Braden