AI Search Optimization: The Technical Setup

AI search optimization is the technical work that makes a site readable by AI retrieval systems: allowing the right crawlers, serving content without JavaScript, structuring pages into retrievable passages, and attaching schema that matches what is on screen. It is the unglamorous half of AI visibility and it comes first. A 2026 Otterly study of over a million citations found 73% of sites had technical barriers blocking AI crawler access. A page a crawler cannot fetch is a page no model can cite.

Publish your alternative page 100% free

100% Free

Turn your product into structured comparison matrices, transparent pricing tables, and buyer tradeoffs that capture competitor search traffic on Google.

Publish your alternative page 100% free →

Step 1: Confirm the crawlers can reach you

The AI crawlers that matter, and what each one does:

A permissive robots.txt looks like this:

User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
Sitemap: https://yoursite.com/sitemap.xml

Step 2: Check what is silently blocking them

robots.txt is the easy layer. The blocks that actually bite live elsewhere:

• Cloudflare bot management. The AI scraper blocking toggle is enabled by default in some plan configurations, and it overrides robots.txt.
• WAF rules that challenge any non-browser user agent.
• Vercel firewall rules doing the same.
• Rate limiting that returns 429s once a crawler moves faster than a human.
• Cookie walls and interstitials that serve the crawler a consent screen rather than the article.

Do not assume. Check server logs, filter by those user agents, and look at the status codes. 403s and 429s tell you the answer in ten seconds.

Step 3: Server-render the content that matters

Googlebot renders JavaScript. AI crawlers largely do not. If your pricing table or comparison matrix is built by a client-side component, it is invisible to Perplexity even when the page returns a clean 200.

The test takes one command:

curl -A "PerplexityBot" https://yoursite.com/your-page | grep "your key fact"

If the fact is not in the raw HTML, no model will ever see it. On Astro, keep content in the `.astro` component and reserve React islands for genuine interactivity, hydrating with `client:visible` rather than `client:load`. On Next.js, that means server components or static generation for anything a model should read.

Step 4: Structure pages for passage retrieval

Retrieval systems chunk pages before they embed them. Write so the chunks survive.

• One idea per paragraph, two to four sentences
• Front-load the conclusion into the first sentence of every section
• Question-shaped H2s and H3s
• Real HTML tables with header rows, never screenshots of tables
• Definitions stated flatly and early, in the form "X is Y that does Z"
• Every section readable with no memory of the section before it

The practical test: copy any single H2 section into a blank document. Does it still answer something on its own? If not, it will not be retrieved on its own either.

Step 5: Attach schema that tells the truth

Schema does not make you rank. It makes you unambiguous, which is what retrieval needs.

Use `Article`, `FAQPage`, `BreadcrumbList`, and `Organization` with `sameAs` pointing to your real profiles. On product and comparison pages add `SoftwareApplication` with `offers.price`.

The rule that matters more than which types you use: the schema must match the visible page. A price of 19 in JSON-LD and "$29/mo" in the body is a contradiction that costs you trust in both systems.

Step 6: Speed, because retrieval has timeouts

Live-fetch crawlers like `ChatGPT-User` operate inside a user's waiting time. Slow pages get abandoned mid-fetch and simply do not make it into the answer.

The same Core Web Vitals targets apply: LCP under 2.5s, INP under 200ms, CLS under 0.1. The common causes on modern stacks are an oversized hero image, a font that blocks render, and a hydrated island sitting above the fold.

The 30-minute audit

1. Fetch robots.txt. Are the seven agents above allowed?
2. Grep 30 days of server logs for each agent. Any 403s or 429s?
3. `curl` your three most important pages with an AI user agent. Is the key content in the raw HTML?
4. Run the pages through a schema validator. Any errors?
5. Check one price in your schema against the price on the page. Do they match?
6. Run PageSpeed Insights on your top page. Is LCP under 2.5s?

Six checks. Most sites fail two or three, and fixing those is worth more than three months of content.

Frequently Asked Questions

How do I know if AI crawlers are visiting my site?

Should I block GPTBot to protect my content?

Does AI search optimization need a different site from SEO?

Do I need llms.txt?

Publish your alternative page 100% free

100% Free

Turn your product into structured comparison matrices, transparent pricing tables, and buyer tradeoffs that capture competitor search traffic on Google.

Publish your alternative page 100% free →

Internal links out: /blog/answer-engine-optimization (pillar), /blog/llm-seo, /blog/chatgpt-seo