How to Optimize for AI Search (GEO) + Templates

StepsOpen the door for search bots

1. Decide your policy

   Template A makes you visible in AI search while opting out of model training. Template B welcomes everything, training crawlers included, and is the policy maxenius.com uses for maximum visibility.

2. Write the file, repeating shared rules

   A crawler obeys only its most specific matching group, and Google’s spec says named groups and the * group are not combined. So repeat every Disallow line you need inside each named group.

   User-agent: *
   Disallow: /api/
   Disallow: /cart/
   Allow: /
   
   User-agent: OAI-SearchBot
   User-agent: ChatGPT-User
   User-agent: Claude-SearchBot
   User-agent: Claude-User
   User-agent: PerplexityBot
   User-agent: Perplexity-User
   Disallow: /api/
   Disallow: /cart/
   Allow: /
   
   User-agent: GPTBot
   User-agent: ClaudeBot
   User-agent: Applebot-Extended
   User-agent: CCBot
   Disallow: /
   
   Sitemap: https://www.example.com/sitemap.xml

   User-agent: *
   Disallow: /api/
   Allow: /
   
   User-agent: OAI-SearchBot
   User-agent: ChatGPT-User
   User-agent: GPTBot
   User-agent: Claude-SearchBot
   User-agent: Claude-User
   User-agent: ClaudeBot
   User-agent: PerplexityBot
   User-agent: Perplexity-User
   User-agent: Google-Extended
   User-agent: Applebot-Extended
   Disallow: /api/
   Allow: /
   
   Sitemap: https://www.example.com/sitemap.xml

3. Keep the mechanics clean

   Put Disallow lines before Allow: /, because simple parsers such as Python’s robotparser stop at the first match. Serve one file per host from /robots.txt. Under RFC 9309, a 5xx response means crawlers assume everything is disallowed, and a 404 means they may crawl everything. robots.txt is not access control. OpenAI and Perplexity both say changes take about 24 hours to apply.

4. Check the layer above robots.txt

   OpenAI says your host or CDN must allow its published IP ranges, and Perplexity documents WAF rules that combine user agent and IP lists. Two traps: since September 15, 2026, Cloudflare blocks mixed-purpose crawlers, including Googlebot, Applebot and BingBot, when a customer turns on the training block, and its managed robots.txt prepends Disallow rules that include Google-Extended. On Vercel, setting the AI Bots ruleset to Deny also blocks search bots.

5. Check it workedCheck it worked

   Run curl -s https://yourdomain.com/robots.txt and confirm the live file matches your repo, since a CDN may rewrite it. Then look in server or CDN logs for 200 responses to OAI-SearchBot, Claude-SearchBot, PerplexityBot and Bingbot. Spoofing a user agent with curl only tests user-agent rules, not IP-based ones.

StepsMake the page readable without JavaScript

1. Read the dated findings first

   In December 2024, Vercel and MERJ found that none of the major AI crawlers they studied rendered JavaScript, apart from Gemini through Googlebot and Applebot. Vercel’s 2026 guidance, Make your site readable by AI agents, still advises server-rendering the content you want read. Those are dated findings, not a promise about every crawler. OpenAI does not say whether OAI-SearchBot renders, so test instead of assuming.

2. Render the essentials on the server

   Title, main text, canonical, JSON-LD and navigation links should be in the initial HTML. Google says not all bots run JavaScript, even though Googlebot generally does.

3. Do not hide answers

   Keep key answers out of closed tabs, images and PDFs. Microsoft and Bing both warn against hiding critical content behind client-side rendering.

4. Check it workedCheck it worked

   Fetch the page the way a non-rendering crawler does, for example with the kit’s script: ./ai-bot-check.sh https://www.example.com/page "a sentence from the body". Every bot row should show status 200, a word count close to what you see in the browser, and PHRASE = yes. The kit has two scripts: ai-bot-check.sh tests seven bot user agents, and check-ai-crawl.sh (shown in the frameworks section) checks one fetch for title, canonical and JSON-LD.

StepsMake each section liftable

1. Lead with the answer

   Put the direct answer in the first sentences under each heading, and make headings specific. Bing’s guidelines advise surfacing key information early and defining facts explicitly on the URL.

2. One topic per URL, no scaled pages

   Do not create a page for every fan-out variation of a query. Google calls that scaled content abuse. Add first-hand, non-commodity detail instead of restating what others say.

3. No hidden text for models

   Bing treats prompt injection and hidden instructions for LLMs as abuse that can suppress or delist a site. Google says there is no ideal length and no need to chunk, so write for people.

4. Check it workedCheck it worked

   Read each H2 section alone. If it answers its own heading without the paragraph above it, it passes.

StepsAdd markup that tells the truth

1. Check the requirement

   Google says no special schema is needed for AI features, and markup must match visible text. Still use Article or BlogPosting, Organization, Product and BreadcrumbList where they apply.

2. Use JSON-LD, escape it, and give dates

   JSON-LD can sit in the head or body, and Google reads injected JSON-LD, but crawlers that skip JavaScript cannot. When you serialize JSON into a script tag, replace every < with \u003c (a backslash followed by u003c) so a string containing a closing script tag cannot end the block early. The React, Next.js and Astro samples below do this. Show a visible date and set datePublished and dateModified to match, per Google’s byline guidance.

3. Treat FAQ schema as optional

   Google ended the FAQ rich result on May 7, 2026. Keep FAQ markup only if it mirrors visible questions, and expect no extra SERP space.

4. Check it workedCheck it worked

   View source and confirm the JSON-LD is in the raw HTML. Run the Rich Results Test or Schema Markup Validator, then compare every value against what a visitor sees.

StepsDecide whether to publish llms.txt

1. Know what llms.txt is

   llms.txt is a Markdown file at your site root, proposed by Jeremy Howard on September 3, 2024, with a v2 format published in August 2026. It has an H1 (the only required part), an optional blockquote summary, and H2 sections listing links as - [name](url): notes.

2. What it does today

   Google says Search neither harms nor helps from it. None of the crawler documents we reviewed from OpenAI, Anthropic, Perplexity, Bing, Apple or Meta says their crawler reads it. Chrome’s Lighthouse has an Agentic Browsing audit that is marked Not Applicable when the file is missing. Vercel’s 2026 guidance reports that adding llms.txt did not change retrieval when the answer was already in server HTML, and that 86 percent of attributed fetches arrived through page links.

3. Decide whether to publish it

   Optional and cheap, so do it after the basics. Keep it small. llms-full.txt is a convention some documentation platforms generate, not part of the spec, and real ones run to tens of megabytes. Shopify’s default /llms.txt serves agent instructions, covered in the Shopify section.

   # Example Co
   
   > Example Co makes refillable cleaning products and sells them online to customers in the US and Canada.
   
   Key facts:
   - Free shipping on US orders over $50; returns accepted within 30 days.
   - Support: support@example.com, Monday to Friday.
   
   ## Products
   
   - [All products](https://www.example.com/collections/all): Full catalog with prices and stock status
   - [Refill subscriptions](https://www.example.com/pages/subscriptions): How refills are billed, paused and cancelled
   
   ## Policies
   
   - [Shipping policy](https://www.example.com/policies/shipping-policy): Carriers, delivery times and costs
   - [Refund policy](https://www.example.com/policies/refund-policy): Return window and how refunds are paid

4. Check it workedCheck it worked

   Run curl -sI https://yourdomain.com/llms.txt and expect 200 with a text content type. Then paste only the file into an assistant and ask it questions about your business.

StepsSay who you are once, at one address

1. Publish Organization markup

   Add Organization JSON-LD on the home page with name, url, logo, description and sameAs links to your real profiles, following Google’s Organization doc. Use the same name everywhere, as Bing’s guidelines ask, and keep Business Profile, Bing Places and Merchant Center data in step.

2. Pick one host

   Your canonical tag, sitemap loc values and redirect target must agree. Google treats redirects and rel=canonical as strong signals and the sitemap as weak, and advises against declaring different URLs by different methods. Bing notes that LLMs cluster near-duplicates and may pick an outdated URL to represent them.

3. Set lastmod from real changes

   Change lastmod only when the main content, structured data or links change, never to the build time. Google ignores priority and changefreq, and Bing says the same. Pair it with a visible date and dateModified.

4. Ping IndexNow for Bing

   IndexNow notifies Bing and other participating engines of changes. Google is not a participant. An HTTP 200 means your URL was received, not indexed.

5. Check it workedCheck it worked

   Run curl -sI https://example.com/page | grep -i -E '^HTTP/|^location' for each host variant, then compare the redirect target with the page’s canonical and its sitemap entry. All three must be the same URL.

StepsSet up WordPress for AI search

1. Make sure the content is in the HTML

   Open a key page, choose View Source (not Inspect), and search for a sentence from the body. Or run curl -s https://example.com/page/ | grep -i "a sentence from the page". Themes and mainstream page builders render on the server, but content in AJAX tabs, sliders and “load more” widgets may not be there.

   Make sure navigation uses real <a href> links, because Google can only crawl links built that way. Then open Settings > Reading and confirm “Search engine visibility” is unchecked. When it is on, current WordPress adds a noindex robots meta to every page rather than a robots.txt Disallow.

   Check: your sentence appears in the curl output, and curl -s https://example.com/ | grep -i noindex returns nothing.

2. Take control of robots.txt

   With no physical file, WordPress builds a virtual robots.txt and passes it through the robots_txt filter. A physical robots.txt in the web root overrides that filter and every plugin editor in the table, and Yoast’s File editor creates exactly such a file. Find out which situation you are in before editing anything.

   If you want an explicit AI group, use the geo-robots.php mu-plugin below. It repeats the admin rules inside the group, because under RFC 9309 a crawler that finds a group naming it obeys only that group.

   Leave AIOSEO’s Block AI Crawlers and Yoast Premium’s Bot blocker off if you want AI search visibility. If the site sits behind Cloudflare, check its managed robots.txt, which adds Disallow rules above your file.

   Check: curl -s https://example.com/robots.txt shows the rules you expect. OpenAI says changes can take about 24 hours to reach ChatGPT search.

3. Decide which bots to allow

   Search and citation bots are OAI-SearchBot, Claude-SearchBot and PerplexityBot. Training bots such as GPTBot and ClaudeBot are a separate decision, and blocking them does not remove you from that vendor’s search. The full vendor table is in the foundations section.

   Check: each bot you want in search is not named in a Disallow group.

4. Publish llms.txt from one source only

   Pick the plugin feature in the table, or use geo-llms-txt.php if you have no plugin. Never run two. Yoast will not overwrite an existing file, and a static file can mask a dynamic one from another plugin.

   The payoff is uncertain, as covered in the llms.txt section. Rank Math’s own documentation says adoption is limited. It costs little, so treat it as a tidy index of your best pages.

   Check: curl -sI https://example.com/llms.txt returns 200, and the body starts with # Your site name.

5. Add structured data through your plugin

   Do not paste a second Organization block. Yoast outputs one linked @graph and exposes filters such as wpseo_schema_organization (Schema API). Set the name, logo and profiles first at Yoast SEO > Settings > Site representation. Rank Math lets you extend the publisher through rank_math/json_ld. Without a plugin, use geo-schema.php.

   Check: curl -s https://example.com/ | grep -c 'application/ld+json', then paste each block into validator.schema.org and confirm there is one Organization node.

6. Fill in author and entity details

   Complete Users > Profile (display name, biographical info, website) because SEO plugins read author data from it. Per Google’s Article guidance, author.name holds only the name and author.url points to a page that uniquely identifies the author. For the company, Organization markup belongs on the home or About page and recommends name, url, logo, sameAs and contact details.

   Check: a post’s byline links to an author page, and the page source shows the same name in the Article JSON-LD.

7. Link internally with descriptive anchors

   From each hub page (services, category, guide) link to detail pages with anchor text that describes the target. “Our WooCommerce setup checklist” tells a crawler more than “read more”.

   Check: every important page has at least one text link pointing to it from another page.

8. Run one sitemap generator

   Use core, Yoast, Rank Math or AIOSEO, not several. Submit the URL in Google Search Console and Bing Webmaster Tools. Bing asks for accurate lastmod values in ISO 8601 with date and time.

   Check: the sitemap URL from the table loads, and lastmod changes only when you actually edit a page.

9. Turn on IndexNow, one sender

   Choose the official IndexNow plugin, Rank Math’s Instant Indexing, AIOSEO, or Yoast Premium. Running two duplicates submissions. IndexNow shares submitted URLs with all participating engines.

   Check: publish a test post and look for it in the plugin’s recent submissions list.

10. Complete WooCommerce product data

    For each product, fill Product data > Inventory > SKU and the “GTIN, UPC, EAN, or ISBN” field, and write a real short description. WooCommerce 11.1.2 outputs Product JSON-LD with sku, gtin, offers and ratings, and lets you extend it with woocommerce_structured_data_product. The filters file below adds a brand. If an SEO plugin also outputs Product, compare the two blocks.

    WooCommerce trunk has OnlineStore and MerchantReturnPolicy markup planned for a later release (marked 11.3.0), so it is not in 11.1.2.

    Check: view source on a product page and confirm exactly one Product node with your GTIN in it.

11. Check it workedVerify everything from a terminal

    Replace the domain and phrase, then run all four:

    curl -s https://example.com/robots.txt shows your intended groups and a Sitemap line.

    curl -sI https://example.com/llms.txt returns 200 and text/plain.

    curl -s https://example.com/ | grep -c 'application/ld+json' returns the number of blocks you expect.

    curl -s https://example.com/some-page/ | grep -i "a sentence from the body" finds your text.

StepsSet up a Shopify store for AI search

1. Check what your store already serves

   Open these four URLs on your store and note what you see: /robots.txt, /llms.txt, /agents.md and /sitemap.xml.

   Do not expect your robots.txt to match anyone else’s. On October 7, 2026 we saw newer stores serving a default whose comments point agents to /agents.md, and other stores still serving the older default with extra groups for tools like Nutch, AhrefsBot and Pinterest. Both leave AI crawlers under *.

   The default /llms.txt is a set of agent instructions (UCP discovery, an MCP endpoint, read-only JSON endpoints, policy links). It is not a curated index of your content. Then run one product URL through Google’s Rich Results Test.

2. Review Agentic storefronts

   In the admin, go to Sales channels > Agentic. Per Shopify Help, this is on by default for eligible stores, and it requires agreeing to the Agentic Storefronts Supplemental Terms. Shopify currently lists ChatGPT, Microsoft Copilot, Google AI Mode and Gemini, and Meta.

   What each channel needs:

   • ChatGPT: it is a discovery referrer. Shoppers finish checkout on your own store. Your store must sell to US customers, though it can be based elsewhere.
   • Copilot and Google: both offer a Shopify-powered checkout inside the chat for eligible US stores, and both need your store policies completed. Google also needs a Merchant Center account, with products synced through the Google and YouTube channel.
   • Shopify Catalog: per the requirements page, you need the Starter plan or higher and a store that is not in private mode. Each product needs a title, at least one image, a price above zero, and publication to a channel with a URL. Unlisted or search-hidden products do not qualify.

   Use the search preview on that screen. It rates each listing on description completeness, image coverage, reviews, variant data and policies. Turning off ChatGPT Catalog access can take up to 7 days to take effect.

3. Fix your product data

   This is where most of the gain is. Shopify’s Optimizing products for AI platforms page covers title, description, images, type, vendor, collections, tags, barcode, variant option names and policies. Items 1, 3, 4 and 9 below are our own recommendations, with sources where we have them:

   1. Title: specific, with product type, key attribute and brand. Keep the SEO title near 60 characters.
   2. Description: answer what buyers ask (materials, dimensions, who it is for, care).
   3. Images: several, each with descriptive alt text.
   4. Category: set under Products > (your product) > Category from the Standard Product Taxonomy, then fill the category metafields.
   5. Type, vendor, collections and tags: fill all four.
   6. Barcode: ISBN, UPC or GTIN on each variant that has one.
   7. Variant option names: “Size” and “Color”, not “Option 1”.
   8. Policies: refund, privacy, terms and shipping, completed.
   9. Reviews: the search preview counts them. The structured_data example on shopify.dev shows no aggregateRating, so check the rendered JSON-LD from your reviews app.
   10. Knowledge Base app (optional): you review auto-generated store facts and write FAQ answers for AI platforms. Shopify says those answers do not appear on your storefront and the app does not change how often you appear in AI results.

4. Customize llms.txt with a theme template (optional)

   Go to Online Store > Themes > … > Edit code, then add a template in the templates folder. Per the agents.md.liquid docs, agents.md.liquid covers all three URLs, while llms.txt.liquid covers only that one. Shopify looks for the path-specific template, then agents.md.liquid, then its own default.

   Only the request and agents objects exist in these templates. You cannot loop over collections or articles, so write your links by hand and keep them current. Keep the UCP and MCP lines dynamic through agents.* rather than hardcoding them, and never output private data like emails or phone numbers.

   Redirects, Files and app proxies do not work for this file. URL redirects fire only on broken URLs, and /llms.txt now returns 200. Uploaded Files do not serve from the root. App proxies only work under /a, /apps, /community and /tools. The template is the supported route.

   Treat this file as a courtesy for agents, not a ranking lever, since Google Search does not use llms.txt.

5. Touch robots.txt.liquid only if you must change access

   Go to Online Store > Themes > … > Edit code > Add a new template > robots > Create template. Shopify calls this unsupported, Support cannot help with it, and it warns that a mistake can cost you all traffic. Copy your current /robots.txt somewhere first. Deleting the template restores the default.

   Keep Shopify’s default loop and add plain-text groups around it. The usual reason is opting out of model training, for example GPTBot. OpenAI documents that GPTBot and OAI-SearchBot are independent, so blocking GPTBot does not remove you from ChatGPT search.

   Do not add Allow groups to welcome AI bots. See the first pitfall below.

6. Leave Product markup to the theme and enrich the rest

   Dawn 16.0.0 and Horizon already output {{ product | structured_data }} and {{ article | structured_data }}. Do not add a second hand-written Product block. Two entities for one product conflict.

   To strengthen your Organization data, fill in the social links under Theme settings > Social media and upload a logo. In Dawn’s header those feed sameAs, and empty settings render as empty strings. For a visible FAQ page, add the faq-schema.liquid section below, using the | json filter, and only for questions that are on the page.

7. Publish content that answers buyer questions

   Write buying guides and comparisons under Content > Blog posts and link them to products and collections. Give each collection a real description with its own SEO title (Shopify recommends up to 60 characters) and a description of about 160. Put a visible FAQ on a page under Online Store > Pages, then link it from your llms.txt.liquid.

8. Check it workedCheck it worked

   Run these from a terminal, replacing the domain and product handle:

   curl -sI https://yourstore.com/llms.txt | head -n 5 should show 200, and a GET should show your edited content.

   curl -s https://yourstore.com/llms.txt | head -n 20 should show your headings and links.

   curl -sI https://yourstore.com/agents.md and curl -sI https://yourstore.com/sitemap.xml should both return 200.

   curl -s https://yourstore.com/robots.txt should still contain the cart and checkout disallows under *.

   curl -s https://yourstore.com/products/your-handle | grep -oE '"@type" ?: ?"(Product|ProductGroup)"' | sort | uniq -c should show one entity.

   Re-check the Agentic search preview afterward.

StepsMake a React SPA readable without JavaScript

1. Diagnose with view-source

   Open your key page, choose View Source (not DevTools) and look for your headline. An empty div id="root" means non-rendering crawlers get nothing. Then run check-ai-crawl.sh above and look for “Phrase in raw HTML: YES”.

2. Pick a fix, in this order

   1. React Router framework mode with prerender and ssr: false. Its docs say pre-rendering is useful for SEO, especially for deployments without server rendering.
   2. Move to Next.js, covered next.
   3. For a content-heavy marketing site, Astro with React islands.
   4. A custom prerender with react-dom/static. It works but is low-level, and frameworks normally call it for you.

   Skip dynamic rendering. Create React App has been deprecated for new apps since February 14, 2025. Check the React Router docs for the version you have installed, since prerender options are documented per version.

3. Prerender every content URL, slugs included

   prerender: true skips dynamic paths such as /blog/:slug. Use the function form and add the slugs yourself.

   import type { Config } from '@react-router/dev/config'
   import { getPostSlugs } from './app/lib/posts'
   
   export default {
   // No runtime server: every listed URL becomes a static HTML file at build time.
   ssr: false,
   async prerender({ getStaticPaths }) {
     const slugs = await getPostSlugs()
     return [...getStaticPaths(), ...slugs.map((s) => '/blog/' + s)]
   },
   } satisfies Config

4. Put head tags in the route component

   React Router recommends the built-in title, meta and link elements over the route module’s meta export. React 19 hoists them into the document head. In our build they landed in head of the prerendered file. Add JSON-LD in the same component and escape <, because JSON.stringify leaves </script> intact.

   import { data, useLoaderData } from 'react-router'
   import { getPost } from '../lib/posts'
   
   const SITE_URL = 'https://www.example.com'
   
   // With prerendering, this loader runs at build time, so the post is in the HTML file.
   export async function loader({ params }: { params: { slug?: string } }) {
   const post = await getPost(params.slug ?? '')
   if (!post) throw data('Not found', { status: 404 })
   return { post }
   }
   
   export default function BlogPost() {
   const { post } = useLoaderData<typeof loader>()
   const url = SITE_URL + '/blog/' + post.slug
   const article = {
     '@context': 'https://schema.org',
     '@type': 'Article',
     headline: post.title,
     description: post.description,
     datePublished: post.publishedAt,
     mainEntityOfPage: url,
   }
   return (
     <article>
       <title>{post.title}</title>
       <meta name="description" content={post.description} />
       <link rel="canonical" href={url} />
       <script
         type="application/ld+json"
         dangerouslySetInnerHTML={{ __html: JSON.stringify(article).replace(/</g, '\u003c') }}
       />
       <h1>{post.title}</h1>
       <p>{post.body}</p>
     </article>
   )
   }

5. Deploy with a narrow fallback

   Serve build/client. Rewrite only paths you did not prerender to __spa-fallback.html. That rewrite answers 200 for every unknown URL, so keep your sitemap exact. In its 2024 data Vercel saw ChatGPT’s crawler spend a large share of fetches on 404s, so broken or stray URLs cost you.

6. Check it workedCheck it worked

   The build log should print a line like Prerender (html): /blog/hello-geo -> build/client/blog/hello-geo/index.html. Open that file, confirm the title, canonical and article text are in it, then run check-ai-crawl.sh against the deployed URL. You want Phrase in raw HTML: YES.

StepsShip answer text and metadata in the first response

1. Keep answer text in Server Components

   Pages and layouts are Server Components by default. Client Components are prerendered too, but data loaded in useEffect is not in that HTML. In our 16.4.0 test, a 'use client' page that loaded its text in an effect shipped HTML containing only “Loading…”. Use Client Components for interactivity, not for the answer.

2. Set root metadata with metadataBase

   In app/layout.tsx, export metadata with metadataBase: new URL(SITE_URL), a title template, alternates.canonical and openGraph. Metadata exports only work in Server Components.

3. Use generateMetadata and generateStaticParams

   params is a Promise in current versions. Prerendering posts means the metadata and the full article are in the initial HTML, and the route shows as SSG in the build table.

   import type { Metadata } from 'next'
   import { notFound } from 'next/navigation'
   import { getPost, getPosts } from '@/lib/posts'
   import { SITE_URL } from '@/lib/site'
   import { jsonLd } from '@/lib/json-ld'
   
   type Props = { params: Promise<{ slug: string }> }
   
   // Prerender every post at build time so the full article is in the HTML response.
   export async function generateStaticParams() {
   return (await getPosts()).map((p) => ({ slug: p.slug }))
   }
   
   export async function generateMetadata({ params }: Props): Promise<Metadata> {
   const { slug } = await params
   const post = await getPost(slug)
   if (!post) return {}
   return {
     title: post.title,
     description: post.description,
     alternates: { canonical: '/blog/' + post.slug },
   }
   }
   
   export default async function Page({ params }: Props) {
   const { slug } = await params
   const post = await getPost(slug)
   if (!post) notFound()
   
   const article = {
     '@context': 'https://schema.org',
     '@type': 'Article',
     headline: post.title,
     datePublished: post.publishedAt,
     dateModified: post.updatedAt ?? post.publishedAt,
     mainEntityOfPage: SITE_URL + '/blog/' + post.slug,
   }
   
   return (
     <article>
       <script type="application/ld+json" dangerouslySetInnerHTML={{ __html: jsonLd(article) }} />
       <h1>{post.title}</h1>
       <div>{post.body}</div>
     </article>
   )
   }

4. Render JSON-LD as a native script, escaped

   The Next.js JSON-LD guide says to use a plain script tag, not next/script, and to replace < with its unicode escape because JSON.stringify does not sanitize. Put that in one helper.

   // JSON.stringify does not escape "<", so "</script>" inside a value would end the tag early.
   export function jsonLd(data: object): string {
   return JSON.stringify(data).replace(/</g, '\u003c')
   }

5. Write robots.ts with repeated disallows

   app/robots.ts accepts an array for userAgent. A crawler obeys only its most specific matching group, so repeat the disallows. Treat training crawlers as a separate decision from search bots.

   import type { MetadataRoute } from 'next'
   import { SITE_URL } from '@/lib/site'
   
   export default function robots(): MetadataRoute.Robots {
   return {
     rules: [
       { userAgent: '*', allow: '/', disallow: ['/api/', '/account/'] },
       {
         userAgent: [
           'OAI-SearchBot', 'ChatGPT-User', 'Claude-SearchBot', 'Claude-User',
           'PerplexityBot', 'Perplexity-User', 'GPTBot', 'ClaudeBot', 'Google-Extended',
         ],
         allow: '/',
         disallow: ['/api/', '/account/'],
       },
     ],
     sitemap: SITE_URL + '/sitemap.xml',
   }
   }

6. Write sitemap.ts with real lastModified

   Google ignores priority and changefreq and uses lastmod only when it is consistently accurate. Take the date from the content, never new Date().

   import type { MetadataRoute } from 'next'
   import { getPosts } from '@/lib/posts'
   import { SITE_URL } from '@/lib/site'
   
   export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
   const posts = await getPosts()
   return [
     { url: SITE_URL, lastModified: '2026-10-01' },
     ...posts.map((p) => ({
       url: SITE_URL + '/blog/' + p.slug,
       lastModified: p.updatedAt ?? p.publishedAt,
     })),
   ]
   }

7. Serve llms.txt from a route handler

   GET Route Handlers are not cached by default, so add force-static. If you enable cacheComponents, delete that line: the option is removed in that mode, and a handler that reads no request data is prerendered. Build the list from the same posts function the pages use.

   import { getPosts } from '@/lib/posts'
   import { SITE_NAME, SITE_URL } from '@/lib/site'
   
   // Delete this line if cacheComponents is enabled.
   export const dynamic = 'force-static'
   
   export async function GET() {
   const posts = await getPosts()
   const body = [
     '# ' + SITE_NAME,
     '',
     '> One or two sentences: what you do, for whom, and where.',
     '',
     '## Guides',
     ...posts.map((p) => '- [' + p.title + '](' + SITE_URL + '/blog/' + p.slug + '): ' + p.description),
     '',
   ].join('\n')
   
   return new Response(body, { headers: { 'Content-Type': 'text/plain; charset=utf-8' } })
   }

   The same caveat as before applies: this file is a supplement to server HTML, not a substitute for it.

8. Handle request-time routes: the htmlLimitedBots trap

   Routes that read cookies, headers or uncached data render at request time. Since Next.js 15.2, their generateMetadata output streams into body for any bot not on the HTML-limited list. We read the default list in the source and it has no AI crawlers. In our 16.4.0 test, GPTBot and ClaudeBot got the title in body by default and in head after we added them.

   Setting the option replaces the default list, so copy the default and append the AI bots. Re-check the copy on every Next.js upgrade. Fully prerendered pages are not affected. The Next.js docs say streaming does not hurt SEO; that statement is about Google, which runs JavaScript, so do not stretch it to AI crawlers.

   import type { NextConfig } from 'next'
   
   // Copied from the Next.js default (canary html-bots.ts, October 2026). Re-check on upgrade.
   const NEXT_DEFAULT = String.raw`[\w-]+-Google|Google-[\w-]+|Chrome-Lighthouse|Slurp|DuckDuckBot|baiduspider|yandex|sogou|bitlybot|tumblr|vkShare|quora link preview|redditbot|ia_archiver|Bingbot|BingPreview|applebot|facebookexternalhit|facebookcatalog|Twitterbot|LinkedInBot|Slackbot|Discordbot|WhatsApp|SkypeUriPreview|Yeti|googleweblight`
   const AI_CRAWLERS = 'GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User'
   
   const nextConfig: NextConfig = {
   htmlLimitedBots: new RegExp(NEXT_DEFAULT + '|' + AI_CRAWLERS, 'i'),
   }
   
   export default nextConfig

9. Decide whether you need static export

   With output: 'export', Next.js writes HTML to out/ and Route Handlers must be force-static to be prerendered. That suits a brochure site hosted on plain storage.

10. Check it workedCheck it worked

    In the next build route table your content routes should show as static or SSG, not as dynamic. Then fetch /robots.txt, /sitemap.xml and /llms.txt (the last should return text/plain; charset=utf-8) and run check-ai-crawl.sh on one post. An unknown slug should return 404.

StepsAdd AI-readable files to an Astro site

1. Keep answer text out of client:only

   Components without a client:* directive render to plain HTML. client:only skips server rendering entirely, so never put answer text in one.

2. Add a JSON-LD component that escapes

   set:html is not escaped by Astro, and JSON.stringify leaves </script> alone. In our Astro 7.0.5 test, a description containing </script> ended the block early and it failed JSON parsing. Escaping < fixed it. Astro does not process a script tag that has attributes other than src, so the block is output as written.

   ---
   interface Props {
   data: Record<string, unknown>;
   }
   const { data } = Astro.props;
   const json = JSON.stringify(data).replace(/</g, '\u003c');
   ---
   <script type="application/ld+json" set:html={json} />

3. Generate llms.txt from the content collection

   A static file endpoint at src/pages/llms.txt.ts builds /llms.txt at build time. Reading from getCollection() means it cannot drift from the published pages.

   import type { APIRoute } from 'astro';
   import { getCollection } from 'astro:content';
   
   export const GET: APIRoute = async ({ site }) => {
   const url = (path: string) => new URL(path, site).href;
   const posts = (await getCollection('blog', ({ data }) => !data.draft)).sort(
     (a, b) => b.data.publishDate.getTime() - a.data.publishDate.getTime(),
   );
   
   const body = [
     '# Example Co',
     '',
     '> One or two sentences: what you do, for whom, and where.',
     '',
     '## Guides',
     ...posts.map((p) => '- [' + p.data.title + '](' + url('/blog/' + p.id) + '): ' + p.data.description),
     '',
   ].join('\n');
   
   return new Response(body, { headers: { 'Content-Type': 'text/plain; charset=utf-8' } });
   };

4. Choose one source for robots.txt

   Use public/robots.txt or an endpoint, not both. The endpoint can reuse site and repeat the rules in the AI group, as the sitemap integration docs show for sitemaps.

   import type { APIRoute } from 'astro';
   
   const AI_AGENTS = [
   'OAI-SearchBot', 'ChatGPT-User', 'Claude-SearchBot', 'Claude-User',
   'PerplexityBot', 'Perplexity-User', 'GPTBot', 'ClaudeBot', 'Google-Extended',
   ];
   
   export const GET: APIRoute = ({ site }) => {
   const sitemapURL = new URL('sitemap-index.xml', site);
   const rules = 'Allow: /\nDisallow: /api/';
   return new Response(
     [
       'User-agent: *\n' + rules,
       // A crawler that matches a named group ignores the * group, so repeat the rules.
       AI_AGENTS.map((ua) => 'User-agent: ' + ua).join('\n') + '\n' + rules,
       'Sitemap: ' + sitemapURL.href,
       '',
     ].join('\n\n'),
   );
   };

5. Set sitemap lastmod from frontmatter, for .md and .mdx

   @astrojs/sitemap needs the site option and cannot list SSR dynamic routes. Use serialize() to set lastmod from each post. Match both extensions, or converting a post to MDX silently drops its date. We found this on our own site.

   import { defineConfig } from 'astro/config';
   import sitemap from '@astrojs/sitemap';
   import { readdirSync, readFileSync } from 'node:fs';
   
   const postDates = Object.fromEntries(
   readdirSync('./src/content/blog')
     .filter((f) => /\.mdx?$/.test(f))
     .map((f) => {
       const fm = readFileSync('./src/content/blog/' + f, 'utf8').split('---')[1] ?? '';
       const date = fm.match(/^updatedDate:\s*(\S+)/m)?.[1] ?? fm.match(/^publishDate:\s*(\S+)/m)?.[1];
       return ['/blog/' + f.replace(/\.mdx?$/, ''), date];
     }),
   );
   
   export default defineConfig({
   site: 'https://www.example.com',
   integrations: [
     sitemap({
       serialize(item) {
         const path = new URL(item.url).pathname.replace(/\/$/, '');
         const date = postDates[path];
         if (date) item.lastmod = new Date(date).toISOString();
         return item;
       },
     }),
   ],
   });

6. Link the file so agents find it

   Add link rel="alternate" type="text/plain" href="/llms.txt" or a footer link. The llms.txt v2 spec uses rel="describedby" for this.

7. Check it workedCheck it worked

   Run npx astro build. Open dist/llms.txt, dist/robots.txt, dist/sitemap-0.xml and one dist/blog/your-post/index.html. After deploy, run check-ai-crawl.sh on the live URL and look for “Phrase in raw HTML: YES”.

StepsMeasure AI search visibility

1. Separate impressions from AI features

   Start here for any page with high impressions and almost no clicks.

   1. In Search Console, open Performance, filter Page to your URL, and note impressions, clicks, CTR and average position.
   2. Open the Generative AI performance report, go to the Pages tab, and find the same URL.
   3. Compare the two impression counts. The AI report is a subset of Web, so never add them. If AI features account for most impressions, a good position with a low CTR is expected, because all links in an AI Overview share one position.
   4. If the report is missing, Google says the site has not had enough AI impressions yet. That is not an error.
   5. Open Settings > Search generative AI and confirm the property is not set to Exclude and is not inheriting Exclude from a parent. See Search generative AI control.
   6. Add an annotation on the day you ship changes: right-click the Performance chart, up to 120 characters (annotations help).

2. Use the built-in AI Assistant channel

   In GA4, go to Reports > Acquisition > Traffic acquisition. Set the primary dimension to Session default channel group and look for the AI Assistant row. Add Session source as a secondary dimension to see which assistant sent each session. Clicks from AI Overviews and AI Mode will not be here. In Plausible, open Sources > Channels > AI Assistants.

3. Add a custom channel group

   Go to Admin > Data display > Channel groups, create a group, and add a channel where Source matches the regex below. Drag it above Referral, because traffic goes to the first channel it matches. Standard properties allow 2 custom groups, and custom groups are not in the BigQuery export.

   Google’s own sample regex in that help article over-matched in our test: it also caught help.openai.com, github.com/copilot, chatgptlist.org and “shanghai”. Use this narrower one instead.

   ^(.*\.)?(chatgpt\.com|chat\.openai\.com|perplexity\.ai|claude\.ai|copilot\.microsoft\.com|gemini\.google\.com)$

   It accepts chatgpt.com, chat.openai.com, www.perplexity.ai, claude.ai, copilot.microsoft.com and gemini.google.com. It rejects google, mail.google.com, openai.com, help.openai.com, notchatgpt.com, perplexity.ai.evil.com and bing.com. It covers the www host and the old ChatGPT host because chat.openai.com redirects to chatgpt.com and perplexity.ai redirects to www.perplexity.ai. GA4 regex is full-match and case-sensitive by default, uses RE2, and report filters cap at 250 characters. The anchors keep this pattern correct in both match modes. Add other assistants only after they show up in your own Session source report.

4. Watch for lost UTMs

   Look in Session source / medium for chatgpt.com rows sitting in Unassigned or Referral, because Google does not document how GA4 resolves a utm_source with no medium against the AI Assistant rule. A redirect that drops the query string, such as a trailing-slash, www or http-to-https rule, also drops the UTM. Test a tagged URL through every redirect you run.

5. Build an Exploration (optional)

   Skip this if the built-in channel report answers your question. In Explore, start a blank exploration. Import the dimensions Session source, Session source / medium, Landing page + query string and Session default channel group, plus the metrics Sessions, Engaged sessions and Key events. Put Session source and then Landing page + query string in Rows, and filter Session source with the same regex.

6. Read Copilot and Bing citations

   1. Verify the site in Bing Webmaster Tools and open the AI Performance report.
   2. Read Total Citations, Average Cited Pages, grounding queries and the Pages list. Click a grounding query to see its cited pages, or a page to see its grounding queries.
   3. Use Compare to overlay the period after you shipped changes. Intents, Topics and Citation Share are preview features.
   4. Export CSV or Excel monthly. Read the FAQ for the sampling and refresh rules.
   5. Submit changed URLs through IndexNow. Bing recommends it so AI answers use the current page version.

7. Count AI bot hits in server logs

   Look for these user agents. Search: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, bingbot. Training: GPTBot, ClaudeBot. User-triggered: ChatGPT-User, Claude-User, Perplexity-User. Google-Extended will never appear, because it is a robots.txt token with no user agent of its own.

   User-triggered hits matter most. Each one means a real conversation read your page, and because these fetches do not run JavaScript, GA4 and Plausible never see them.

   A quick count from a combined-format log. This awk version counts each line once, where a plain grep counted bingbot twice per line:

   awk 'match($0, /OAI-SearchBot|ChatGPT-User|GPTBot|Claude-SearchBot|Claude-User|ClaudeBot|PerplexityBot|Perplexity-User|Googlebot|bingbot/) { print substr($0, RSTART, RLENGTH), $7 }' access.log | sort | uniq -c | sort -rn

   User agents are easy to fake, so verify IPs. Our ai_bot_report.py (Python 3, standard library only, in the kit) counts hits per day, bot, status and path, and checks each IP against the operator’s published list for OpenAI, Anthropic, Perplexity, Google and Bing. A row with ip_verified=NO is a spoofed user agent. It flags a fake GPTBot from an unlisted IP.

   python3 ai_bot_report.py access.log > ai-bots.csv
   zcat access.log.*.gz | python3 ai_bot_report.py - > ai-bots.csv

   Without the script, use the double lookup: reverse DNS on the IP must return a Google hostname (googlebot.com, google.com or googleusercontent.com) or one ending in search.msn.com for Bingbot, and a forward lookup of that name must return the same IP.

   On Vercel, go to Firewall > Rules > Bot Management and set the AI Bots ruleset to Log, not Deny (WAF managed rulesets). Then use Firewall > Traffic and group by User Agents and Request Paths (observability). Runtime Logs show only static requests served from cache. A full log needs Drains, which are on Pro and Enterprise plans, with proxy.userAgent in each record:

   jq -r 'select(.proxy) | [(.proxy.timestamp/1000|todate), .proxy.clientIp, .proxy.path, (.proxy.statusCode|tostring), (.proxy.userAgent|join(" "))] | @tsv' drain.ndjson \
   | awk -F'\t' 'match($5, /OAI-SearchBot|ChatGPT-User|GPTBot|Claude-SearchBot|Claude-User|ClaudeBot|PerplexityBot|Perplexity-User|Googlebot|bingbot/) { print substr($5, RSTART, RLENGTH) "\t" $3 }' \
   | sort | uniq -c | sort -rn

   On Cloudflare, open AI Crawl Control. If AI crawler hits are near zero, check whether the managed robots.txt setting is prepending Disallow rules for GPTBot, ClaudeBot or Google-Extended.

8. Run a repeatable prompt test

   No official tool reports ChatGPT, Claude or Perplexity citations, so test by hand with a fixed protocol.

   1. Write 20 to 40 prompts across how-to, comparison, platform-specific and two or three brand questions. Give each a permanent ID. To change wording, retire the ID and add a new one.
   2. Engines: ChatGPT with search on, Google AI Mode, Google AI Overviews (a normal search), Gemini, Perplexity, Microsoft Copilot, Claude with web search on.
   3. Hold settings constant: country, language, signed in or out. In ChatGPT use Temporary chat set to Unpersonalized, which skips memory and custom instructions. Personalized can still use existing memories. Elsewhere, use a fresh profile or signed-out session and note the mode or model.
   4. Run each prompt 3 times per engine. Answers vary between runs.
   5. Cadence: a monthly baseline, plus one rerun 2 to 4 weeks after a major content change.
   6. Log every run in ai-prompt-tracker.csv (26 columns, two clearly labeled example rows). Compute mention rate, citation rate and citation share (your cited URLs divided by all cited URLs, the same idea as Bing’s metric).
   7. Keep screenshots as private evidence. Do not publish third-party interfaces.

   Mention rate   =COUNTIF(runs!N2:N1000,"yes")/COUNTA(runs!C2:C1000)
   Citation rate  =COUNTIF(runs!P2:P1000,"yes")/COUNTA(runs!C2:C1000)
   Citation share =SUM(runs!R2:R1000)/SUM(runs!S2:S1000)

9. Check it workedCheck it worked

   Confirm four things:

   • You have a before figure for the URL (total impressions and AI-feature impressions) and an annotation dated on the ship day.
   • The GA4 filter returns only assistant hosts.
   • The bot output shows your own requests. Make one with curl -A 'OAI-SearchBot' -s -o /dev/null https://yourdomain.com/robots.txt, then look for a row with that bot name and status 200.
   • The prompt tracker has a baseline row for every prompt ID and engine.