
Quick answer
Agent Experience Optimization (GXO) is the practice of making a website usable by AI agents that browse, compare, and transact on behalf of human users. Agents do not scan layouts or respond to persuasion. They parse structure, extract facts, and execute steps. Optimizing for them means exposing accurate machine-readable data, keeping critical information in text rather than images or scripts, building predictable navigation and clean flows, and deciding deliberately what automated access you permit.

A new category of visitor has started arriving at websites. It does not see your hero image, is unmoved by your social proof, ignores your newsletter modal, and will not be persuaded by a well-written value proposition. It reads your markup, extracts the facts it was sent to find, and either completes a task or reports back that your site made the task impossible.
These are AI agents, and they are operating on behalf of people. A user asks an assistant to find a project management tool under thirty dollars per seat that integrates with their existing stack, or to book a table for four on Friday, or to compare three suppliers and summarize the differences. The agent visits sites, reads them, and reports or acts.
GXO is the discipline of being usable by that visitor. It is early, the standards are unsettled, and much of the commercial hype around it is premature. But the underlying requirement is not speculative: as more decisions are mediated by software reading your site, sites that are readable by software win those decisions and sites that are not are simply skipped.
This guide covers how agents actually read websites, what they need that human visitors do not, how to expose machine-readable data properly, how to build architecture and flows an agent can traverse, what agentic commerce readiness involves, how to decide your access policy, and how to measure agent traffic that mostly does not identify itself.
In this guide
- What Agents Are and How They Differ From Crawlers
- Why This Matters Now
- The Agent Landscape
- Foundation One: Expose Machine-Readable Data
- Foundation Two: Keep Critical Facts in Text
- Foundation Three: Build Traversable Architecture
- Foundation Four: Agentic Commerce Readiness
- A Worked Example of Agent Task Failure
- Deciding Your Access Policy
- A GXO Audit Checklist
- Measuring Agent Activity
- Sizing This Correctly
- Frequently asked questions
What Agents Are and How They Differ From Crawlers
The distinction from a search crawler matters, because the optimization is different.
Crawlers Index, Agents Act
A search crawler visits broadly and shallowly, building an index for later retrieval. It does not have a goal beyond coverage. It does not fill forms, click through flows, or make decisions.
An agent visits narrowly and deeply with a specific task. It may visit three pages on your site and nothing else. It may need to extract a price, compare a specification, check availability, or complete a multi-step process. Its visit succeeds or fails on whether it can accomplish that task.
This changes what matters. For a crawler, broad indexability is the objective. For an agent, task completability is the objective, and a site can be perfectly indexed and completely unusable.
What Agents Actually Read
Current agents work through one of three mechanisms, and increasingly through combinations.
Text and markup extraction reads the rendered HTML, pulling text content, structured data, link relationships, and form definitions. This is the most common approach and the one most affected by how you build pages.
Visual interpretation renders the page and processes it as an image, identifying elements the way a human would. This handles visually-presented information that markup omits, but it is slower, more expensive, and more error-prone.
API and feed consumption uses structured endpoints where available, which is by far the most reliable path when one exists.
The practical implication is that information available in text and markup is read reliably, information available only visually is read unreliably, and information available only through interaction may not be read at all.
What Agents Cannot Do Well
Agents struggle with content that requires JavaScript interaction to reveal, information encoded in images without text alternatives, flows that depend on hover or drag, CAPTCHA and bot challenges, authentication without a programmatic path, and layouts where the meaningful relationship between elements is conveyed only by visual proximity.
Each of these is a point where an agent task fails. For a site whose goal is being chosen by agent-mediated decisions, each is a leak.
Why This Matters Now
A reasonable response is that agent traffic is currently a rounding error. For most sites that is true, and GXO should be sized accordingly. Three things make it worth attention anyway.
The Work Overlaps Almost Entirely With Existing Priorities
Everything GXO asks for is something you should be doing regardless. Accurate structured data helps search engines. Text-based information rather than image-based helps accessibility and AI retrieval. Clean semantic HTML helps crawlers. Predictable navigation helps users. Fast server responses help Core Web Vitals.
There is very little GXO-specific work that does not pay for itself through other channels. This makes the investment low-risk even if agent traffic grows slower than predicted.
Decisions Are Already Being Mediated
Even where agents are not transacting, they are increasingly summarizing and comparing. When an assistant tells a user that three tools meet their requirements and yours is not among them because the agent could not determine your pricing, you lost that decision without ever appearing in it.
This is happening now, at modest volume, in categories where people research before buying. The volume is growing.
Accessibility and Agent-Readability Converge
The requirements for a screen reader and the requirements for an agent are substantially the same: semantic markup, text alternatives for images, properly labeled form fields, logical heading structure, and meaningful link text.
A site built accessibly is largely agent-ready as a side effect, which means GXO work can be justified on accessibility grounds alone in organizations where agent traffic is not yet a persuasive argument.
The Agent Landscape
Several distinct categories of automated visitor are often conflated, and the right response differs for each.
Search Crawlers
Googlebot, Bingbot, and their equivalents index for search. They have visited your site for years, you already optimize for them, and nothing about GXO changes that relationship.
Retrieval Crawlers
These fetch content at query time to answer a specific user question, which is the retrieval layer behind AI Overviews, Perplexity, and search-enabled assistants. They behave much like search crawlers but with narrower, query-driven targeting. Optimizing for them is largely GEO work: clean extractable passages and clearly stated facts.
Training Crawlers
These collect content for model training rather than for answering a present query. They provide no referral traffic and no attribution. This is the category most publishers want to control, and it is controllable through specific robots.txt directives. Examples include GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl) and the Google-Extended token, which controls use of your content for Gemini training without affecting Google Search.
User-Directed Task Agents
These act on behalf of a specific person with a specific task: compare these options, find this information, book this thing. They are the genuine subject of GXO, because their visit has an outcome that can succeed or fail and that outcome affects whether you get the customer.
Examples include ChatGPT agent and the ChatGPT Atlas browser from OpenAI, Perplexity’s Comet browser, and Google’s agentic features in Gemini and AI Mode. Some identify themselves with user agents such as ChatGPT-User, but OpenAI’s own crawler documentation notes that because these fetches are initiated by a user, robots.txt rules may not apply to them. Browser-based agents often look like ordinary Chrome traffic.
The volume here is currently modest but it is the category where failure is most costly, because a failed agent task is a lost decision rather than a lost impression.
Scrapers and Abusive Bots
Content theft, price scraping, credential stuffing, and inventory hoarding. These are a security problem rather than an optimization question, and the challenge is blocking them by behavior without blocking legitimate agents by category.
Why the Distinction Matters
A blanket block aimed at training crawlers frequently catches user-directed agents too, because the identifiers overlap and the enforcement is often behavioral rather than name-based. Sites that intended to protect their content from training discover they also excluded themselves from the comparison that would have sent them a customer.
Getting this right requires treating the five categories separately in both robots.txt and your security layer, which most sites have never done.
Foundation One: Expose Machine-Readable Data
Structured data is the primary channel through which a site states facts unambiguously. For agents it is substantially more important than for search engines, because an agent needs the fact rather than a ranking.
Implement the Types That Describe Your Entities
Organization schema should state who you are, with your name, URL, logo, description, contact details, and sameAs links to your authoritative profiles.
Product schema should state name, description, SKU or identifier, brand, price with currency, availability, and condition. For an agent comparing products, these fields are the comparison.
Service schema should state what you offer, the area served, and pricing structure where applicable.
LocalBusiness schema should state address, phone, opening hours per day, geographic coordinates, and accepted payment methods.
Article schema should state headline, author as a linked Person entity, publication date, and modification date.
FAQPage and HowTo schema should mark up genuine question-answer sets and genuine procedures. Google no longer shows HowTo rich results and limits FAQ rich results to well-known government and health sites, but the markup is still a clean, machine-readable statement that agents can use to extract answers and steps.
Keep Structured Data Truthful and Current
Schema that contradicts your visible content is worse than no schema, because an agent acting on a stale price or incorrect availability produces a failed transaction and a user who distrusts your site.
The discipline required is that structured data be generated from the same source as the displayed content rather than maintained separately. Hand-maintained schema drifts out of date within months. Where your platform generates it automatically from product or page data, verify the mapping is correct and leave it alone.
Validate and Monitor
Run your key page types through the Rich Results Test and the Schema Markup Validator after any template change. Monitor the Enhancements reports in Search Console for errors appearing over time, which usually indicate a template or data problem affecting many pages at once.
Consider Feeds and Endpoints
Where you have substantial structured content, an accessible feed or API is the most reliable path for an agent. A product catalog feed, an availability endpoint, or a simple JSON representation of your core data removes all ambiguity.
This is disproportionately valuable for commerce, booking, and listing sites, and largely unnecessary for content publishers. Size the effort to the category.
Foundation Two: Keep Critical Facts in Text
The single most common GXO failure is information that exists only in a form an agent cannot read.
The Common Offenders
Prices rendered inside images. Specifications presented in an image-based table. Opening hours in a graphic. Contact details in a logo image. Shipping and return terms inside a PDF. Pricing tiers loaded by a script after interaction. Availability shown only as a color state with no text label.
Each of these is readable by a human and invisible or unreliable to a text-extracting agent. The fix in every case is the same: ensure the fact exists as plain text in the rendered HTML, whatever else you also do visually.
Audit by Reading Your Own Markup
The practical test is to view your page as text. Disable CSS and images, or use a text-only rendering, and check whether the facts a buyer would need are present and comprehensible. Alternatively, fetch the page with curl and read the raw HTML.
Anything absent from that output is absent for a substantial share of agents. This audit takes minutes per template and reliably finds problems on most commercial sites.
Handle JavaScript Rendering Carefully
Content injected by client-side JavaScript may or may not be seen depending on whether the agent renders. Search crawlers generally render; many agents do not, or do so inconsistently.
Where critical facts depend on client-side rendering, consider server-side rendering or static generation for those elements. At minimum, ensure the facts are also present in structured data, which is parsed from the initial HTML.
Foundation Three: Build Traversable Architecture
An agent navigating your site needs to find things predictably without exploring.
Use Semantic HTML
Proper heading hierarchy, nav and main and article elements, real lists for lists, real tables for tabular data, and labeled form fields. Divs styled to look like buttons and headings are invisible as buttons and headings.
This is unglamorous work that most sites have partially wrong, and it is the single highest-return GXO investment because every mechanism benefits from it.
Keep URLs Logical and Stable
Readable URLs that reflect site hierarchy let an agent infer structure and construct paths. Stable URLs mean an agent returning to complete a task finds the page where it was. Gratuitous URL changes break agent workflows more severely than they break human ones, because the agent has no judgment to recover with.
Maintain Complete Sitemaps and Clean Robots Rules
An accurate XML sitemap is the most efficient discovery mechanism available. Keep it complete, current, and free of non-canonical URLs. Keep robots.txt free of accidental blocks on resources needed for rendering, which is a common and silent problem.
Make Navigation Explicit
Breadcrumbs with markup, descriptive link text, and visible navigation links rather than script-driven menus all help an agent understand where it is and where it can go. Link text reading click here or learn more conveys nothing about the destination.
Respond Quickly and Consistently
Agents frequently operate under time constraints and may abandon slow responses. They also make more requests per task than a human visit. Server response time, consistent availability, and absence of aggressive rate limiting all affect whether tasks complete.
This creates a genuine tension with bot management, addressed below.
Foundation Four: Agentic Commerce Readiness
For sites that sell or book, the agent may need to complete a transaction rather than merely read. This is the most demanding category and the least settled.
Expose the Decision Facts Completely
An agent evaluating a purchase needs the full price including shipping and tax where determinable, availability and stock status, delivery timeframe, specifications in comparable units, return terms, and any eligibility constraints.
Information revealed only at checkout cannot be used in a comparison that happens before checkout. A product that appears cheapest until shipping is added loses the comparison it should have won, and the agent reports your competitor.
Build Flows an Agent Can Traverse
Checkout and booking flows should work without hover interactions, complete in a predictable number of steps with stable element identifiers, avoid mandatory account creation before purchase, label every form field properly, and surface errors as readable text rather than only as color changes.
Guest checkout matters disproportionately here, because an agent may have no credential to create or use an account with.
Think About Authentication and Authorization
The hardest problem in agentic commerce is how an agent proves it acts legitimately on a user’s behalf, and how a merchant knows what the user actually authorized. Several competing standards now exist. OpenAI and Stripe launched the Agentic Commerce Protocol in September 2025 to power Instant Checkout in ChatGPT. Google announced the Agent Payments Protocol (AP2) the same month and the Universal Commerce Protocol, built with Shopify, Etsy, Wayfair, Target and Walmart, in January 2026. Visa (Trusted Agent Protocol) and Mastercard (Agent Pay) are building the card-network layer, and Cloudflare’s Web Bot Auth lets agents sign their requests so sites can verify them.
None of these has won yet. For most sites the correct position today is to adopt them through your commerce or payment platform when it supports them, rather than building custom integrations.
What is worth doing now is ensuring that your flows do not break for a logged-out visitor who has all the necessary information, since that is the state most agents operate in.
Reconsider Blanket Bot Blocking
Aggressive bot detection that blocks anything non-human will block legitimate agents acting for real customers. CAPTCHA at checkout, in particular, terminates agent transactions entirely.
The resolution is selective rather than blanket: block abusive patterns by behavior rather than by category, allow known legitimate agent identifiers where you can verify them, and avoid interaction challenges on paths you want completed. This requires working with whoever manages your security layer and is frequently where GXO work actually stalls in organizations.
A Worked Example of Agent Task Failure
The abstract advice becomes concrete through a specific failure, which is also the most common one.
A user asks an assistant to find project management software under thirty dollars per user per month with time tracking and a Slack integration. The assistant identifies six candidate vendors and visits each.
On the first site, pricing is published as plain text in a table with per-seat figures, the feature list names time tracking explicitly, and the integrations page lists Slack. The agent extracts all three facts in one visit and the vendor qualifies.
On the second site, pricing is behind a contact sales form. The agent cannot determine the price, so it cannot evaluate the constraint, and it reports the vendor as pricing unavailable. From the user’s perspective this vendor was not recommended. It may well have been the best fit and cheapest option.
On the third site, pricing exists but is rendered inside a styled image for design reasons. The agent reads the page text and finds no price. Same outcome as the second site, except the vendor believed its pricing was public.
On the fourth site, pricing is in text but loads via JavaScript after the page renders, triggered by a plan-selector toggle. An agent that renders and interacts may find it. One that reads initial HTML will not. The outcome is inconsistent, which is arguably worse than consistent failure because it is harder to detect.
On the fifth site, pricing is readable but the feature list is a marketing page of benefit statements rather than capability names. Time tracking exists as a feature but the page says things like stay on top of your team’s hours. The agent cannot confirm the capability and excludes the vendor on uncertainty.
On the sixth site, everything is readable, but a bot challenge fires on the pricing page because the security layer flags non-browser user agents. The agent is blocked entirely.
Five of six vendors failed, and in four of the five cases the vendor had no idea. None of these are content quality problems or ranking problems. They are machine-readability problems, and each has a specific fix: publish pricing in text, do not put facts in images, render critical facts server-side, name capabilities literally alongside benefit language, and exempt informational pages from bot challenges.
Deciding Your Access Policy
Permitting automated access is a genuine business decision with costs on both sides, and it deserves an explicit answer rather than an inherited default.
The Case for Permitting
Agents acting for users represent demand. Blocking them does not redirect that demand to you, it redirects it to a competitor the agent could read. For commerce, booking, service, and lead-generation businesses, being unreadable means being excluded from consideration.
The Case for Restricting
For publishers whose revenue depends on human page views and advertising impressions, automated extraction consumes content without generating revenue. Training crawlers in particular take content for model development with no reciprocal traffic.
Infrastructure cost is also real at scale, since agents make more requests per outcome than human visitors.
A Workable Middle Position
The distinction worth drawing is between training crawlers and user-directed agents. A training crawler takes content for model development. A user-directed agent is acting for a specific person with a specific task, and serving it may produce a customer.
Many sites reasonably restrict the former while permitting the latter. Implement this through specific robots.txt directives per user agent rather than blanket rules, keep a documented record of which you permit and why, and revisit it as the landscape changes.
Whatever you decide, verify your robots.txt and your security layer actually implement the decision. The most common situation is a site whose stated policy and actual behavior differ because nobody checked.
A GXO Audit Checklist
The following sequence covers the practical work for most sites and can be completed in a day.
Markup and Data
Confirm Organization schema exists on the homepage with sameAs links. Confirm the schema type appropriate to your core pages is implemented and validates without errors. Verify that structured data values match what is visibly displayed, particularly prices and availability. Check the Enhancements reports in Search Console for accumulating errors.
Fact Exposure
Fetch each key template with curl and read the raw HTML. Confirm that price, availability, specifications, hours, location, contact details, and terms are present as text. List anything missing and move it out of images, PDFs, or script-dependent rendering.
Structure
Confirm one H1 per page and a logical heading hierarchy without skipped levels. Confirm navigation uses real links rather than script handlers. Confirm forms have labeled fields. Confirm link text describes destinations. Confirm tables are real tables.
Access
Read your robots.txt and list every directive by user agent. Confirm it implements a policy you actually chose rather than a default. Confirm no rule blocks resources required for rendering. Check whether your security layer or CDN applies rules your robots.txt does not reflect. Since July 2025, for example, Cloudflare blocks known AI crawlers by default on new domains, so a site can be blocking agents without anyone having written a rule.
Flows
Attempt your primary conversion path with JavaScript interaction minimized. Confirm guest checkout or guest enquiry works. Confirm no bot challenge fires on informational or pricing pages. Confirm error messages appear as readable text.
Verification
Ask two or three assistants with browsing enabled to accomplish your site’s core tasks. Record each failure, trace it to a specific defect, and fix it. Repeat quarterly, because template changes reintroduce problems silently.
Measuring Agent Activity
Measurement is poor because many agents do not identify themselves and analytics tools are built around human sessions.
Read Server Logs, Not Analytics
JavaScript-based analytics misses most agent traffic because agents frequently do not execute analytics scripts. Server logs capture every request regardless.
Review logs for known agent user agents, for request patterns inconsistent with human browsing such as very fast sequential page fetches or repeated structured-data requests, and for traffic to your feeds and sitemap files.
Track Referrals From Assistant Platforms
Where an agent reports back to a user who then visits, the visit may carry a referrer from an assistant domain. Monitor these in analytics as a proxy for agent-mediated discovery, and segment their behavior separately since they often convert differently.
Test Task Completion Directly
The most informative measurement available is attempting the tasks yourself. Ask several assistants with browsing capability to accomplish things on your site: find your pricing, identify whether a product meets a specification, determine your opening hours, begin a booking.
Record what succeeds and what fails. Each failure is a specific, fixable defect, and this test typically surfaces more actionable problems than any amount of log analysis. Run it quarterly.
Watch for Failure Signals
Rising blocked-request volume in your security logs, bot challenges firing on commercial paths, and structured data errors accumulating in Search Console are all signals that agent tasks are failing on your site.
Sizing This Correctly
A closing note on proportion, because the commercial noise around this subject substantially exceeds its current importance.
For a content publisher, GXO is a modest set of hygiene tasks: accurate schema, facts in text, semantic markup, a deliberate crawler policy. A few hours of work, justified mainly because it also serves search and AI retrieval.
For a commerce, booking, or service business, it is more consequential, because agent-mediated comparison is already affecting purchase decisions in some categories. The pricing and specification exposure work and the checkout traversability work both deserve real attention.
For anyone, the authentication and agent-payment layer is not yet worth building custom code for, because the competing standards are still settling. Let your commerce and payment platforms carry that work, and switch it on when they support it.
The honest summary is that GXO today is mostly good engineering practice with a new justification. The sites that will be ready when agent traffic becomes material are the ones that built accessible, well-marked-up, fast, logically-structured sites for ordinary reasons. That has been good advice for twenty years and it remains the answer.
Key takeaways
- GXO makes websites usable by AI agents acting for people.
- Agents parse structure and data rather than design or persuasion.
- Expose machine-readable data and keep key facts in plain text.
- Predictable navigation and simple checkout flows help agents complete tasks.
- Decide your agent access policy and monitor agent activity in your logs.
Frequently Asked Questions
What is GXO?
GXO is the practice of optimizing a website so AI agents can understand it, navigate it and complete tasks such as comparing products or booking services on behalf of users.
How do AI agents differ from search crawlers?
Crawlers collect pages for an index. Agents act in real time for a specific user goal, reading pages, filling forms and making decisions.
What is agentic commerce?
Agentic commerce is shopping in which AI agents research, compare and sometimes complete purchases for a person, relying on clear product data, prices and policies.
Should I block AI agents?
It depends on your business model. Many sites benefit from allowing agents that bring customers, while restricting scrapers that offer nothing in return.
What is the first step in a GXO audit?
Check whether your key facts, such as prices, availability, specifications and policies, are present in plain text and structured data rather than only in images or scripts.
Related guides
Go deeper on Search Experience
Researched from Google documentation, published studies and hands-on experience, then reviewed for accuracy before publishing. Learn how we work in our editorial policy.
