Measurement & Analytics

    How to Test AEO Changes: Experiments That Hold Up

    How to design AEO experiments that survive scrutiny: hypotheses, control groups, repeated prompt runs, recrawl checks and difference-in-differences.

    Saarthak Srivastav

    Saarthak Srivastav

    · 6 min read

    Key takeaways

    • Single before-and-after screenshots prove nothing in AEO, because AI answers vary from run to run and platforms change constantly.
    • Reliable tests compare a group of changed pages or prompts with a similar control group that wasn't changed.
    • Measure difference-in-differences: the change in the test group minus the change in the control group.
    • Confirm crawlers have actually fetched the changed pages before you start the measurement clock.
    • Don't split crawlers and users onto different versions of a URL; Google's testing guidance treats that as cloaking.

    To test an AEO change properly, change a group of pages, leave a similar group untouched as a control, run the same prompts repeatedly before and after, and compare how much each group moved. A single before-and-after check is noise: AI answers vary from run to run, platforms update their models and retrieval, and recrawling takes time. This post covers why AEO tests are hard, how to design one, how to calculate results, what to test first and a log template.

    Why AEO tests are hard

    Four things make naive tests unreliable, and a good design handles each one:

    Problem What it does How to handle it
    Answer variability The same prompt gives different answers on different runs Repeated runs and enough prompts
    Platform changes Model or retrieval updates shift everyone's results A control group measured over the same period
    Recrawl lag Assistants keep using the old version for a while Confirm crawler fetches before measuring
    Personalization and location Results differ by user and place Clean sessions, fixed location, tracking tools

    More on variability in why AI answers change every time.

    Step 1: Write a hypothesis you can falsify

    A testable hypothesis names the change, the pages, the metric and the expected direction. Use this format:

    If we [change] on [test group], then [metric] on [prompt group] will rise compared with [control group] within [time window].

    For example: "If we rewrite the first 100 words of 20 integration pages to answer the integration question directly, then citation rate for our pages on 60 integration prompts will rise compared with 20 unchanged integration pages within six weeks."

    One change per test. If you rewrite the intro, add schema and update the date at once, you won't know which one mattered.

    Step 2: Build matched test and control groups

    The control group is what makes a result believable. Split similar pages into two groups:

    • Same page type and template (all integration pages, all comparison pages)
    • Similar baseline performance, so one group isn't already stronger
    • Similar traffic and age
    • Each page mapped to its own prompts, so you know which prompts measure which page

    Alternate assignment after sorting by baseline performance (first page to test, second to control, and so on) to keep the groups balanced. With small sites, pair pages one-to-one.

    Don't split by user or bot. Classic A/B testing tools show different versions to different visitors, but for AEO the "visitor" that matters is a crawler, which sees one version. Google's guidance on website testing warns that showing Googlebot different content from users is cloaking; if you do run user-level tests with separate URLs, it advises rel="canonical" on variants, 302 rather than 301 redirects, and ending the test promptly. Group-based tests sidestep the issue entirely.

    Step 3: Measure a baseline

    Run the full prompt set for both groups for two to three weeks before the change. Record:

    • Mention rate and citation rate for your pages, per group
    • Runs per prompt (three or more)
    • Platforms and modes, held constant throughout

    If the groups differ a lot at baseline, rebalance before you start. See how to build a prompt set for structuring prompts by page and topic.

    Step 4: Ship the change and confirm recrawl

    The measurement clock starts when crawlers have fetched the changed pages, not when you hit publish. After shipping:

    • Update lastmod in the sitemap for changed pages only
    • Submit changed URLs through IndexNow and request indexing in Search Console where appropriate
    • Check server logs for fetches of each changed URL by the relevant crawlers (Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, Claude-SearchBot). See AI crawler log analysis.
    • Log the date most test pages were recrawled

    Make no other changes to test or control pages during the test window.

    Step 5: Calculate difference-in-differences

    Difference-in-differences subtracts the control group's change from the test group's change, removing the effect of anything that hit both groups. If a model update lifted everyone by five points, the control group absorbs it.

    Effect = (test after − test before) − (control after − control before)

    Then check the effect against the noise band. For a rate p measured over n answers, the margin of error at 95% confidence is about 1.96 × √(p × (1 − p) ÷ n). Since the effect combines four measurements, it's noisier than any single one; as a rule of thumb, treat effects smaller than the margin of a single group as inconclusive. For more on margins, see setting AEO targets.

    Example: testing answer-first intros

    This example is illustrative; the numbers are made up.

    A software company tests answer-first intros on integration pages. 20 test pages, 20 control pages, 60 prompts per group, three runs per prompt on three platforms (540 answers per group per period).

    Group Citation rate before Citation rate after Change
    Test (rewritten intros) 12% 21% +9 points
    Control (unchanged) 13% 15% +2 points
    Difference-in-differences +7 points

    At around 15% and 540 answers, a single group's margin is roughly ±3 points. A 7-point effect is well beyond it, so the team rolls the change out to all integration pages, then keeps measuring to confirm the effect holds at scale. If the effect had been 2 points, they'd have called it inconclusive rather than declared a win. The writing pattern itself is covered in the answer-first writing framework.

    What's worth testing

    Test changes you'd roll out widely if they work, since that's where a reliable answer pays off. Good candidates:

    • Answer-first openings on templated pages (integrations, use cases, locations)
    • Comparison tables added to category and comparison pages
    • FAQ sections with direct answers on product pages
    • Statistics, quotes and sourced claims, which the GEO research suggested can raise visibility. See what the GEO research found.
    • Structured data added to a template
    • Visible "last updated" dates with genuine content refreshes
    • llms.txt, a cheap change where evidence of impact is thin, so a test is useful. See llms.txt explained.

    Off-site changes, such as review campaigns or PR, can't be split into neat groups. For those, use a longer time series with competitor and unrelated-prompt controls, and be more cautious about claiming cause.

    Keeping a test log

    Keep every test, including the failures, in one log. A minimal record:

    Field Example
    Test ID and name T-014: answer-first integration intros
    Hypothesis As written in Step 1
    Test and control groups Page lists, with prompts mapped
    Metric Citation rate on mapped prompts
    Baseline window Two weeks before change
    Change shipped and recrawl confirmed Dates
    Measurement window Six weeks after recrawl
    Result Effect, margin, verdict (win, loss, inconclusive)
    Decision Roll out, revise or drop

    Over time, the log becomes your team's evidence base and stops the same debates from recurring.

    Pitfalls that invalidate tests

    • Comparing this month to last month with no control group
    • Changing control pages mid-test (often someone "fixes" them)
    • Starting the clock before recrawl
    • Testing too few prompts, then declaring a win on a two-point move
    • Stopping as soon as results look good
    • Running several overlapping tests on the same pages

    The bottom line

    AEO experiments hold up when they compare a changed group with a matched control, use the same prompts run repeatedly before and after, start measuring only once crawlers have fetched the changes, and judge results by difference-in-differences against the noise band. Test one change at a time, keep a log of every result, and roll out only what clears the bar.

    FAQ

    Frequently asked questions

    How do I know if an AEO change worked?

    Compare a group of changed pages with a similar group of unchanged pages, using the same prompts run repeatedly before and after. If the changed group improves by more than the control group and more than the noise band, the change likely worked.

    Can I A/B test pages for AI search?

    Not in the usual way. Crawlers see one version of a URL, so AEO tests usually change some pages and leave similar pages as controls. Google's guidance warns that showing crawlers different content from users is cloaking.

    How long should an AEO experiment run?

    Long enough for crawlers to fetch the changed pages and for several measurement runs afterward. For search-grounded assistants, that's often four to eight weeks; changes that depend on third-party coverage take longer.

    How many prompts do I need for an AEO test?

    Enough answers in each group that your margin of error is smaller than the effect you're looking for. In practice, dozens of prompts per group with several runs each, measured before and after.

    What AEO changes are worth testing first?

    Changes you'll roll out widely if they work: answer-first page openings, added comparison tables, FAQ sections, original data and statistics, and structured data. Test them on a subset before applying them sitewide.

    • experiments
    • testing
    • measurement
    • control groups
    • methodology
    Saarthak Srivastav

    Written by

    Saarthak Srivastav

    AEO & GEO, AEO-Rank

    Saarthak Srivastav writes and reviews articles on the AEO-Rank blog, drawing on what the AEO-Rank team sees tracking AI answers across ChatGPT, Gemini, Claude, Perplexity and Grok.