To test an AEO change properly, change a group of pages, leave a similar group untouched as a control, run the same prompts repeatedly before and after, and compare how much each group moved. A single before-and-after check is noise: AI answers vary from run to run, platforms update their models and retrieval, and recrawling takes time. This post covers why AEO tests are hard, how to design one, how to calculate results, what to test first and a log template.
Why AEO tests are hard
Four things make naive tests unreliable, and a good design handles each one:
| Problem | What it does | How to handle it |
|---|---|---|
| Answer variability | The same prompt gives different answers on different runs | Repeated runs and enough prompts |
| Platform changes | Model or retrieval updates shift everyone's results | A control group measured over the same period |
| Recrawl lag | Assistants keep using the old version for a while | Confirm crawler fetches before measuring |
| Personalization and location | Results differ by user and place | Clean sessions, fixed location, tracking tools |
More on variability in why AI answers change every time.
Step 1: Write a hypothesis you can falsify
A testable hypothesis names the change, the pages, the metric and the expected direction. Use this format:
If we [change] on [test group], then [metric] on [prompt group] will rise compared with [control group] within [time window].
For example: "If we rewrite the first 100 words of 20 integration pages to answer the integration question directly, then citation rate for our pages on 60 integration prompts will rise compared with 20 unchanged integration pages within six weeks."
One change per test. If you rewrite the intro, add schema and update the date at once, you won't know which one mattered.
Step 2: Build matched test and control groups
The control group is what makes a result believable. Split similar pages into two groups:
- Same page type and template (all integration pages, all comparison pages)
- Similar baseline performance, so one group isn't already stronger
- Similar traffic and age
- Each page mapped to its own prompts, so you know which prompts measure which page
Alternate assignment after sorting by baseline performance (first page to test, second to control, and so on) to keep the groups balanced. With small sites, pair pages one-to-one.
Don't split by user or bot. Classic A/B testing tools show different versions to different visitors, but for AEO the "visitor" that matters is a crawler, which sees one version. Google's guidance on website testing warns that showing Googlebot different content from users is cloaking; if you do run user-level tests with separate URLs, it advises rel="canonical" on variants, 302 rather than 301 redirects, and ending the test promptly. Group-based tests sidestep the issue entirely.
Step 3: Measure a baseline
Run the full prompt set for both groups for two to three weeks before the change. Record:
- Mention rate and citation rate for your pages, per group
- Runs per prompt (three or more)
- Platforms and modes, held constant throughout
If the groups differ a lot at baseline, rebalance before you start. See how to build a prompt set for structuring prompts by page and topic.
Step 4: Ship the change and confirm recrawl
The measurement clock starts when crawlers have fetched the changed pages, not when you hit publish. After shipping:
- Update
lastmodin the sitemap for changed pages only - Submit changed URLs through IndexNow and request indexing in Search Console where appropriate
- Check server logs for fetches of each changed URL by the relevant crawlers (Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, Claude-SearchBot). See AI crawler log analysis.
- Log the date most test pages were recrawled
Make no other changes to test or control pages during the test window.
Step 5: Calculate difference-in-differences
Difference-in-differences subtracts the control group's change from the test group's change, removing the effect of anything that hit both groups. If a model update lifted everyone by five points, the control group absorbs it.
Effect = (test after − test before) − (control after − control before)
Then check the effect against the noise band. For a rate p measured over n answers, the margin of error at 95% confidence is about 1.96 × √(p × (1 − p) ÷ n). Since the effect combines four measurements, it's noisier than any single one; as a rule of thumb, treat effects smaller than the margin of a single group as inconclusive. For more on margins, see setting AEO targets.
Example: testing answer-first intros
This example is illustrative; the numbers are made up.
A software company tests answer-first intros on integration pages. 20 test pages, 20 control pages, 60 prompts per group, three runs per prompt on three platforms (540 answers per group per period).
| Group | Citation rate before | Citation rate after | Change |
|---|---|---|---|
| Test (rewritten intros) | 12% | 21% | +9 points |
| Control (unchanged) | 13% | 15% | +2 points |
| Difference-in-differences | +7 points |
At around 15% and 540 answers, a single group's margin is roughly ±3 points. A 7-point effect is well beyond it, so the team rolls the change out to all integration pages, then keeps measuring to confirm the effect holds at scale. If the effect had been 2 points, they'd have called it inconclusive rather than declared a win. The writing pattern itself is covered in the answer-first writing framework.
What's worth testing
Test changes you'd roll out widely if they work, since that's where a reliable answer pays off. Good candidates:
- Answer-first openings on templated pages (integrations, use cases, locations)
- Comparison tables added to category and comparison pages
- FAQ sections with direct answers on product pages
- Statistics, quotes and sourced claims, which the GEO research suggested can raise visibility. See what the GEO research found.
- Structured data added to a template
- Visible "last updated" dates with genuine content refreshes
- llms.txt, a cheap change where evidence of impact is thin, so a test is useful. See llms.txt explained.
Off-site changes, such as review campaigns or PR, can't be split into neat groups. For those, use a longer time series with competitor and unrelated-prompt controls, and be more cautious about claiming cause.
Keeping a test log
Keep every test, including the failures, in one log. A minimal record:
| Field | Example |
|---|---|
| Test ID and name | T-014: answer-first integration intros |
| Hypothesis | As written in Step 1 |
| Test and control groups | Page lists, with prompts mapped |
| Metric | Citation rate on mapped prompts |
| Baseline window | Two weeks before change |
| Change shipped and recrawl confirmed | Dates |
| Measurement window | Six weeks after recrawl |
| Result | Effect, margin, verdict (win, loss, inconclusive) |
| Decision | Roll out, revise or drop |
Over time, the log becomes your team's evidence base and stops the same debates from recurring.
Pitfalls that invalidate tests
- Comparing this month to last month with no control group
- Changing control pages mid-test (often someone "fixes" them)
- Starting the clock before recrawl
- Testing too few prompts, then declaring a win on a two-point move
- Stopping as soon as results look good
- Running several overlapping tests on the same pages
The bottom line
AEO experiments hold up when they compare a changed group with a matched control, use the same prompts run repeatedly before and after, start measuring only once crawlers have fetched the changes, and judge results by difference-in-differences against the noise band. Test one change at a time, keep a log of every result, and roll out only what clears the bar.
