Your AI focus group already knew the answer

By Mo Touzani · 2026-11-12 · AI Conversion Optimization

Every growth team faces the same blank spreadsheet before launch: a new product, a new creative, five audience segments, and no history. You still have to split the budget. A paper posted to arXiv on November 6, 2026 offers a tempting fix. Choi, Lee, Seo, Lee and Choi asked an LLM to role-play each audience segment, decide whether it would click a display ad, and repeat that decision once for every real impression. The method predicted segment-level clicks with 51.3% less error than the best non-LLM baseline. If that holds, you could swap pre-launch testing budgets for API calls. Before you do, look at what sits under the headline number.

Key takeaways

On one Google Ads display campaign (974,423 impressions, 3,578 clicks, five segments, seven days), LLM persona simulation reached a mean absolute percentage error of 26.1% on clicks.

Bare demographic personas beat rich ones, and a relative qualitative cue beat numeric CTR anchors. An LLM given an explicit CTR prior produced 77.3% more error than the final configuration.

The winning cue says younger adults ignore ads more than older adults. In the evaluation data, the two older segments clicked at 0.61% to 0.68% and the three younger ones at 0.17% to 0.27%. The cue encodes the outcome's ranking.

The method predicts clicks only. It has no conversion, revenue or incremental-lift target, so it cannot tell you where to put a launch budget.

What the authors built

The team took a real Google Ads display campaign for a jelly product, targeted at five demographic segments from April 5 to 11, 2026. For each segment they prompted gpt-5-nano with a persona, the ad creative and one question: would this user click? The model answered yes or no at temperature 1.0. They averaged the answers into a CTR and multiplied by planned impressions.

The contribution is a set of ablations on what they call signal form. Thin personas beat rich ones: the targeting fields Google Ads exposes (age, gender, parental status, income, interests) produced lower error than personas with personality and occupation added. Qualitative cues beat numbers: telling the model that ad CTR is typically 0.5% made results worse, while a relative statement about banner blindness by age worked best. Wording counted too. "Tend to show" and "exhibit" carry the same meaning, yet the authors measured different accuracy for each.

The cue that gave away the answer

Look at the observed CTRs. Segments A and B, parents aged 45 to 64, clicked at 0.680% and 0.607%. The three younger segments clicked at 0.174%, 0.200% and 0.266%. Now reread the winning cue: younger adults ignore ads more than older adults. That cue states the ranking of the outcome the model is graded on.

The authors chose among cues and phrasings on the same campaign they report. They had no held-out campaign and no second product. They flag transfer as a limitation, to their credit. When the experimenter writes the outcome's ordering into the prompt, a win shows you instruction-following. A forecast needs a campaign the authors never touched.

If a simulator needs the answer's shape in its prompt, treat its output as a prior.

The baselines set a low bar

The 51.3% improvement comes from comparing against two simple heuristics. One multiplies impressions by a flat benchmark CTR. The other applies an age curve borrowed from research-paper recommendation clicks, which the authors call a cross-domain heuristic. Any experienced media buyer beats both on day one, because buyers already expect older display audiences to click more. The LLM matches a rule of thumb you already hold, at the cost of nearly a million inference calls.

CTR is the wrong target for capital allocation

Suppose the method generalized perfectly. You would still be forecasting clicks, the cheapest event in the funnel and the one bots and accidental taps inflate most. Funding the segment with the highest predicted CTR bets that clickers become buyers at the same rate in every segment. Pull conversion rate by segment from your last launch and check that bet. A pre-launch forecast earns its keep when it predicts cost per incremental acquisition, and a 26% error on clicks compounds through every funnel step you leave unmodeled.

What breaks at scale

Picture a retailer launching 300 SKUs a quarter. The method needs a hand-picked qualitative cue per campaign, and someone has to decide which direction it points. That person needs prior data, and having no prior data is the premise of cold start. Picture a seed-stage brand with $20,000 to spend. A forecast with 26% error leaves the split near an educated guess, while a $2,000 two-day split test buys real signal for the same decision.

Variance goes unstudied. The authors sample at temperature 1.0 from one small model, so a model version change shifts the simulated population with it. The paper reports no seed stability, no cross-model comparison and no per-segment confidence intervals.

What to take from it

1. Strip your synthetic personas. Use the targeting fields you can buy and stop there. The paper gives you evidence that backstory adds noise.
2. Never hand a simulator the number you want it to estimate. It repeats your anchor with false confidence. Phrase priors as relative statements and test them on held-out data.
3. Version-control your prompts. If changing "tend to" moves accuracy, your prompt is a model parameter. Log it, diff it and test it.
4. Rank creative, never split spend. Use persona simulation to screen concepts and catch misfit segments, then run a short split test on conversions and let real data overwrite the simulation.

If a vendor sells you synthetic audiences in place of experiments, ask for results on campaigns they never tuned on. This paper gives you the reason to ask.

For a test design that measures purchases instead of clicks, read why most A/B tests don't change anything.

To see where your own measurement stack stands, run the Growth Readiness Diagnostic.

Source: Choi, J., Lee, I. I., Seo, W. S., Lee, M. and Choi, Y. Y. (2026). What Matters for LLM Persona Simulation: Signal Form in Cold-Start E-Commerce Advertising. GenAIECommerce'26 workshop, RecSys 2026. arXiv:2611.08840.