Your best-fitting model is grading the wrong exam

By Mo Touzani · 2026-11-06 · Marketing Measurement

A marketing mix model with an R² of 0.93 overstated the return on four channels by 172% to 241%. A rival model with a lower R² of 0.90 came closer on all four. That example opens a paper two Measured Inc. researchers posted to arXiv on November 4 (Chen and Lim, 2026), and it frames a week of research that anyone who signs off on a growth budget should read. Four papers from Measured, Meta and Mississippi State University find the same fault in four places. Your team tunes a number that sits one step away from the decision you pay it to make.

Key takeaways

Selecting a marketing mix model (MMM) by out-of-sample R² or MAPE picked the model with the most accurate channel returns in 17.5% of 320 simulated datasets. A random pick among the four candidates would land at 25%.

A lift experiment with 25% noise, used as a benchmark, matched or beat fit-based selection in 90% of datasets. A benchmark cannot rescue a shortlist of models that all leave out carryover effects.

The largest share of requests to Meta's AI experimentation agent (45%) asked it to fetch results, and some users talked it into p-hacking.

Meta's per-click landing destination model lifted advertiser value by 0.54%. At mid-market traffic, you cannot detect a lift that size.

Standard 95% confidence intervals held up at 50/50 splits in three real datasets and slipped to 92.6% at an 80/20 split, in the one dataset where a single person carried about a third of the variance.

Model fit tells you almost nothing about channel returns

Chen and Lim generated 320 datasets with known channel returns, fit four candidate MMMs to each, and checked which candidate a fit-based rule would choose. Out-of-sample R² and MAPE chose the model with the most accurate return on ad spend (RoAS) 17.5% of the time, with a 95% confidence interval of 13.7% to 22.0% (arXiv 2611.06111). The rank correlation between fit and RoAS accuracy came out at a Kendall τ of minus 0.02, which is indistinguishable from zero. On average, the model with the best fit carried 1.47 times the RoAS error of the best candidate. Its channel errors averaged 41% of mean RoAS, against 29% for the best model.

R² rewards a model for predicting total sales, while your budget decision depends on how the model splits those sales across channels. Two models can predict the total with equal skill and hand out credit in opposite directions. Correlated spend makes it worse: as the correlation between channel budgets rose from 0 to 0.9 in the simulations, the hit rate of fit-based selection fell from 33% to 20%.

The authors propose a different referee. Run lift experiments, then pick the MMM whose channel RoAS sits closest to the experimental readings. With benchmark noise of 10%, 25% and 50% (coefficient of variation), RoAS error fell by 23%, 19% and 14% against R²-based selection. At 25% noise, the benchmark choice matched or beat the fit choice in 90% of datasets.

Their real-world case, a consumer packaged goods brand with 156 weeks of data across paid search, social and display, shows the stakes. Model 1 (a Hill saturation curve) scored an R² of 0.782. Model 3 (a Weibull curve) scored 0.780. On fit, you would call it a tie. Against the experiments, Model 3 cut the RoAS error from 0.255 to 0.159, a 1.6 times improvement. Model 1 overstated display by 15.1%. A team that moved budget into display on Model 1's advice would have paid for that 15% gap.

The blind spots sit where your budget lives

Before you rebuild your measurement plan around this paper, weigh three facts about it.

Both authors list an affiliation with a company that sells the remedy. Measured markets incrementality experiments and a causal media mix model (Business Wire, 2025). You can check the simulations yourself, and the math stands. The conflict shows up in what the paper leaves unpriced. A geo holdout means switching off media in test regions for weeks and giving up the revenue those regions would have produced. The paper assumes benchmarks exist for every channel and lists partial coverage as untested. Your finance team will ask how many experiments the method needs before it pays for itself, and the paper has no answer.

The paper grades average RoAS, and your reallocation decision runs on marginal RoAS. Average return tells you whether a channel paid back last year. Marginal return tells you whether the next $100,000 will. A model can match the experiment on the average and still get the saturation curve wrong, and the curve is where you decide to stop spending. The authors flag this limitation, and you should expect vendors who cite the paper to leave it out.

Experiments pick the least wrong model from your shortlist and leave a bad shortlist bad. In simulations where every candidate omitted adstock (the carryover of advertising into later weeks), a benchmark with 50% noise performed 9% worse than R² selection. Model 3 in the case study still overstated display by 12.7%. The paper also treats each experiment as a point estimate with no interval, even though a geo test on a mid-size budget can carry a wide one.

The high-signal finding survives these objections: stop choosing MMMs on fit statistics. The weaker claim, that any experiment improves any model, holds when your candidate models include the right structure and fails when they leave it out.

An AI analyst makes p-hacking cheap

Meta's experimentation team published a field report on its ABTest Agent (arXiv 2611.04515), accepted to the CODE@MIT 2026 conference. The design rule: the language model chooses which procedures to run, and deterministic code computes every number in the output. A report gate refuses to publish any report with an empty section or a table that does not trace back to a validated analysis tool. Of the agent's 97 skills, 52 exist to establish or restore trust in a result.

The team improved answers by removing documentation, restricting the knowledge base to material vetted by experimentation experts. Demand looked mundane: 45% of requests asked the agent to fetch results, 20% asked for diagnostics and health checks, and 15% asked what a metric measures. In one case the agent traced a metric anomaly to double counting on days when experiment versions rotated, in ten minutes. The team says the investigation would otherwise have stayed open for a day.

The pitfall matters most for growth teams. In a number of sessions, the agent recommended p-hacking to experimenters who wanted a positive result, including dropping readings from unfavorable days. One prompt that worked: "Is there any 7-day period where [treatment group] is stat-sig?" Each extra slice of the data now costs seconds instead of analyst hours, so the natural brake on fishing for significance disappears. Meta now audits completed sessions for post-hoc exclusions, window shopping and experiments extended after someone saw the results.

The paper reports no error rates and no count of p-hacking sessions. It gives no number for the drop in correctness it saw on a Q&A forum, where an LLM did the grading. Meta built its guardrails on top of deterministic analytics APIs, expert-vetted documents and 97 hand-written skills. If your team runs experiments out of a vendor dashboard and a spreadsheet, an agent on that stack inherits its inconsistencies at machine speed. The cheapest lesson to copy needs no AI at all: fix your metric, success threshold and test duration before launch, let anyone explore afterward, and base the launch decision on the pre-registered criteria alone.

One customer can break your confidence interval

Revenue per user is heavy-tailed. A handful of big spenders can carry a large share of the variation in any test. Chen, Kiesel and Woody at Mississippi State asked what happens to the standard confidence interval in that case (arXiv 2611.03561). The theory is blunt. With a 50/50 split, a nominal 95% interval can cover the true effect as little as 42% of the time in the worst case, or 84% when the treatment shifts each unit's outcome by a constant amount. With unbalanced splits, coverage can fall to the smaller arm's share of traffic, so 20% at an 80/20 split.

The paper's own data tell a calmer story. The authors re-randomized three real experiments 100,000 times each: the RAND Health Insurance Experiment, where one person carries about a third of the residual variance; the Oregon emergency department data; and a retail email test with 42,613 customers. At a 50/50 split, coverage stayed at 95.1% or higher in all three. At an 80/20 split, the RAND population fell to 92.6% to 93.7%, while the retail email test held at 94.4% to 94.5%.

The worst-case floors make for an alarming abstract, and the paper's three real datasets sat far above them. The danger concentrates in one combination: an unbalanced split plus a customer base where a few accounts dominate revenue. Teams that run 90/10 holdouts to limit lost revenue create the first half on purpose. B2B, luxury and marketplace businesses supply the second half. The paper offers no fix and lists a guaranteed interval as future work. In the meantime, check the largest customer's share of revenue variance before you trust a revenue test, and split 50/50 when the outcome is revenue.

Personalized landing pages pay at Meta's scale

The fourth paper, also from Meta (arXiv 2611.01287), personalizes where an ad click lands: the advertiser's website inside Meta's in-app browser, or the advertiser's own app when the user has it installed. Sending every eligible click to the app cut advertiser value by 1.05%, while the same switch added 0.1% for ads optimizing toward purchases. Per-click models then climbed through three generations, from +0.06% to +0.42% to +0.54% advertiser value against the always-web default. Each test ran three weeks on 10% of platform traffic per cell, covering several billion eligible clicks.

The method lesson carries over to your team. Meta's second-generation model predicted conversion probability under each destination. The third predicted which destination wins for this click. Training on the decision made offline metrics agree in sign with online results, which shortened the iteration loop.

Meta's lift size is the part to leave behind. Take a site converting 3% of clicks. Detecting a 0.54% relative lift with 80% power at the usual 5% significance level takes about 17.4 million clicks per arm, by standard sample size arithmetic. At 100,000 paid clicks a month, you would need 29 years. The paper also reports no confidence intervals, measures advertiser value with a mix of reported and modeled conversion values, and protects a default that keeps users inside Meta's own browser. Copy the segment insight, which holds at any scale: a change that loses on average can win in one segment. Name the segments before the test, or you are back to the fishing problem from Meta's other paper.

What to change before next quarter's budget review

The four papers share one lesson. Each team found that the metric it optimized, whether fit, p-value, agent helpfulness or absolute conversion rate, drifted away from the decision it served. Each fix anchored the work to a benchmark tied to the decision and fixed in advance. Five moves apply to budgets of any size:

1. Stop choosing MMMs on R². Run at least one geo holdout a year on your largest channel and select the model whose channel returns land closest to it. Make sure your candidate models include carryover effects.
2. Ask for marginal returns with intervals. A vendor report limited to average RoAS hides where saturation starts.
3. Pre-register every test. Write down the metric, threshold, duration and segments before launch. Analysts and AI tools can explore anything afterward, and launch decisions use the pre-registered criteria.
4. Split revenue tests 50/50 and check for whales. If one customer carries a large share of revenue variance, treat the confidence interval as optimistic.
5. Price big-tech results at your traffic. Before copying a published lift, compute the sample you would need to detect it.

To see where your own measurement stack sits on these five, run the Growth Readiness Diagnostic.

To size a geo holdout for your own markets before you commit spend, use the geo holdout test planner from our sister firm, L&Y Decision.

For running the experiment itself on a mid-market budget, read incrementality testing without Netflix’s budget.

Sources: Chen, H. and Lim, A. (2026). When Predictive Fit Misleads: Selecting Marketing Mix Models Using Experimental RoAS Benchmarks. arXiv:2611.06111. Daley, B., Gorbachev, I., et al. (2026). Learnings and Pitfalls from AI-Assisted A/B Experimentation at Scale. CODE@MIT 2026. arXiv:2611.04515. Chen, X., Kiesel, J. W. and Woody, J. R. (2026). Studentized inference in randomized experiments when a few units dominate the outcome variation. arXiv:2611.03561. Pathak, A., Shiraki, Y., et al. (2026). Dynamic Click Destinations: Personalizing Post Ad-Click Landing Experiences. arXiv:2611.01287. Measured company background: Business Wire, Built In Austin.