A/B TESTING · BEFORE THE SPEND
A/B test before you buy the traffic. Spend only on the winner.
Your AI agent runs the A/B test mid-task over the CrowdOS MCP server, on every plan, Free included. A census-grounded panel that disagrees with itself — not one model's opinion — reads every variant and ranks them. No traffic split. No budget burned on the variant that loses.
HOW IT RUNS
Every respondent sees every variant.
A live A/B test splits real visitors between variants. CrowdOS shows every variant to every respondent and asks for a choice. The same person judges A and B, so the difference between them is the variant, not the sample.
Over MCP the agent calls compare_options on two to ten text options and gets a Borda leaderboard, a winner and sample reasoning. For image and video ads, Ads Labo runs the same within-panel design on 2–4 variants and backs its call with a paired sign test. Your agent gets the same functions over MCP as the web app on your plan.
THREE WAYS TO RUN IT
The same crowd, from your agent or the web app — the same functions on your plan.
01
compare_options, over MCP.
Headlines, primary text, slogans, value propositions. Each respondent ranks every option; Borda counts produce the leaderboard. Panels follow your plan, the same as in the web app. preview_cost and estimate_reach are free to call.02
Ads Labo, from your agent or the web app.
2–4 image or video variants with Meta-style targeting. A forced choice, or neither. A ship / tied / rework verdict with a paired sign test, preference shares with 95% Wilson intervals, and cohort flips.03
Pulse Comparison, from your agent or the web app.
A/B/C/D headlines, taglines and value propositions, with verbatim reasoning on which language resonates with which segment.04
Then the live test.
Take the winner live. The panel reports relative preference; the live budget reports click rate, CPM and conversion.
THE OPINION BENCHMARK
The published opinion benchmark.
On the Pew opinion benchmark, the crowd's opinion distribution matches published polling at 93.3% parity, with a mean absolute error of 3.36 percentage points and a Pearson r of 0.984. That figure measures opinion. Creative ranking against a live campaign is a separate measurement.
THE CROWD
Census distributions, then the question.
The crowd is drawn from census distributions across 24 markets.
WHERE THIS FITS
The same crowd, asked a different question.
Pre-launch testing is the step before a live A/B test. Ad creative testing is the ad itself. This page is the A/B test, run on a panel instead of on paid traffic.
FAQ
Can you A/B test without traffic?
Yes, as a pre-flight read. Your AI agent runs compare_options mid-task over the CrowdOS MCP server, on every plan, Free included. Every respondent in a census-grounded panel ranks every variant, and the leaderboard names the winner. Live traffic then confirms the absolute numbers on that one.
How do I A/B test ad creatives before spending on ads?
Put two to four image or video variants into Ads Labo — in the web app, or have your agent send them over MCP. One panel sees all of them in randomized order and picks the one they would tap, or neither. The report leads with ship, tied or rework, backed by a paired sign test, with a 95% Wilson interval on each share.
Does it replace a live A/B test?
It decides which variant earns the live test. The panel reports relative preference. Click rate, CPM and CPI are auction outcomes, and CrowdOS does not output them.
Why not ask one model which variant is better?
One model acting like N personas is effectively n=1. The CrowdOS panel is drawn from census data and disagrees with itself, so the winner has to hold up against disagreement.
How many variants can I compare?
Two to ten text options per compare_options call, within your plan's limits — the same as the web app. Ads Labo compares two to four image or video variants on one panel, from the web app or over MCP.
Spend on the variant that already won.
Run the A/B test from your agent first. Buy the traffic second.