AdOpt experiment dashboard

Two sources, each used for what it is good at. Elasticsearch carries the arm assignment, so it supplies every ratio and uplift. MySQL carries the billed ledger, so it supplies every absolute dollar figure.

Arms

• Baseline = control — roughly 5% of requests.
• Optimized = adopt_thompson_balanced — roughly 95%.
• Grouping uses model_group, the randomised assignment, not treatment.

Why per 1M requests, not per impression

AdOpt changes how many impressions you win, so impressions are an outcome of the test, not a fixed denominator. Requests are the randomisation unit, so every rate is divided by SUM(sample_weight) across both log streams (no_bid weight 100, has_bid weight 10). Measured per impression, a winning treatment can look like a loss.

Which denominator a breakdown uses

Fields beginning win_ — endpoint, preset, AdOpt flags — only exist on documents where a bid won. So the request count inside one of those buckets is really a win count, and a win is an AdOpt outcome. Dividing by it would silently turn every rate into a per-impression figure and flip the sign of the uplift. Those breakdowns are therefore normalised against the arm's total client requests, the same denominator the headline figures use.

Request-level fields — country, SSP, device, publisher — appear on every document, no_bid included, so their own bucket count is a genuine request count and is used directly.

Money in Elasticsearch

Gated on is_impression = 1. Revenue is SUM(cost_w) and cost is SUM(rev_w) — the two materialised fields are stored swapped relative to their names. Prices are CPM, so both are divided by 1000.

Where the dollars come from

Every dollar on this page comes from MySQL — <client>.stats, with revenue = dsp_spend and cost = ssp_spend. Clients are spread over two servers and both are read, so no client falls back to an estimate.

Elasticsearch never supplies a dollar amount. It sees only a sampled slice of traffic — about a third of anzu's revenue, for instance — so its absolute figures would understate the business badly. What it does supply is the split between the two arms, and that ratio is unbiased, because both arms are sampled in exactly the same way.

Extra revenue and extra profit

The MySQL amount multiplied by the Elasticsearch uplift, counting only the share of the book that actually ran optimized: extra $ = MySQL $ × treatment share × u/(1+u). So if the ledger booked $40,000 of profit, ~95% of it on the optimized arm, and that arm earned 6% more per request than the baseline, AdOpt added roughly $2,150. Baseline traffic never carried the uplift, so it is excluded from the attribution.

Which endpoints appear

By default the table lists only endpoints actually running AdOpt — those with the margin optimiser switched on, or with at least half their optimized-arm impressions served with it active. Everything else is a bystander whose arm-to-arm difference is noise. Turn off “AdOpt endpoints only” to see them all.

The uncertainty band

Baseline is only ~5% of traffic, so thin slices move around a lot. The uplift tiles show an indicative ±band (95%): when the uplift sits inside its band, the window is too short or the slice too thin to call a direction — treat it as "no measurable effect", not a win or a loss. Rows with under 3,000 baseline impressions carry a low n badge (their percentages are noisy); rows with under 50 have no usable baseline at all and show n/a. The "AdOpt active" column shows what fraction of an endpoint's window actually ran AdOpt — a small uplift on a low-active endpoint is dilution, not a weak result.

Coverage

Elasticsearch retention sets how far back uplift can be measured — currently about a month, and the date picker is bounded by what is actually retained for the selected client (some clients were onboarded later and have a shorter history). MySQL goes back much further, so booked dollars still render for older ranges. Long windows (14–30 days) scan far more data and can take a minute or two to load. All times are UTC.