AdOpt experiment dashboard

Two sources, each used for what it is good at. Elasticsearch carries the arm assignment, so it supplies every ratio and uplift. MySQL carries the billed ledger, so it supplies every absolute dollar figure.

Arms

• Baseline = control — roughly 5% of requests.
• Optimized = adopt_thompson_balanced — roughly 95%.
• Grouping uses model_group, the randomised assignment, not treatment.

Why per 1M requests, not per impression

AdOpt changes how many impressions you win, so impressions are an outcome of the test, not a fixed denominator. Requests are the randomisation unit, so every rate is divided by SUM(sample_weight) across both log streams (no_bid weight 100, has_bid weight 10). Measured per impression, a winning treatment can look like a loss.

Which denominator a breakdown uses

Fields beginning win_ — endpoint, preset, AdOpt flags — only exist on documents where a bid won. So the request count inside one of those buckets is really a win count, and a win is an AdOpt outcome. Dividing by it would silently turn every rate into a per-impression figure and flip the sign of the uplift. Those breakdowns are therefore normalised against the arm's total client requests, the same denominator the headline figures use.

Request-level fields — country, SSP, device, publisher — appear on every document, no_bid included, so their own bucket count is a genuine request count and is used directly.

Money in Elasticsearch

Gated on is_impression = 1. Revenue is SUM(cost_w) and cost is SUM(rev_w) — the two materialised fields are stored swapped relative to their names. Prices are CPM, so both are divided by 1000.

Where the dollars come from

Every dollar on this page comes from MySQL — <client>.stats, with revenue = dsp_spend and cost = ssp_spend. Clients are spread over two servers and both are read, so no client falls back to an estimate.

Elasticsearch never supplies a dollar amount. It sees only a sampled slice of traffic — about a third of anzu's revenue, for instance — so its absolute figures would understate the business badly. What it does supply is the split between the two arms, and that ratio is unbiased, because both arms are sampled in exactly the same way.

Extra revenue and extra profit

Simply the MySQL amount multiplied by the Elasticsearch uplift: extra $ = MySQL $ × uplift %. So if the ledger booked $40,000 of profit and the optimized arm earned 6% more per request than the baseline, AdOpt added roughly $2,400.

Which endpoints appear

By default the table lists only endpoints actually running AdOpt — those with the margin optimiser switched on, or with at least half their optimized-arm impressions served with it active. Everything else is a bystander whose arm-to-arm difference is noise. Turn off “AdOpt endpoints only” to see them all.

The uncertainty band

Baseline is only ~5% of traffic, so thin slices move around a lot. The "AdOpt active" column shows what fraction of an endpoint's window actually ran AdOpt — a small uplift on a low-active endpoint is dilution, not a weak result.

Coverage

Elasticsearch keeps about 7 days, so uplifts are unavailable before that. MySQL goes back much further, so booked dollars still render for older ranges. All times are UTC.