Platform Foundation Experimentation

Stop guessing what works. Let it prove itself.

AI agents turn goals into experiments, run them as adaptive tests that learn while they run, and keep a fixed control so the result is honest. If it can be measured, it can be improved.

  • Contextual bandits and classic A/B
  • Optimises for margin, not clicks
  • Guardrails, approvals and rollback
Laboratory flasks next to a rising bar chart
Winner scaled on day 9Lift held against the control

Traffic share · day 7

  1. Creative A41%
  2. Creative B28%
  3. Holdout10%
Tests per month Valid experiments completed
Time to decision Days from hypothesis to a call
Cost of learning Less traffic spent on losing variants
Proven lift Gains that survive a fair test

Why change

A before-and-after chart is not an experiment.

Most marketing decisions are still made on correlation. The teams that do test properly run a handful a quarter, because every test is hand-built.

Decisions ride on before-and-after

Seasonality and audience mix get mistaken for impact.

Too few tests get run

Ideation, setup and analysis are all manual.

Tests collide

Overlapping exposures and broken assignment contaminate the result.

Winners get lost

The learning lives in a slide and the rollout never happens.

What it does

Test more. Waste less. Trust the answer.

Language models plan, explain and generate. Statistical learning chooses and allocates. A control group keeps everyone honest.

01 Adaptive learning

Traffic moves towards what works, while it is still working.

A fixed A/B/n test gives every variant an equal share until it ends. An adaptive policy shifts share as evidence builds, keeps exploring in case it is wrong, and records the exact probability behind every choice.

  • Multi-armed bandits for the best overall creative, subject line or offer.
  • Contextual bandits for the best action for each kind of customer.
  • Reinforcement learning, introduced selectively for multi-step journeys and lifetime value.

Traffic share by creative

Creative A
25%
Creative B
25%
Creative C
25%
Creative D
25%

Fixed A/B/n test

25% each for 14 days, then pick a winner.

Adaptive agents

Shift share as evidence builds. Keep exploring.

Illustrative allocation, not a prediction. Actual shares depend on evidence, uncertainty and guardrails.

02 The right action for each context

The best offer overall is rarely the best offer for everyone.

A simple test might crown one winner. A contextual policy learns that the discount wins with leisure travellers, lounge access with business travellers and extra baggage with families, and serves each accordingly.

  • Reward defined by you: incremental contribution margin, not click-through.
  • Only pre-decision signals used, so the model cannot cheat.
  • Delayed outcomes handled, including returns and cancellations.
Learned policy · Premium seat offerPolicy v12
Customer context10% discountExtra baggageLounge access
Leisure travellers55%30%15%
Business travellers10%25%65%
Family travellers20%65%15%
3eligible actions
10%fixed holdout
Marginreward metric
Share of decisions by action, per contextIllustrative

Illustrative learned preferences for a fictional airline, not conversion rates.

03 Trustworthy measurement

Adaptive optimisation is not proof. So there is always a control.

Every policy runs against a randomised holdout. Results come with intervals, data-quality checks and a clear recommendation: ship, iterate or stop. For spend decisions, experiments calibrate attribution and mix modelling.

  • Sample-ratio and event-quality checks before anyone reads a result.
  • Collision detection and mutual exclusion across concurrent tests.
  • Incrementality, geo tests and mix modelling, each used for the question it suits.
Result · Policy v12 vs holdoutChecks passed
Contribution per visitorPrimary metric · 95% interval +2.1% to +6.3%+4.2%
Conversion rateSecondary+3.1%
Cancellation rateGuardrail · must not riseStable
Unsubscribe rateGuardrail · must not riseStable
Sample ratio and event qualityAssignment as designedPassed
Recommendation: ship to 100%, keep 5% long-run holdoutIllustrative

Agents at work

Six jobs, one learning loop.

Planner, creative, optimisation, measurement, learning and governance agents each own a step. They are logical roles, not six chatbots: the choosing is done by statistics, the checking by policy, and the final call on anything material by a person.

  • Hypotheses from real signals: customer, campaign and conversion data, with a rationale.
  • Power and duration planned, so tests end with an answer.
  • An evidence library, so nobody re-runs the test someone ran last year.
Agent activity  Live
Planner: brief turned into 3 hypothesesReward: contribution margin · 14-day windowAutomated
Creative: 4 variants generatedBrand, legal and channel checks passedNeeds approval
Optimiser: share shifted towards A and BExploration floor kept at 5% per armAutomated
Measurement: lift holds against controlRecommends full rolloutNeeds approval
Governance: variant D exceeded complaint guardrailArm stopped, traffic reallocatedStopped
Every step logged with inputs, reason and outcomeIllustrative

You decide what the agents may do here.

AutomatedRunning the experiment

Assignment, allocation within exploration limits, quality checks, stopping a variant that breaches a guardrail.

Human-approvedDeciding what it means

Launching a new test, changing the reward metric, full rollout of a winner and budget moves above threshold.

RestrictedHard stops

Tests without a control, customers outside consent, and any variant that breaks brand, legal or frequency rules.

What's inside

A complete experimentation practice, not a testing tool.

Across web, app, messaging, recommendations, loyalty and media, on one set of metrics.

Idea registry

Hypotheses, rationale and prioritisation in one backlog.

Experiment design

Primary, secondary and guardrail metrics; power and duration.

A/B and A/B/n testing

Client-side and server-side, with feature flags.

Bandits and contextual bandits

Adaptive allocation with logged selection probabilities.

Reinforcement learning

For sequential decisions, used where the complexity is earned.

Assignment and exclusion

Consistent bucketing, targeting and mutual exclusion groups.

Quality diagnostics

Sample-ratio mismatch, event checks and collision detection.

Inference

Frequentist or Bayesian, sequential testing, variance reduction.

Holdouts

Short-run controls and long-run holdouts for novelty and retention.

Incrementality and mix modelling

Geo tests, response curves and budget scenarios.

Staged rollout

Gradual release, approvals and one-click rollback.

Learning repository

Every result, searchable, linked to its assets and data.

How it runs

From a hypothesis to a decision that sticks.

Each step is a named, logged stage. Nothing autonomous is switched on until its data, permissions, monitoring and rollback have been checked.

  1. Frame a falsifiable hypothesis
  2. Define reward and guardrails
  3. Size it and set the control
  4. Run randomised or adaptive exposure
  5. Validate assignment and events
  6. Estimate effect with uncertainty
  7. Approve rollout or roll back

How you'll measure it

A winning idea is one that survives a fair test.

We agree the numerator, denominator, cohort and success gate with you before anything goes live, and compare against a baseline or a holdout, never against a hunch.

  • Valid experiments completedTests per month that pass data-quality and assignment checks.
  • Median decision cycleDays from an approved hypothesis to a decision.
  • Experiment validity rateTests passing predefined quality gates, divided by tests completed.
  • Incremental liftTreatment minus control on the primary metric, with its interval.
  • Guardrail breachesConfirmed safety or business metric breaches during tests.
  • Win-to-rollout timeDays from a validated winner to full approved deployment.

Questions

Things people ask before they start.

Is a bandit just a faster A/B test?

Not quite. A bandit minimises the cost of learning by sending less traffic to losers while the test runs. It does not, by itself, prove causal lift, which is why every adaptive policy here also runs against a randomised holdout.

Do language models decide where the traffic goes?

No. Language models plan experiments, generate variants and explain results. Allocation and policy updates are made by statistical learning algorithms on logged reward data.

When is full reinforcement learning worth it?

When the decision is sequential and the reward is long-term, such as a multi-step lifecycle journey. Most programmes should start with bandits, move to contextual bandits and add full RL selectively.

We have little traffic. Can we still test?

Yes, with realistic expectations. Power planning tells you up front what size of effect you can detect, and adaptive methods make better use of limited traffic than fixed splits.

Composable by design

Useful alone. Sharper when connected.

Experimentation is the proof layer. Each capability it connects to gets a fair test and a growing evidence library.

Keeps your existing stack

Runs on your web, app, messaging and ad channels through SDKs and APIs. Reads outcomes from your analytics, warehouse and order systems.

Runs where your data lives

Small language models trained on marketing data are deployed inside your cloud or data centre, so customer data and the models reasoning over it stay under your control.

Getting started

Start with one decision you argue about every quarter.

Pick a question with real money attached. Agree the reward and the guardrails, run it properly once, and let the result settle the argument.

  1. Agree the question, reward and guardrails
  2. Instrument events and set the control
  3. Run the test with adaptive allocation
  4. Decide, roll out and log the learning