Skip to main content

Software simulations

A software run drives real browsers — or a real Android emulator — through your product with a population of synthetic users, and reports where they got to and what they made of it.

Launching a runDirect link to Launching a run

From a software cohort, Launch run:

FieldNotes
GoalWhat agents try to accomplish, in plain English — "complete checkout", "finish onboarding". Required. It is what the agents are trying to do; without it there is nothing to succeed or fail at.
TargetURL or uploaded APK. Defaults to the cohort's target.
Agent countNumber of sessions. Defaults to the cohort's.
ArmsOptional A/B — see below.
Debug captureOptional. Records what each agent saw and why it acted. See Reading results.

Runs take hours to days depending on agent count. Status moves QUEUED → RUNNING → COMPLETED, or FAILED / CANCELLED. You can cancel a run that is queued, running or paused.

A/B armsDirect link to A/B arms

An arm is a target plus a label. Give a run two arms and one cohort is split deterministically between them, with every figure reported per arm and a delta.

Arm A (Control)  →  https://app.example.com
Arm B (Variant)  →  https://staging.example.com/new-checkout

This matters more than it looks. The obvious alternative — run two cohorts, compare the numbers — means the two populations were drawn separately and any difference might be the populations rather than the change. Splitting one cohort removes that confound, and because the split comes from the session seed, re-running the seed reproduces the same allocation.

The audience mix is assigned before the arm, so both arms draw from the same audience.

One arm is not a special case

A run with no arms, or one arm, behaves exactly like an ordinary single-target run. Arms are purely additive.

The staging workflowDirect link to The staging workflow

The pattern the feature exists for:

  1. Calibrate against production. Run the cohort against your live product and calibrate it, so the population matches observed behaviour.
  2. Point it at staging. Same cohort, same calibration, different target — your feature branch, new paywall, redesigned checkout.
  3. Diff the archetypes. The group that moves most is the group to pay attention to.

Or do it in one run with two arms, which is the same comparison with the population confound removed.

AndroidDirect link to Android

Android runs drive a real emulator rather than a browser. Upload an APK to the cohort or the run; the rest of the flow is identical. Screenshots are captured per session the same way.

What a software run reportsDirect link to What a software run reports

Covered in full in Reading results. In short:

  • Conversion and goal fulfilment — did agents reach the goal.
  • The funnel and drop-off — per page, how many arrived and how many left from there. A page nobody reached is reported as not reached, which is not the same as 0% drop-off.
  • Session duration — as a distribution, not just a mean.
  • The response battery — what agents made of it: task clarity, friction, confidence, intent, and why they abandoned. Asked after the session; it never influences navigation.
  • Predicted versus observed — the action distribution your model predicted for each bucket, against what the agents actually did. If those diverge sharply, either the model is wrong about this cohort or the simulation is, and both are worth knowing.

Every figure breaks down by archetype, and by arm when you used arms.

NextDirect link to Next