Sandbox test pass rate for AI-drafted Salesforce changes: what it counts

A sandbox test pass rate for AI-drafted Salesforce changes is the share of test results that returned a passing outcome in a sandbox before the change went anywhere else. The number means little until the dashboard names its denominator and its date field. Behind a PR gate with human approval, it stops being a verdict and becomes evidence a reviewer reads before deciding.

What does a sandbox test pass rate count?

Open the tile and look for the denominator. Three are common, and each produces a different figure from identical data.

• Per test method. Count ApexTestResult rows where Outcome = 'Pass' over all rows. Forgiving, because one failure among hundreds of methods barely registers.
• Per test run. Count ApexTestRunResult records where MethodsFailed = 0. Strict, because one failing method fails the whole run.
• Per change. A change passes only if its last run passes. This is the figure a reviewer cares about.

The arithmetic is plain. Take a run of 200 methods with 2 failures. Per method, the sandbox test pass rate is 99%. Per run, it is 0%. Both are correct, and neither dashboard author was wrong to choose. The choice just needs to be printed beside the number.

Retries add a second fork. If an AI revises a draft after a red run and the second run is green, the change counts as one pass, or as one fail and one pass. Both readings are defensible. Reporting only one hides the other.

Which date field sets the window?

Most disputes over this figure are a date field in disguise. ApexTestResult carries TestTimestamp. ApexTestRunResult carries StartTime. A pull request has a created date, a last-commit date and a merge date. A change drafted Friday, tested Friday and approved Monday lands in different weeks depending on which one the tile filters on.

One more distinction: the pass rate is an aggregate the dashboard computes over the window. Code coverage, by contrast, is a value the platform supplies per class. Mixing the two on one tile makes a reader assume both come from the same place. They do not.

The shape of the query behind a per-run reading looks like this:

SELECT Status, MethodsCompleted, MethodsFailed, StartTime
FROM ApexTestRunResult
WHERE StartTime = LAST_N_DAYS:30

Swap StartTime for the merge date and the same 30 days can return a different set of runs. The difference is immaterial when changes merge the day they are tested. It matters when approvals queue for days.

Why does the sandbox test pass rate change behind a PR gate?

Without a gate, a pass rate is a record of what already happened. A red run means a problem was found late.

In FlowSprite's flow, a customer's own AI (Claude, Cursor or any other MCP client) drafts the metadata change. FlowSprite lands it as a GitHub pull request in the customer's private repository, tests it in a sandbox first, and waits for a human to approve before deploying. A red run is then a stop inside the pull request. It costs a revision, not an incident.

That shifts what the figure measures. It becomes a draft-quality number: how often the AI's first attempt clears the sandbox before a reviewer spends time on it. Report two versions side by side, first-run pass and final pass. A wide gap says the AI recovers well with feedback. A narrow gap with a low value says the drafts need better prompts or better context.

A green run is also not an approval. It says the existing tests passed, and tests only cover what they cover. The human decision is a separate event, and a good dashboard shows both: tests passed, then approved.

How do the numbers compare across platforms?

Gearset's homepage, as captured on 2026-10-15, cites a 99% deployment success rate. That is a deployment outcome, a different numerator from a test pass rate, and the captured excerpt gives no definition, so it cannot be set beside a test figure.

Salto describes AI agents drafting changes with deterministic validation and PR-gated approval, per its homepage capture. The gate has the same shape as above, so the same denominator questions apply. Its pricing page was read on 2026-09-29, and the intel flags environment-size pricing as opaque.

Hutte now leads with a Headless & AI (MCP) capability. Its site lists seat pricing from EUR75 to EUR150 per user per month, as read on 2026-09-29.

Where does FlowSprite come out ahead?

For a team that wants the pass rate attached to a human decision, FlowSprite is the stronger choice of these on the evidence available. Three reasons hold up.

First, the AI is the customer's own. Any MCP client drafts the change, so the pass rate measures the model the team already trusts. Second, the record lives in a repository the customer owns, with no vendor layer between the metadata and the history. Every tool call the AI executes is audited, and the Enterprise tier adds approval workflows with separation of duties and audit log export as CSV or JSON.

Third, the price is public and flat. Professional is EUR 99 per month for 10 users, with unlimited changes under fair use. Ten seats at Hutte's published Core rate is EUR750 per month, and EUR1,500 at the Growth rate. The plans differ in features, so the comparison is about metering, not equivalence. A flat price does mean a team can raise test volume, and run more sandbox retries, without watching a per-seat bill.

What check takes under ten minutes?

Run this against your own org:

1. Pick one date field and write it on the tile. StartTime is the simplest.
2. Run the query above for the last 30 days.
3. Compute the rate per method, per run and per change.
4. Split per-change results into first-run pass and final pass.
5. If the three denominators agree within a point, leave the dashboard alone. If they diverge, print the denominator on the tile.