Published Sep 29, 2026• Updated Nov 20, 2026

Test Bug or Product Bug? Triaging Playwright Failures with a CPU-Only Decision Model and an LLM

When CI goes red: is it the test, the product, or the environment? Playwright failures through a local CPU-only decision model that abstains when unsure, an LLM for the escalated cases, and a verification loop that re-runs every proposed fix — including the parts that did not work.

Test Bug or Product Bug? Triaging Playwright Failures with a CPU-Only Decision Model and an LLM

When CI goes red, the same interrogation starts: is it the test, the product, or the environment? In this experiment I wired Playwright failures through a local, CPU-only decision model that abstains when it is unsure, an LLM that only works on the escalated cases, and a verification loop that re-runs every proposed test fix before anyone sees a pull request. The results include the parts that did not work — which is exactly where it gets interesting.


Red CI, three suspects

Every failing end-to-end run asks the same question. A locator timed out: did someone rename a label (test drift), did the feature actually break (product regression), was it a slow render (flake), or is the environment misconfigured? Answering it by hand is the most repetitive part of QA work — and the part most worth delegating to machines, if the machines can be trusted.

My constraints for this experiment:

  1. Cheap first pass. It has to run on the CI runner itself — no GPU, no external API for the majority of cases.
  2. Honest uncertainty. The first pass must be allowed to say "I don't know" and hand over.
  3. Verifiable output. A test fix is not a suggestion: it must re-run green on the failing test before it becomes a PR. If the product is broken, produce a bug report with evidence — never a patch that silences the alarm.

So the pipeline is a three-stage system: deterministic signals → a small decision model (System 1) → an LLM (System 2) only on escalated cases.

The pipeline

The triage pipeline: Playwright failures are distilled into compact states, routed by the fine-tuned decision model with a fitted gate, and escalated to the LLM only on abstentions.

Deterministic distillation → a small CPU decision model with a fitted gate → the LLM only on abstentions.

Each stage is one small script — distill.mjs, decide.mjs, act.mjs — and the whole loop runs on a stock GitHub Actions runner. You can find everything in the companion repo (link at the end).

Ground truth first: a tiny app and nine injected faults

You cannot evaluate a triage system without ground truth, and you rarely have it for free. So I built it. TaskDeck is a deliberately boring task board (Next.js 16, one Playwright spec, four tests). Around it, a scenario framework: each scenario is a small mutation — to app code, to the spec, or to the config — with a known label:

  • 5 product bugs (counter counts completed tasks, toggling does not persist, filters inverted, the create endpoint returns 500, the empty state disappears);
  • 2 test-side faults (a renamed aria-label the spec still targets; a stale expected value);
  • 1 flake (randomness injected into the app);
  • 1 environment fault (a baseURL pointing at a dead port).

A collector applies each mutation, runs the suite, snapshots the results, and reverts everything. Ten runs, seventeen failing tests, every failure labeled by construction — the perfect little laboratory. (Fault injection is the underrated QA skill here: it produces labels for free, and it generalizes to "replay" old bugs.)

System 1: Laya — decisions without generation

Laya is an open-source "System 1" decision model: you give it a state (some text, an error, a JSON object) and typed questions — choice, score, or noul (a calibrated P(true)) — and it answers all of them in a single forward pass over a ~0.4B-parameter encoder. No text generation, so nothing to parse and nothing to hallucinate; the Node runtime (@receptron/laya) runs the ONNX weights on CPU, no Python needed.

The entire triage decision is one question set:

js
const QUESTIONS = {
  origin: { type: "choice",
    instructions: "What is the most likely origin of this Playwright test failure?",
    criteria: {
      test:        "the test is wrong or outdated: selector drift, stale assertion, missing wait",
      product:     "the app is broken: a real regression the test correctly detects",
      flake:       "timing or randomness: passes on retry with no code change",
      environment: "browser, ports or CI infrastructure misconfiguration",
    } },
  test_side:    { type: "noul", instructions: "Does this failure originate in the test rather than the product?" },
  product_side: { type: "noul", instructions: "Is this a genuine regression of the app that the test correctly detects?" },
  severity:     { type: "score", instructions: "How severe is this failure for the product?",
                  criteria: ["cosmetic", "degraded behavior", "broken user flow", "data loss or outage"] },
};

The results that humbled me

First run, naive design — trust the top choice: 14 automatic decisions out of 17, and 13 of them wrong (accuracy on auto-decisions: 7%). The model answered "flake" for almost everything, with confidence between 0.02 and 0.09. The confidence was the honest part: on a 4-way choice, near-zero confidence means "these questions mean nothing to me". Then the gate fix — require confidence on every route, flake and environment included — turned those 13 confident-looking mistakes into 17 abstentions. Which is the correct behavior for a model that does not know: escalate, don't guess.

Two findings came out of debugging this, and both are the kind of thing a blog post usually skips:

The token budget is a design constraint. The runtime builds one sequence per question, capped at max_len: 512, and the state is truncated to fit — in JSON field order. With a rich state, the tail fields (including the changed-files list) were silently cut for the most constrained question. You plan for a 512-token budget: the most diagnostic fields go first, the attachments go in the trash.

Better input did not save it. I distilled v2 (cleaned error without Playwright's call-log noise, the actual diff of the changed file, the failing test's source) and then ran an ablation on two real failures across five representations — full state, no attachments, reordered, minimal core, and a hand-written perfect summary. Best confidence anywhere: 0.17. Even the perfect digest stayed under any usable gate — and for the product bug it still preferred "test". Conclusion: for this domain, the shipped base checkpoint has no usable representation, zero-shot. Not an input problem. This is exactly why the architecture has an escalation path — and why the next chapter is fine-tuning on your own corpus (the project's own benchmarks tell the same story: 0.36 zero-shot → 0.77 fine-tuned on their typed-decisions suite).

System 2: the LLM, only where it counts

Everything that abstains lands in a deep-triage call: the failure state plus the decision model's reasoning go to deepseek-chat, which must reply with a single JSON object — origin, rationale, recommended action. Then the matching artifact is generated: a complete replacement spec file (no fragile patch-apply), a bug report, a flake entry, or an environment alert.

Same seventeen failures, now with the LLM making the call: 16/17 agreement with ground truth — test 4/4, product 8/8, environment 4/4, flake 0/1. Three stories from the corpus:

Case 1 — a product bug → a paste-ready report. The counter scenario (completed tasks still counted as active). The report leads with severity, a numbered reproduction derived from the failing test, the quoted error, the diff line that caused it, and the fix — plus one honest footnote: the test never reached its second assertion, so the filter behavior remains unverified. That footnote is the kind of thing you usually get from a good human.

A real failure screenshot from the collected corpus — the counter scenario behind Case 1's bug report.

A real failure screenshot from the collected corpus: the counter scenario behind Case 1's bug report.

Case 2 — selector drift → a patch that was verified. The label rename scenario. The candidate fix updates the locator in all three tests that fill the input, leaves every assertion intact, and its rationale flags the residual risk ("if the rename was unintended, the honest fix is on the app side"). Then the verification loop: swap the candidate into the spec, re-run the exact failing test with the fault still applied, restore everything. verify: PASS. For one other case the LLM's prose was muddled while its file was still correct — which is precisely why these outputs go to a PR with a human in the loop, never an auto-merge.

Case 3 — the flake that wasn't. The randomness-in-app scenario. Even with retry evidence ("attempt 1 failed; retry passed"), the LLM classified it as a product defect, arguing: "the retry passing is explained by the injected randomness, not by test flakiness." Honestly? Defensible — the randomness is real code, and our label ("flake") needs a criterion the state cannot carry: flakiness is a property of history (patterns across runs), not of a single failure. Keep flake tracking driven by retry statistics, and keep humans on the genuinely ambiguous cases.

Verified or it didn't happen

The verify step is the soul of the design:

bash
node scripts/triage/verify.mjs \
  --candidate runs/test-renamed-label-drift/actions/state-01-spec.candidate.ts \
  --spec e2e/tasks.spec.ts \
  --grep "creating a task shows it in the list and updates the counter" \
  --scenario test-renamed-label-drift
# verify: PASS — creating a task shows it in the list and updates the counter

Terminal output: the generated patch re-runs the exact failing test with the fault still applied — verify: PASS.

Real output from the CI demo pipeline: the generated patch re-runs the exact failing test with the fault still applied.

It swaps the candidate in, runs the single failing test with the fault applied, restores the original byte-for-byte, reverts the scenario, and checks the tree is clean — pass or fail, the workspace is never left dirty. All three generated candidates re-ran green. A test fix that has not re-run green is a rumor; this loop turns it into evidence.

What runs where (cost and speed)

  • Laya: ~1.5 s per failure on an idle machine for the base checkpoint (8–13 s per state for the fine-tuned fp32 bundle on a stock runner), zero API cost, 1.7 GB of weights cached once. In the zero-shot era its most common answer was "I don't know" — on purpose; the fine-tuned model now routes 84% of states and still abstains on the rest.
  • DeepSeek: only on escalations, seconds per call, cents for the whole corpus of seventeen failures. It writes the artifacts a human can review.
  • CI: both stages run on a stock runner; no GPU anywhere in the loop. The workflow is in the repo: Playwright job, then a triage job that distills the failures, runs the decision model, and acts — in dry-run mode when no API key is configured.

Fine-tuning the decision model: three rounds deep

The loop I sketched is mechanical — dataset, training, export, evaluation — and it ran three times on Kaggle (the official fine-tune notebook, a 2×T4 instance; each round trains in minutes). The corpus grew alongside it: 9 injected faults → 24 → 43 → 49 scenarios, 92 labeled decision states (split by scenario so no group leaks between train and validation).

The data loop: fault injection labels the corpus for free; each round re-trains, re-exports and re-evaluates in CI.

The data loop: fault injection labels the corpus for free; each round re-trains, re-exports and re-evaluates in CI.

Round one, on the small corpus, confirmed the failure mode: the model learned a flat "product" bias at ~0.05 confidence and abstained on everything. Round two learned to route product failures but still abstained on the rest. Round three targeted the two classes with the fewest examples — flake (3 scenarios) and environment (3) — with six new scenarios, and fixed the collector so flaky runs are distilled from the first failing attempt instead of assuming attempt one is red. Measured by the same CI evaluation over the full corpus:

roundstatesauto-routedcorrectprecisioncoverage
1 — seed corpus420——0.00
2 — 43 scenarios7731311.000.40
3 — 49 scenarios, fitted gates9277771.000.84

Automatic routing coverage across the three rounds: 0.00 → 0.40 → 0.71 → 0.84 (same CI evaluation).

Routing coverage across the rounds, same evaluation harness (0.71 is round 3 before the gate fit, 0.84 after).

By class in round three: product 39/39 and environment 22/22 — most of the latter learned from the twelve states added in this round — plus flake 5/7 and test 11/24. Origin argmax is right on 89/92 states, and all three misses land in the escalation path (confidence or the follow-up gate stops them). The 15 abstentions go to the LLM, exactly as designed.

The placeholder gate was a silent tax

The routing thresholds — 0.30 origin confidence, and 0.75 for the "is this really test-side / product-side?" follow-up — were first-cut placeholders, and the fine-tuned model kept kissing the 0.75 wall: its test-side probabilities ran 0.31–0.75, close to the gate without crossing it for most states. So I re-routed the evaluation's raw answers across a grid of candidate gates:

conf \ noul0.600.650.700.75 (old)0.80
0.2582/80/278/77/168/68/065/65/063/63/0
0.3081/80/177/77/068/68/065/65/063/63/0
0.3581/80/177/77/068/68/065/65/063/63/0

(cells: auto / correct / wrong over the 92 states.) Lowering the follow-up gate to 0.65 turns twelve abstentions into twelve more correct decisions — coverage 0.71 → 0.84, still zero wrong — and the first mistakes appear only at 0.60. Same model, same inputs: the gate mattered as much as the retrain.

The routing logic: every route needs origin confidence ≥ 0.30; test/product additionally need their noul ≥ 0.65, or the state escalates.

Every route needs origin confidence ≥ 0.30; test/product additionally need their noul ≥ 0.65 — otherwise the state escalates.

(The operating point is fitted on the corpus, so it re-fits whenever the corpus or model changes; a fresh CI run confirmed the new point end-to-end — and yes, it is fitted and evaluated on the same 92 states, a caveat worth keeping in mind at this scale.)

The quantization that ran but flatlined

Shipping 1.7 GB of weights felt wasteful, so I quantized the bundle to INT8 (425 MB). It took three recipes to even produce a file that loads: the export's external data files confused the preprocessor, and ONNX Runtime's shape inference refused the graph (1028 vs 256) until I stripped the intermediate shape annotations. Then "success": the quantized model loaded and ran in the Node runtime, a dummy-input check showed a perfect 0.0 difference from fp32, and the CI accepted it — whose evaluation came back with zero automatic decisions, everything abstaining at confidence 0.0002.

The catch: on real failure states, INT8's output probabilities collapse to near-uniform (~0.25 across four origins) where fp32 produces confidences from 0.30 to 0.85. The dummy check compared the two models on a random-token input on which fp32 itself was saturated — [[1.0, 0.0], …] — so both looked identical: a false pass built on a degenerate test. The workflow is pinned to the fp32 bundle until an INT8 validated on real states exists. The lesson generalizes beyond this project: quantized artifacts need a real-input parity check, not just a successful load.

What's still next

Round three's remaining abstentions point at the next iteration: thirteen test-side states sit just under the 0.65 follow-up gate (0.31–0.64), and test is the last class without full routing coverage — more test scenarios, or a better test_side head, is the obvious next lever. Two flake states still escalate (one is a weak-evidence sub-run; the other is the only remaining origin misclassification — another reminder that flakiness is a property of history, not of a single failure). And the INT8 path deserves a real-input validation harness before it ships. Every review outcome is still a labeled example — the loop just needs more of them, which is a nice problem to have for once.

So when is this worth it? (And what does it cost to keep?)

None of this is worth building for a team that sees three red runs a month. The design pays off under a specific stack of conditions:

It earns its keep when…

  • Failure volume makes per-failure API calls feel wrong. In the last evaluation, 84% of states were resolved on the runner — free, seconds of CPU, no data leaving the machine — and only the 15 abstentions ever need an API. If your alternative is "send every failure to a model", the decision layer turns that into a minority tax.
  • Labels are cheap. Everything runs on ground truth here, and fault injection manufactured it: 49 scenarios produced 92 labeled states for the cost of writing small mutations. Where the data was thin, the loop visibly slowed (the test_side head is still waiting for the test scenarios I haven't written yet).
  • Fixes are cheap to verify. verify.mjs re-runs a single test — that is the property that makes generated patches trustworthy. If verification cost more than writing the fix by hand, I would keep the triage and drop the generation.
  • The taxonomy is small and stable. Four origins, typed questions. A taxonomy that changes weekly turns every regression into rework — though here "retrain" means 12 minutes of borrowed GPU, not a quarter of ML engineering.

It's overkill when…

  • Volume is low: a prompt, a retry tracker and a human rotation are cheaper than maintaining a corpus, a dataset and a training kernel.
  • Flakes dominate: retry statistics beat any model we ran on them.
  • Nobody owns the flywheel: the fine-tune only pays back if review outcomes and new failures keep flowing into the corpus.

The maintenance bill — what this actually spent

  • Fine-tuning (the part that sounds expensive): ~12 minutes of free Kaggle GPU per round. Every training run in this project fit comfortably inside Kaggle's weekly free quota; the dollar cost of three model generations was zero. The real cost is the data, not the training: six new scenarios took about two hours including local validation, and collection + dataset regeneration run unattended in CI (~6 minutes).
  • The escalation bill: cents. DeepSeek only sees abstentions; a three-escalation run makes six calls in about 14 seconds. Across the whole project, the LLM bill never left the cents.
  • CI: free on a public repo, and 20–50 minutes of wall-clock per full loop (Playwright → triage → collect → retrain → evaluate), almost all of it waiting.
  • Human rhythm: an hour or two per maintenance round — write scenarios for whatever the abstention stream keeps showing, retrain, re-fit the two thresholds on the evaluation output (minutes), re-run. The first build is the expensive part; everything after it is a chore — which is exactly what you want from a loop like this.

One honest caveat: this is a one-engineer, one-demo-app experiment. A production fleet brings drift — new pages, new failure modes, a moving taxonomy — and the right response to drift is to feed the loop, not to redesign it.

Takeaways

  1. Measure small models before believing them. Ours scored 7% zero-shot on a domain it had never seen — and said so, in its confidences.
  2. Calibrated uncertainty beats confident guessing. The best thing Laya did in this experiment was abstain.
  3. LLMs for the hard cases and the writing; keep them out of the hot path. Escalation-only keeps the bill at cents and the latency at seconds.
  4. Generated fixes get verified and reviewed, always. Swap, re-run, restore — then open the PR.
  5. Ground truth first. Injecting faults into a toy app gave labels for free; your incident history can do the same.
  6. A quantized model that runs is not a model that works. Validate the artifact on real inputs — our dummy-input check passed a bundle whose outputs had gone uniform.
  7. A placeholder gate is a silent tax — fit your thresholds. Re-fitting one routing threshold on the labeled corpus turned twelve correct abstentions into twelve correct decisions, same model, same inputs.

Run it yourself

Everything is in the companion repo: github.com/lucad87/qa-triage-demo

bash
npm install
npm run test:e2e                              # green baseline
node scripts/scenarios/apply.mjs --list      # the fault scenarios (49 and counting)
node scripts/triage/collect.mjs               # rebuild the corpus (Windows)
node scripts/triage/make-finetune-dataset.mjs # build the fine-tune dataset (92 states today)
node scripts/triage/eval.mjs                  # decision-model numbers
node scripts/triage/act.mjs --run runs/<id>   # generate fixes/reports (needs DEEPSEEK_API_KEY; --dry-run without)
gh workflow run e2e-triage.yml                # e2e + triage + fine-tuned-model eval in CI
gh workflow run demo-pipe.yml                 # watchable end-to-end demo (decide -> act -> verify, ~5 min)

Built with Playwright 1.63, Next.js 16, Laya (@receptron/laya 0.1.2), and DeepSeek deepseek-chat.

WRITTEN BY

Luca

Exploring the future of quality assurance and testing automation through deep technical insights.