A test that was weakened to pass is not a pass.

After signing in, go to /tasks/new.
Fill 'Task title' with 'QA smoke task'.
Click 'Save task'.
Verify the text 'Task created' appears.

def test_task_create(page: Page):
    login(page)
    page.goto(f"{BASE_URL}/tasks/new")
    page.get_by_label("Task title").fill("QA smoke task")
    page.get_by_role("button", name="Save task").click()
    expect(page.get_by_text("Task created")).to_be_visible()

Before that test executes, a 44-rule sanitizer repairs the mistakes language models make writing Playwright. When a repair relaxes an assertion, the run is recorded as weakened rather than passed.

1. 1SupervisorReads the requirement and routes it.
2. 2GeneratorWrites Gherkin, then Playwright code.
3. 3ExecutorRuns it in a real browser, heals locators.
4. 4TriageRoot-causes failures, finds duplicates.
5. 5RegressionPicks the suites the change touches.
6. 6ReporterEmits JUnit, the verdict, the exit code.

pass

The assertion the requirement asked for ran, and held.

pass (weakened)

The sanitizer relaxed an assertion so the test could run. It went green, but it no longer proves what you asked. Policy can block the build on this alone.

fail

The assertion ran and did not hold. The reason is named, not summarised.

did not run

The stage never executed. It is never painted green to fill the row.

product: taskdesk
environments:
  staging: https://staging.example.internal
auth:
  username_ref: TASKDESK_USER   # env var name, never a password
  password_ref: TASKDESK_PASS
flows:
  - id: task-create
    text: "After signing in, go to /tasks/new. Fill 'Task title'
           with 'QA smoke task'. Click 'Save task'. Verify the
           text 'Task created' appears."
    tags: [regression, feature]
policy:
  block_on: [failed, weakened]
  min_pass_rate: 100

1. Live test

   Describe one check in plain English. The engine writes it, runs it, and returns a verdict in about a minute. For trying an idea.

2. Baseline suite

   Hand it the manifest. Every flow becomes a generated, executed and reviewed test. What passes review becomes your approved suite.

3. CI gate

   The approved suite replays on every change. Nothing is regenerated, so nothing drifts, and a blocked build names its reasons.

What does "pass (weakened)" mean?

Before a generated test runs, a 44-rule sanitizer repairs the mistakes language models make when writing Playwright code. Some of those repairs relax an assertion to let the test execute. When that happens the result is recorded as pass (weakened) and the relaxed assertion is named. It is never counted as a clean pass, and your gate policy can block the build on weakened passes alone.

How do you know the generated tests are real and not always-green?

By mutation testing. A live application was deliberately broken five times: a renamed heading, changed seed data, a renamed selector. The approved suite replayed against each mutation. Every mutation produced exactly the expected failure, named the exact broken assertion, and left unrelated flows passing. Restoring the application restored the green run.

What happens if the application under test is unreachable?

The gate exits 2 with an infra_error verdict. An unreachable target is never reported as a test failure, because a broken environment and a broken feature call for different responses from whoever is on call.

Am I locked into the platform?

No. Every approved test exports as plain pytest plus Playwright and runs anywhere Python does. The CI gate verdict is a pure function over coverage metrics and your policy, and it reports through standard JUnit XML and process exit codes.

Which exit codes does the CI gate use?

Exit 0 when the suite passes your policy. Exit 1 when the gate blocks the build, with every blocking reason named. Exit 2 when the run could not be completed for infrastructure reasons.

Does it need my application’s source code?

No. Flows are described in plain English in a qa.yaml manifest and executed against a running environment through a real browser. Credentials are referenced by environment variable name and never stored in the manifest.