CI Integration

Whether you use the GitHub Action or run the CLI directly, agentiqa run exposes a governed, versioned contract for CI: a fixed exit-code model and a machine-readable JSON envelope.

To choose which plans a pipeline runs and how they execute — a labeled subset, sequential or parallel — see Labels and Parallel runs. This page covers how to gate on and parse whatever you run.

Exit codes

The CLI exits with a status that gates your job automatically. It's a fixed, governed contract, so you can rely on it:

The 1 vs 3 split is deliberate and stable: 1 is a genuine test failure you should investigate; 3 is a transient/infra problem that is safe to retry. A gate can retry on 3 without masking regressions:

A plan that cannot sign in is skipped, not fatal to the batch (since CLI v1.1.55). If a plan's sign-in step declares a credential you have not stored, that one plan is skipped and every other plan in the batch still runs. The skipped plans are named at the end of the output and the CLI exits 2 — so the job stays red and you keep the results of everything that could run. A plain retry cannot clear it: store the named credential in Project settings → Project credentials and run again. Earlier versions refused the whole invocation, so one such plan meant the batch tested nothing.

An automated run draws on the same shared accounts as people. If the project uses shared accounts, each running plan leases one account of the role it signs in as, and returns it when it finishes. A scheduled job and a teammate can therefore run at the same time — as long as the role has enough accounts. When they are all busy, the run is refused up front, naming the role. Give a regularly-scheduled job enough headroom (one account per plan you run concurrently, plus what your team uses) rather than letting the nightly compete with the working day.

A lost connection is not automatically exit 3 (since CLI v1.1.49). A dropped connection says nothing about whether the run finished — the engine can record the final result at the same instant the connection breaks. When a plan's connection ends before the result arrives, the CLI reads that run's own record and, if a final result was already recorded, reports that result (exit 0 or 1). Only a run with no recorded final state stays retryable (exit 3), so the retry loop below re-runs plans that genuinely produced nothing — never ones that already finished. Earlier CLI versions classified every persistent disconnect as exit 3.

- name: Run test plans (retry only on infra/runtime errors)
  run: |
    for attempt in 1 2 3; do
      npx -y agentiqa@latest run --engine https://engine.agentiqa.com
      code=$?
      # 0 = pass, 1 = plan failure (do NOT retry), 2 = usage error (do NOT retry)
      [ "$code" -ne 3 ] && exit "$code"
      echo "Infra/runtime error (exit 3) on attempt $attempt — retrying…"
      sleep 15
    done
    exit 3

Exit 2: account and plan limits

When the account behind the run cannot start one, agentiqa run exits 2 before any plan starts. With --json, the failure envelope's error.code names the block:

Any other code that names a plan or quota limit behaves the same way: it is exit 2, a retry cannot clear it, and error.message says what has to change.

Exit 3: infrastructure, not a verdict

Exit 3 says nothing about your app: the run could not reach a verdict for reasons on the infrastructure side. With --json, each affected plan's outcome says why:

If the run never started (for example, the engine was unreachable), there are no plans at all: the failure envelope's error.code says why — run_error, or a specific reason such as plan_run_active when another run of the same plan was still in progress.

Retrying is safe, but check the run before you re-run it:

1. agentiqa runs get <runId> shows the interrupted run — the runId is in the plan's JSON entry even on exit 3 — and agentiqa runs get <plan-id> lists the plan's newest runs. If the interrupted run reached a final result after the connection dropped, use that result instead of re-running. A run that still shows as running long after the CLI exited was cut off — treat it as unfinished.
2. A run that stopped part-way may already have created data in the app under test — a record, an upload, a sign-up. Clean it up, or make sure the plan tolerates it, before you run the plan again. Dynamic values help here: each run names what it creates differently.

JSON envelope

Add --json (or set AG_OUTPUT=json) to emit exactly one JSON document on stdout. Every document carries "schemaVersion": 1 at the top level; a breaking change to the shape bumps that number.

• All logs go to stderr. stdout is only the JSON document — no banners, no progress, no ANSI color. stdout | jq is always safe.
• The exit code is independent of the envelope — always branch on the exit code for pass/fail; use the JSON for detail.

Success (ok: true):

{
  "ok": true,
  "schemaVersion": 1,
  "outcome": "passed",
  "plans": [
    {
      "title": "Checkout flow",
      "outcome": "passed",
      "durationSec": 42,
      "exitCode": 0,
      "runId": "run_…",
      "runUrl": "https://web.agentiqa.com/projects/proj_…/test-plans-v2/tp_…/history/run_…",
      "videoUrl": "https://assets.agentiqa.com/e2e-videos/run-…/checkout-flow.mp4"
    },
    { "title": "Login", "outcome": "failed", "durationSec": 18, "exitCode": 1, "summary": "…" }
  ]
}

outcome is "passed" only when every plan passed, otherwise "failed". Each plans[] entry carries a per-plan outcome, durationSec, exitCode, and an optional summary. When artifacts are captured, an entry may also carry runUrl, videoUrl, videoPath, and artifactDir — each present only when available (omitted, never null, so schemaVersion stays 1).

runUrl is the deep link to the run in the Agentiqa web app. It is set for every cloud run, and the CLI also prints it to stderr as soon as the run starts, as [<plan title>] Run: <url> (run_id: run_…). Anyone you send it to opens it with their own Agentiqa login, as a member of the run owner's organization — there are no public, login-free run links.

runId is the run's id. It is reported as soon as the run starts, so it is present even when a plan ends engine_disconnect or engine_failure (exit 3); runUrl is too, on hosted cloud runs. After an engine_disconnect the run may still finish on the server: check it with agentiqa runs get <runId> before re-running. The CLI prints the batch page link (Batch: <url>) once, when the batch starts.

Failure (ok: false) — emitted for usage errors and thrown infra/runtime errors:

{ "ok": false, "schemaVersion": 1, "error": { "code": "run_error", "message": "…" } }

Gate on the exit code, extract detail with jq

- name: Run test plans (JSON)
  run: |
    npx -y agentiqa@latest run --engine https://engine.agentiqa.com --json > result.json
    code=$?
    jq -r '.plans[] | "\(.outcome)\t\(.title)"' result.json || cat result.json
    exit "$code"

Because logs are on stderr, > result.json captures only the envelope; the human-readable run log still streams to the console.

Action outcome buckets

The GitHub Action maps these exits to outcome buckets for gating: 0 (with a valid envelope and ≥1 plan) → passed; 0 with zero plans or no valid envelope → config-error; 1 → plan-failure; 2/unknown → config-error; 3 → infra-error. fail-on: plan-failure (default) fails on plan failures and config errors and swallows retryable infra errors; fail-on: any gates on anything nonzero.