CI Integration

Whether you use the GitHub Action or run the CLI directly, agentiqa run exposes a governed, versioned contract for CI: a fixed exit-code model and a machine-readable JSON envelope.

To choose which plans a pipeline runs and how they execute — a labeled subset, sequential or parallel — see Labels and Parallel runs. This page covers how to gate on and parse whatever you run.

Exit codes

The CLI exits with a status that gates your job automatically. It's a fixed, governed contract, so you can rely on it:

The 1 vs 3 split is deliberate and stable: 1 is a genuine test failure you should investigate; 3 is a transient/infra problem that is safe to retry. A gate can retry on 3 without masking regressions:

A plan that cannot sign in is skipped, not fatal to the batch (since CLI v1.1.55). If a plan's sign-in step declares a credential you have not stored, that one plan is skipped and every other plan in the batch still runs. The skipped plans are named at the end of the output and the CLI exits 2 — so the job stays red and you keep the results of everything that could run. A plain retry cannot clear it: store the named credential in Project settings → Project credentials and run again. Earlier versions refused the whole invocation, so one such plan meant the batch tested nothing.

An automated run draws on the same shared accounts as people. If the project uses shared accounts, each running plan leases one account of the role it signs in as, and returns it when it finishes. A scheduled job and a teammate can therefore run at the same time — as long as the role has enough accounts. When they are all busy, the run is refused up front, naming the role. Give a regularly-scheduled job enough headroom (one account per plan you run concurrently, plus what your team uses) rather than letting the nightly compete with the working day.

A lost connection is not automatically exit 3 (since CLI v1.1.49). A dropped connection says nothing about whether the run finished — the engine can record the final result at the same instant the connection breaks. When a plan's connection ends before the result arrives, the CLI reads that run's own record and, if a final result was already recorded, reports that result (exit 0 or 1). Only a run with no recorded final state stays retryable (exit 3), so the retry loop below re-runs plans that genuinely produced nothing — never ones that already finished. Earlier CLI versions classified every persistent disconnect as exit 3.

- name: Run test plans (retry only on infra/runtime errors)
  run: |
    for attempt in 1 2 3; do
      npx -y agentiqa@latest run --engine https://engine.agentiqa.com
      code=$?
      # 0 = pass, 1 = plan failure (do NOT retry), 2 = usage error (do NOT retry)
      [ "$code" -ne 3 ] && exit "$code"
      echo "Infra/runtime error (exit 3) on attempt $attempt — retrying…"
      sleep 15
    done
    exit 3

JSON envelope

Add --json (or set AG_OUTPUT=json) to emit exactly one JSON document on stdout. Every document carries "schemaVersion": 1 at the top level; a breaking change to the shape bumps that number.

• All logs go to stderr. stdout is only the JSON document — no banners, no progress, no ANSI color. stdout | jq is always safe.
• The exit code is independent of the envelope — always branch on the exit code for pass/fail; use the JSON for detail.

Success (ok: true):

{
  "ok": true,
  "schemaVersion": 1,
  "outcome": "passed",
  "plans": [
    {
      "title": "Checkout flow",
      "outcome": "passed",
      "durationSec": 42,
      "exitCode": 0,
      "runUrl": "https://web.agentiqa.com/projects/proj_…/test-plans-v2/tp_…/history/run_…",
      "videoUrl": "https://assets.agentiqa.com/e2e-videos/run-…/checkout-flow.mp4"
    },
    { "title": "Login", "outcome": "failed", "durationSec": 18, "exitCode": 1, "summary": "…" }
  ]
}

outcome is "passed" only when every plan passed, otherwise "failed". Each plans[] entry carries a per-plan outcome, durationSec, exitCode, and an optional summary. When artifacts are captured, an entry may also carry runUrl, videoUrl, videoPath, and artifactDir — each present only when available (omitted, never null, so schemaVersion stays 1).

runUrl is the deep link to the run in the Agentiqa web app. It is set for every cloud run, and the CLI also prints it to stderr as [<plan title>] Run: <url>. Anyone you send it to opens it with their own Agentiqa login, as a member of the run owner's organization — there are no public, login-free run links.

Failure (ok: false) — emitted for usage errors and thrown infra/runtime errors:

{ "ok": false, "schemaVersion": 1, "error": { "code": "run_error", "message": "…" } }

Gate on the exit code, extract detail with jq

- name: Run test plans (JSON)
  run: |
    npx -y agentiqa@latest run --engine https://engine.agentiqa.com --json > result.json
    code=$?
    jq -r '.plans[] | "\(.outcome)\t\(.title)"' result.json || cat result.json
    exit "$code"

Because logs are on stderr, > result.json captures only the envelope; the human-readable run log still streams to the console.

Action outcome buckets

The GitHub Action maps these exits to outcome buckets for gating: 0 (with a valid envelope and ≥1 plan) → passed; 0 with zero plans or no valid envelope → config-error; 1 → plan-failure; 2/unknown → config-error; 3 → infra-error. fail-on: plan-failure (default) fails on plan failures and config errors and swallows retryable infra errors; fail-on: any gates on anything nonzero.