# CI Integration (/docs/ci)



Whether you use the [GitHub Action](/docs/github-action) or run the
[CLI](/docs/cli) directly, `agentiqa run` exposes a **governed, versioned
contract** for CI: a fixed exit-code model and a machine-readable JSON envelope.

To choose *which* plans a pipeline runs and *how* they execute — a labeled subset,
sequential or parallel — see [Labels](/docs/guides/labels) and
[Parallel runs](/docs/cli#parallel-vs-sequential). This page covers how to **gate**
on and **parse** whatever you run.

## Exit codes [#exit-codes]

The CLI exits with a status that gates your job automatically. It's a fixed,
governed contract, so you can rely on it:

| Code | Meaning                                                                                                                                                                                                                                                     |
| ---- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `0`  | Success — all selected plans passed (or there was nothing to run).                                                                                                                                                                                          |
| `1`  | **Plan failure** — plans executed, at least one failed. A real product-quality signal.                                                                                                                                                                      |
| `2`  | **Usage / configuration error** — bad flags, not authenticated, a selector that matched no plans, **a quota / plan-limit block** (account state — retrying won't help), or a batch with **skipped plans** (see below). Something that had to run could not. |
| `3`  | **Infra / runtime error** — engine unreachable, an auth failure, or an unexpected error. Nothing reached a verdict, so it's safe for CI to retry.                                                                                                           |

The `1` vs `3` split is deliberate and stable: `1` is a genuine test failure you
should investigate; `3` is a transient/infra problem that is safe to retry. A gate
can retry on `3` without masking regressions:

**A plan that cannot sign in is skipped, not fatal to the batch (since CLI v1.1.55).**
If a plan's sign-in step declares a credential you have not stored, that one plan is
skipped and every other plan in the batch still runs. The skipped plans are named at
the end of the output and the CLI exits `2` — so the job stays red and you keep the
results of everything that could run. A plain retry cannot clear it: store the named
credential in Project settings → Project credentials and run again. Earlier versions
refused the whole invocation, so one such plan meant the batch tested nothing.

**An automated run draws on the same shared accounts as people.** If the project
uses [shared accounts](/docs/guides/shared-accounts), each running plan leases one
account of the role it signs in as, and returns it when it finishes. A scheduled job
and a teammate can therefore run at the same time — as long as the role has enough
accounts. When they are all busy, the run is refused up front, naming the role.
Give a regularly-scheduled job enough headroom (one account per plan you run
concurrently, plus what your team uses) rather than letting the nightly compete with
the working day.

**A lost connection is not automatically exit `3` (since CLI v1.1.49).** A dropped
connection says nothing about whether the run finished — the engine can record the
final result at the same instant the connection breaks. When a plan's connection ends
before the result arrives, the CLI reads that run's own record and, if a final result
was already recorded, reports **that** result (exit `0` or `1`). Only a run with no
recorded final state stays retryable (exit `3`), so the retry loop below re-runs plans
that genuinely produced nothing — never ones that already finished. Earlier CLI
versions classified every persistent disconnect as exit `3`.

```yaml
- name: Run test plans (retry only on infra/runtime errors)
  run: |
    for attempt in 1 2 3; do
      npx -y agentiqa@latest run --engine https://engine.agentiqa.com
      code=$?
      # 0 = pass, 1 = plan failure (do NOT retry), 2 = usage error (do NOT retry)
      [ "$code" -ne 3 ] && exit "$code"
      echo "Infra/runtime error (exit 3) on attempt $attempt — retrying…"
      sleep 15
    done
    exit 3
```

## JSON envelope [#json-envelope]

Add `--json` (or set `AG_OUTPUT=json`) to emit exactly **one JSON document on
stdout**. Every document carries `"schemaVersion": 1` at the top level; a breaking
change to the shape bumps that number.

* **All logs go to stderr.** stdout is only the JSON document — no banners, no
  progress, no ANSI color. `stdout | jq` is always safe.
* **The exit code is independent of the envelope** — always branch on the exit
  code for pass/fail; use the JSON for detail.

**Success** (`ok: true`):

```json
{
  "ok": true,
  "schemaVersion": 1,
  "outcome": "passed",
  "plans": [
    {
      "title": "Checkout flow",
      "outcome": "passed",
      "durationSec": 42,
      "exitCode": 0,
      "runUrl": "https://web.agentiqa.com/projects/proj_…/test-plans-v2/tp_…/history/run_…",
      "videoUrl": "https://assets.agentiqa.com/e2e-videos/run-…/checkout-flow.mp4"
    },
    { "title": "Login", "outcome": "failed", "durationSec": 18, "exitCode": 1, "summary": "…" }
  ]
}
```

`outcome` is `"passed"` only when every plan passed, otherwise `"failed"`. Each
`plans[]` entry carries a per-plan `outcome`, `durationSec`, `exitCode`, and an
optional `summary`. When artifacts are captured, an entry may also carry `runUrl`,
`videoUrl`, `videoPath`, and `artifactDir` — each present **only** when available
(omitted, never `null`, so `schemaVersion` stays `1`).

`runUrl` is the deep link to the run in the Agentiqa web app. It is set for every
cloud run, and the CLI also prints it to stderr as `[<plan title>] Run: <url>`.
Anyone you send it to opens it with their own Agentiqa login, as a member of the
run owner's organization — there are no public, login-free run links.

**Failure** (`ok: false`) — emitted for usage errors and thrown infra/runtime
errors:

```json
{ "ok": false, "schemaVersion": 1, "error": { "code": "run_error", "message": "…" } }
```

## Gate on the exit code, extract detail with jq [#gate-on-the-exit-code-extract-detail-with-jq]

```yaml
- name: Run test plans (JSON)
  run: |
    npx -y agentiqa@latest run --engine https://engine.agentiqa.com --json > result.json
    code=$?
    jq -r '.plans[] | "\(.outcome)\t\(.title)"' result.json || cat result.json
    exit "$code"
```

Because logs are on stderr, `> result.json` captures only the envelope; the
human-readable run log still streams to the console.

## Action outcome buckets [#action-outcome-buckets]

The [GitHub Action](/docs/github-action) maps these exits to `outcome` buckets for
gating: `0` (with a valid envelope and ≥1 plan) → `passed`; `0` with zero plans or
no valid envelope → `config-error`; `1` → `plan-failure`; `2`/unknown →
`config-error`; `3` → `infra-error`. `fail-on: plan-failure` (default) fails on
plan failures and config errors and swallows retryable infra errors; `fail-on: any`
gates on anything nonzero.