CI/CD Runs & External Signals

Problem we solve

• Teams already run builds, security scans, and tests in GitHub Actions / Jenkins — but the ship story is scattered across tabs.
• You need one place that shows: “Did TestNeo run our suite?” and “Did other checks also finish?”
• Release owners want a simple Also ran in CI line next to Release Readiness — without turning TestNeo into a log warehouse.

What TestNeo does: execute your NLP/API tests from CI (Runs + Report), accept small receipts from any pipeline (External signals), and optionally soft-weight those receipts into the release gate.

Read this first: two pipes

Where in the product

Setup A — Recent Runs (TestNeo executes your suite)

Goal: push to GitHub → see a row like CI_8614… with 10✓ / 0✗ and a Report button.

Setup B — External CI signals (receipt only)

Goal: after CodeQL / image build / foreign Jenkins smoke finishes, TestNeo shows a row under External CI signals and on Release Readiness Also ran in CI.

No tech-stack limit. Anything that can POST JSON with a Bearer API key works: GitHub Actions, Jenkins, GitLab, Azure DevOps, CodeQL, Snyk, Sonar, Trivy, or a shell script.

- name: Tell TestNeo about this check
  if: always()
  env:
    TESTNEO_API_KEY: ${{ secrets.TESTNEO_API_KEY }}
    TESTNEO_SERVER: ${{ vars.TESTNEO_SERVER }}
  run: |
    STATUS=pass
    [ "${{ job.status }}" = "failure" ] && STATUS=fail
    curl -fsS -X POST "$TESTNEO_SERVER/api/web/v1/release-readiness/ci-evidence" \
      -H "Authorization: Bearer $TESTNEO_API_KEY" \
      -H "Content-Type: application/json" \
      -d "{
        \"project_id\": ${{ vars.TESTNEO_PROJECT_ID }},
        \"provider\": \"github_actions\",
        \"kind\": \"security\",
        \"status\": \"$STATUS\",
        \"external_ref\": \"actions/runs/${{ github.run_id }}\",
        \"environment_name\": \"Staging\",
        \"sha\": \"${{ github.sha }}\",
        \"summary\": \"CodeQL finished with status $STATUS\",
        \"details_url\": \"${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}\",
        \"payload\": { \"source\": \"codeql\" }
      }"

Evidence field reference

Tip: If TestNeo already ran the suite, use Recent Runs → Report. Do not also Record signal for the same suite unless you intentionally want a duplicate billboard.

How to read /web/ci

1. Recent Runs (top) — TestNeo jobs. Click Report for per-test detail.
2. “How to read this page” — short reminder of the two pipes.
3. External CI signals (bottom) — receipts. Click a row for detail; Open CI console only if details_url is a real external URL.

Filter Failed on Runs to focus on broken TestNeo jobs. External fails/warns show as status badges in the signals table.

How to read a Report

From Recent Runs → Report opens /web/ci/report/CI_….

• Use this for which tests passed/failed, duration, and triage of a TestNeo execution.
• For trends across many days, use Analytics — not the External signals panel.
• External signal detail will never replace Report unless that foreign system posted rich payload counts (still no TestNeo step video).

Not observability

External signals are a release billboard, not Datadog / logs / APM.

Deep analysis stays in CodeQL / Jenkins / your observability tool via details_url. TestNeo answers: “did that check also finish for this ship?”

Optional soft gate

By default, external receipts are display-first — they do not change confidence.

• Open Release Readiness → select a bundle → check Soft-weight external CI → Evaluate gate.
• Critical fails (e.g. smoke/security/e2e fail) can demote PASS; security warn can demote to WARN.
• Full gate policies (BLOCK PRs, min confidence): see Release Intelligence · CI/CD Gate.

FAQ & troubleshooting

Related documentation

Last updated for DevOps control-plane v1 · display-first external evidence · soft gate opt-in on Release Readiness.