API testing in CI/CD: gate every deploy

Which tests belong at which stage

The organising principle is that fast, deterministic tests run often, and slow or noisy ones run on a schedule. Putting a 40-minute load test on the pull-request path guarantees it gets disabled.

Keeping the pipeline under ten minutes

Ten minutes is roughly the limit of a developer's willingness to wait. Beyond it they start another task, and the feedback loop that makes CI valuable is broken.

• Run tests in parallel. API tests are independent by nature if the data strategy is right — the main reason suites can't parallelise is shared mutable state.
• Shard by endpoint group. Four shards of 100 tests finish in roughly a quarter of the time of 400 sequential ones.
• Fail fast on smoke. Run a 30-second smoke subset first; if authentication is broken there is no value in running 400 more tests.
• Cache dependencies. Installing packages repeatedly often costs more than the tests.
• Move slow suites off the commit path. Load and fuzz belong on a schedule.

Post-deploy smoke tests

The tests that ran before deployment validated a build artefact. They said nothing about whether the deployment itself succeeded — whether configuration was applied, migrations ran, secrets resolved and the load balancer picked up the new instances.

A post-deploy smoke suite hitting the real environment is the only thing that verifies this, and it should be small enough to complete in under two minutes: authenticate, read the primary resource, exercise one write path, check the health endpoint. Wire its failure to an automatic rollback, and a bad deploy is reverted before most users encounter it.

Handling secrets

• Use the CI provider's secret store and reference secrets as masked environment variables. Never commit a token, even privately — repositories get forked and history is permanent.
• Use a dedicated test account with least privilege. It should not be able to touch real customer data.
• Rotate on a schedule and immediately if a log ever exposes one.
• Point at staging by default. If tests must hit production, restrict them to read-only smoke checks.
• Be careful with pull requests from forks. Most CI systems withhold secrets from fork builds by design — an attacker who can run arbitrary code in your pipeline can print your environment.

Blocking versus warning

A gate that blocks too aggressively gets bypassed; one that never blocks is decoration. The distinction that works in practice is determinism:

• Block on functional, contract and smoke failures. These are deterministic — a failure means something is genuinely broken.
• Warn on performance regressions unless they cross a threshold. Latency measurements are noisy, and a build that fails on ordinary variance trains people to re-run until green.
• Alert on new security findings while allowing the existing backlog through. A gate failing from the first day is disabled by the second.

For performance, express the rule as a budget against a rolling baseline — for instance, fail when p95 latency regresses more than 20% against the previous release. That catches genuine regressions without halting delivery on noise.

A worked example

A typical GitHub Actions arrangement:

on:
  pull_request:          # functional + contract, 4 shards, ~4 min
  push:
    branches: [main]     # + regression and integration, ~12 min
  deployment_status:     # smoke against the deployed environment, ~90s
  schedule:
    - cron: '0 2 * * *'  # nightly load, fuzz and security

The pull-request job needs only a staging base URL and a masked test-account token. The post-deploy job reads the environment URL from the deployment event, so the same workflow validates every environment without duplication.