Continuous integration and continuous deployment are solved problems — in theory. In practice, we found that half our pipelines were either too slow to be useful, too fragile to be trusted, or too clever to be maintained.
July was cleanup month. We audited every pipeline across our active projects and rebuilt the ones that were not earning their keep.
The pipeline audit
We measured three things for each pipeline:
- Time from push to deploy. Anything over ten minutes was flagged.
- Failure rate on the main branch. Failures not caused by actual code problems — flaky tests, timeout issues, dependency resolution errors — were counted separately.
- Recovery time. When a pipeline broke, how long did it take someone to fix it?
The results were sobering. Our average pipeline took fourteen minutes. One took thirty-two. Flaky test failures accounted for twenty percent of all build failures. And when a pipeline broke due to infrastructure issues, the average recovery time was three hours — because nobody fully understood the configuration.
Principles for reliable pipelines
Fast feedback first. The pipeline should tell you if something is obviously wrong within two minutes. Linting, type checking, and unit tests run first. Integration tests and end-to-end tests run later, ideally in parallel.
No mystery steps. Every pipeline step should be runnable locally with the same command. If CI runs npm run test:ci, that command should work on a developer’s machine. Secret-dependent steps are the exception, not the rule.
Cache aggressively. Dependency installation should be cached between runs. Build artifacts that have not changed should be reused. We cut our average build time from fourteen minutes to five by caching node_modules and build outputs properly.
Fail loudly and clearly. When a step fails, the error message should be actionable. “Exit code 1” is not actionable. We added structured error reporting that links directly to the failing test or the lint violation.
The four-stage pipeline
After the cleanup, every project follows the same four-stage structure:
Stage one: Validate. Linting, type checking, and formatting checks. Runs in under a minute. Catches the mistakes you should have caught locally.
Stage two: Test. Unit tests and integration tests. Runs in two to four minutes. Flaky tests are quarantined — they run but their failures do not block the build until they are fixed.
Stage three: Build. Compiles the project and produces deployable artifacts. Runs in one to three minutes depending on the project size.
Stage four: Deploy. Pushes artifacts to the hosting platform. Preview deployments for pull requests, production deployments for the main branch. Runs in under a minute for most static sites.
Total time: five to eight minutes for most projects.
Handling flaky tests
Flaky tests are pipeline poison. They erode trust — developers start ignoring failures, assuming they are flaky, and real bugs slip through.
Our policy: a test that fails intermittently gets quarantined within twenty-four hours. It moves to a separate test suite that runs but does not block deployment. The author has one week to fix it or delete it.
This sounds harsh, but it restored confidence in our pipelines. A red build means a real problem, every time.
Preview deployments changed our review process
Every pull request gets a preview deployment — a live URL with the changes applied. Code reviewers click the link and see the actual result instead of reading diffs and imagining the outcome.
This single change reduced our review cycle time by roughly forty percent. Reviewers catch visual issues, broken layouts, and interaction bugs that are invisible in code review.
The maintenance commitment
A pipeline is not a one-time setup. It requires maintenance — dependency updates, runner version upgrades, cache invalidation adjustments. We schedule thirty minutes per month per project for pipeline maintenance. It is not glamorous work, but it prevents the slow decay that made our old pipelines unreliable.
A pipeline you cannot trust is worse than no pipeline at all. It gives you false confidence. The goal is not automation for its own sake — it is automation you believe in.
