A pipeline has one job: tell you quickly and honestly whether your change is safe to ship. Most of the ones we have inherited fail at "quickly," and once feedback takes thirty minutes, developers batch up changes, and big batches are exactly what makes deploys risky. Speed and safety reinforce each other.
Order stages by how fast they fail
We run the cheapest, most likely-to-fail checks first. Compilation and unit tests in the first two minutes, before anything slow starts. There is no reason to spin up Testcontainers and run the full integration suite if the code does not even compile. On GitHub Actions we split this across jobs so a fast failure stops the expensive work.
- Stage 1: compile and unit tests - under two minutes, catches most mistakes
- Stage 2: integration tests with Testcontainers - runs only if stage 1 passes
- Stage 3: build and push the image - only on the main branch
- Cache dependencies aggressively; a cold Maven build is a self-inflicted wound
Safe deploys mean small and reversible
The safest deploy is a small one you can undo. We deploy on every merge to main, which keeps each change tiny, and we keep the previous image one command away. Database migrations are the exception that breaks rollback, so we make them backward compatible: add a column before the code that uses it, drop the old one a release later, never both at once.
If your rollback plan is "deploy forward fast," you do not have a rollback plan, you have hope.
Health checks gate the rollout
A new version does not take traffic until it says it is ready. Spring Boot Actuator gives us readiness and liveness endpoints, and the orchestrator waits on readiness before routing requests. This catches the boring failures - a missing environment variable, a database it cannot reach - before any user sees them.
We resisted automatic rollback for a long time, worried it would mask real problems. We were wrong. Tying rollback to a clear error-rate threshold means a bad deploy is reverted in under a minute while we sleep, and the post-mortem happens in the morning with the system already healthy.