Most teams achieve zero-downtime application deploys and then take an outage anyway, because a migration locked a table or removed a column the previous version still referenced. The application layer is the solved part; the schema is where the discipline is required.
Two versions run simultaneously. Always.
During any rolling deploy, old and new code serve traffic at the same time. Every change must therefore be compatible in both directions: the new code must work with the old schema, and the old code must survive the new one. Internalising this single constraint prevents most deployment incidents.
During a rolling deploy, both versions are live. Every change must be safe in both directions.
Expand, migrate, contract
Renaming a column in one step breaks the running version. Instead: add the new column, write to both, backfill, switch reads, stop writing the old, then drop it — across several releases. It feels laborious and it is the difference between a routine change and a 3am rollback.
Locks are the thing that bites
Adding a column with a default, adding a NOT NULL constraint, or creating an index can take an exclusive lock and stall every query behind it. In PostgreSQL, CREATE INDEX CONCURRENTLY and adding constraints as NOT VALID then validating separately avoid the worst of it. Test migrations against production-sized data, because a migration that takes 20ms on a dev database can take 20 minutes on the real one.
Backfill in batches, off the deploy path
Never backfill millions of rows inside a migration that blocks a deployment. Run it as a separate, resumable, rate-limited job that can be paused if it affects production load. Coupling data movement to deployment turns a slow backfill into an outage.
Health checks that mean something
A health endpoint returning 200 because the process started tells you nothing. It should verify the things that make an instance able to serve: database reachable, migrations at the expected version, critical dependencies responding. Otherwise the orchestrator routes traffic to instances that will fail every request.
Rehearse the rollback
A rollback path that has never been executed is a hypothesis. Practise it in staging until it is boring, and be explicit that forward-only is the correct choice for some schema changes — knowing which is which, before the incident, is the point of the exercise.
