Database Migrations in Production: The Strategies That Don't Break Deployments

A 30-second database migration once delayed an entire production deployment.

Not because the migration was wrong.

Because it locked a table that was serving live traffic.

That incident changed how we approach database migrations in Rails.

Early in a project, migrations feel simple.

Add a column.
Run deploy.
Move on.

But production systems behave differently.

A migration that runs in seconds on a local machine can impact thousands of users when the table contains millions of records.

One migration we reviewed was going to update data across a table with more than 15 million rows.

Running it directly during deployment would have increased deployment time significantly and introduced unnecessary risk.

Instead, we broke the process into stages:
* Deploy schema changes first
* Backfill data using background jobs
* Roll out application changes separately
* Remove old columns only after verification

The result:
* Zero downtime deployment
* No customer impact
* Predictable release process

The biggest lesson?

Database migrations are not just database changes.

They are production events.

The safest migrations are usually the ones users never notice happened.