Who owns the schema? Database migrations when many services share one database

A quick test first: how many versions of your application are running in production right now?

If you said "one", check again. A Kubernetes rolling update keeps old and new pods side by side for minutes. A canary release does it on purpose for hours. The CronJob that builds a monthly report will start from an image that is three weeks old. Add the forgotten export script at a partner's site. The code exists in many versions at once. The database always has exactly one.

That is the heart of the problem. When several deployments — a backoffice, an API, background workers, scheduled jobs — share one database, every schema migration changes a contract with many parties at once, and some of them do not yet know the contract is changing. Below we describe a way of working in which data evolution and code evolution are one process, not two that somebody keeps in sync by hand. We use Flyway, but the principles carry over to Liquibase, Django migrations, Alembic or whatever your framework ships with.

By hand? Not with two databases

Before getting to many services, a simpler question: why versioned migrations at all?

Two copies of the same data — test and production — are enough to make manual schema management fall apart. Someone adds an index in production "because it was slow", someone else changes a column type in test "just to try". Six months later the environments differ in ways nobody can describe, and tests passing on something that is not a copy of production prove nothing.

A tool like Flyway turns schema history into an ordered sequence of files — V1__…, V2__… — kept in the repository next to the code and applied exactly once, in a fixed order. The benefits go beyond the usual list:

• Reproducible environments. A new environment is an empty database plus migrate — a deterministic sequence, not "a production dump and some manual fixes".
• The schema gets reviewed. A change to the data structure lands in the same merge request as the code that needs it. The reviewer sees both sides of the contract.
• An audit trail. flyway_schema_history records which migration ran, when, with which checksum and whether it succeeded. "Since when does this column exist?" becomes a query, not archaeology.
• Drift detection. Checksums catch migration files edited after the fact; validate catches a mismatch between what the code knows and what the database holds.
• Data migrations, not only DDL. Moving values to a new column, normalising, filling dictionaries — versioned like the structure itself.
• Tests on the real schema. The same migrations build the database in integration tests (e.g. Testcontainers), so tests run against the schema that will actually reach production.
• Onboarding. The history of the data model reads like a changelog. Nobody has to ask "where did this table come from?"
• A predictable direction. Migrations move forward. A mistake is fixed by the next migration, not by a manual "undo" in production.

Without this, repeated deployments across several environments are hard to imagine. With many services on one database, though, migrations are only the starting point.

Thesis one: exactly one service migrates

The simplest setup — every service runs Flyway migrate on startup — looks harmless. Flyway takes a lock while migrating, so two replicas will not run the same script concurrently. The problem lies elsewhere: who decides the schema, and in which version?

If every service runs migrations, the database ends up in whatever state the first service to start wanted. Deploying a new API version changes the schema under a backoffice that knows nothing about it. A hotfix from an old branch of another service may try to add a migration in a slot already taken on the main line. Responsibility blurs, and predictability goes with it.

Hence the rule: there is exactly one "migration master". Only it runs migrate. Every other deployment runs no migrations — at most it checks that the database is in the state it expects.

Who should it be? In our experience: the deployment with the administrative UI, the backoffice. It changes along with the data model anyway (every new field has to be editable somewhere), it has the least external traffic, and its brief unavailability during a longer migration hurts the least. API clients are not left waiting for an ALTER TABLE to finish.

The alternative is a separate Kubernetes Job (e.g. a Helm hook that runs before the deployments are updated) or an initContainer. That works well when no application is the natural owner of the model. The rule stays the same: one executor, one line of history.

01 Deployment timeline of the four expand/contract steps for the backoffice, the API and a monthly CronJob

Thesis two: everyone else checks — and fails immediately

If only the master migrates, what do the others do? On startup they check that the schema version matches what their code expects. If it does not, they exit with an error. Immediately, with no attempt to "cope somehow".

Sounds harsh? In Kubernetes it is the safest thing a pod can do. A new version that does not get through startup never reports readiness, so it never receives traffic. The rolling update stalls and the old version keeps serving users exactly as before the deployment. A failing new version is not a disaster. It is a fuse.

Flyway has the tool for it: validate. It compares the migrations the application knows (on its classpath) with the history table. The behaviour, which we verified on Flyway 10 and PostgreSQL 17, is exactly what we need:

• New code, old database — the code knows a migration the database lacks. validate fails with Detected resolved migration not applied to database. The pod exits. Good: the master has not migrated yet.
• Old code, new database — the database has a migration the code does not know. The default ignoreMigrationPatterns is *:future, so "future" migrations are ignored and validate passes. Also good: that is precisely the situation we want to allow during the expand phase.

This asymmetry is crucial and rarely mentioned. If someone cleared ignoreMigrationPatterns "to be strict", old code would fail after every master migration — and a rolling update would turn into an outage.

In Spring Boot you can keep Flyway enabled everywhere and, outside the master, swap the migrate strategy for validate:

1@Configuration
2@Profile("!migration-master")
3class SchemaVersionCheck {
4
5    // This service never migrates. It only checks that
6    // the database has every migration its code knows.
7    // Anything missing = exception on startup = fail fast.
8    @Bean
9    FlywayMigrationStrategy validateOnly() {
10        return flyway -> flyway.validate();
11    }
12}

The deployment order follows: master first, everything else after. If the pipeline gets it backwards, nothing bad happens — the new pods of the other services fail, the old ones keep running, and once the master is deployed a rollout restart is enough. Fail fast turns an ordering mistake from an incident into a delay.

Thesis three: migrations live in one place

Many deployments usually mean many projects or subprojects. And the temptation follows: each team keeps the migrations for "its" tables. That is the shortest route to nobody knowing the full schema history, version numbers colliding, and the order depending on who deployed first.

Migrations must not be scattered. They have to be collected systematically, in one place, on one version line.

What works well for us is a shared module that holds the migrations together with the data-access layer: entities, object-relational mapping, repositories. Every service links it and depends on it. The consequences are very practical:

• the module version is also the schema version the service understands, so validate has something to compare against,
• an entity change and the migration that enables it sit in the same commit and the same review,
• a service that has not bumped the dependency simply keeps seeing the old model — and, thanks to the tolerance of "future" migrations, keeps working.

To be fair about the price: a shared data module is deliberate coupling between services. It is not microservices by the textbook ("every service owns its database"). But in many systems a shared database is a fact — historical, cost-driven, or required for transactional consistency. Given that it exists, explicit, versioned coupling beats the same classes copied into five repositories.

Thesis four: the database must be sealed

One "quick ALTER TABLE" on a Friday evening, run by hand in production because "a migration is too much ceremony for one index". What happens next? A week later the migration that creates the same index fails in production and passes in test. Six months later nobody knows why test is not production.

Once you commit to automated migrations, manual DDL access has to be taken away. Not with a wiki page — with privileges:

1-- migrator: the only role that changes structure
2CREATE ROLE app_migrator LOGIN;
3GRANT CREATE, USAGE ON SCHEMA app TO app_migrator;
4
5-- applications: data only, no DDL
6CREATE ROLE app_runtime LOGIN;
7GRANT USAGE ON SCHEMA app TO app_runtime;
8ALTER DEFAULT PRIVILEGES FOR ROLE app_migrator IN SCHEMA app
9  GRANT SELECT, INSERT, UPDATE, DELETE ON TABLES TO app_runtime;
10
11-- people: read only; changes only through migrations
12CREATE ROLE app_readonly LOGIN;
13GRANT USAGE ON SCHEMA app TO app_readonly;
14ALTER DEFAULT PRIVILEGES FOR ROLE app_migrator IN SCHEMA app
15  GRANT SELECT ON TABLES TO app_readonly;

Add spring.jpa.hibernate.ddl-auto=none (more on that under anti-patterns) and a periodic validate on every environment as a drift detector. Sealing the database is not bureaucracy. Without it the whole migration history is fiction.

Expanding is easy. Contracting is another conversation

Now the interesting part. Most schema changes are expansions: a new table, a new column, a new index. A well-designed expansion is backward compatible — old code does not notice it, because it never asks for the new columns. That is why the master can migrate while other services are updated at their own pace.

Changes that narrow the schema are harder: dropping a column, changing its type, renaming it, splitting a table. Old code that still reads customer.legacy_code will simply break once the column is gone. Before such a migration we need to be sure that every consumer of the old structure has been updated.

Who is "every"? Services deployed daily — easy to check. But also:

• a CronJob that runs once a month, whose pod nobody has seen for three weeks,
• a BI report with its own SQL query,
• a partner integration that reads a single view,
• a script someone "temporarily" runs from a laptop before quarter-end.

The pattern that brings order to such changes is called parallel change, or expand/contract. A narrowing change is split into steps, each safe for every running version of the code:

1. Expand — add the new structure next to the old one (a new column, a new table). The master migrates, nothing breaks.
2. Migrate — code writes to both and reads from the new one; historical data is moved by a separate, batched backfill. Services move to the new version one by one.
3. Contract — once nobody uses the old structure, remove it.

So renaming a column is really three releases, not a single RENAME. A lot of work? Yes. The alternative is a maintenance window and the hope that no consumer was forgotten.

Can anything block a contract while old code is alive?

Worth asking directly, because intuition says such a safeguard "must exist somewhere". We checked.

Flyway itself does not know who reads the database. flyway_schema_history records what was applied, not who depends on it. validate protects new code from an old database, but not old code from a migration that takes something away. What Flyway does provide is hooks:

• Callbacks (beforeMigrate, beforeEachMigrate and others), in SQL or Java. An error in a callback aborts the migration — we verified this experimentally. The Java API receives information about the migration about to run, so it can react only to selected ones.
• The migration itself can start with a condition that raises an error. In PostgreSQL DDL is transactional, so an aborted migration leaves nothing half-done.

Liquibase's equivalent is preconditions — for example sqlCheck with onFail="HALT" stops the whole changelog if the query does not return the expected value. Same mechanism, different packaging: the tool can be stopped, but it does not know when to stop.

pgroll for PostgreSQL goes furthest. There, expand/contract is an operation of the tool itself: pgroll start creates a new schema version as a set of views over the physical tables, with triggers carrying writes between the old and new columns. Old code sees the old schema, new code the new one (chosen through search_path). Old structures are removed only by an explicit pgroll complete, run once nobody uses the previous version. The question "is anyone still on the old version?" remains ours to answer.

The conclusion: the missing piece is not a tool, it is knowledge about consumers. And a simple local pattern can provide it.

Pattern: a schema consumer registry

Every participant — service, CronJob, worker — records on startup (and periodically afterwards) who it is and which schema version it understands:

1CREATE TABLE schema_consumers (
2  consumer       text PRIMARY KEY,  -- e.g. 'api', 'monthly-report'
3  schema_version integer NOT NULL,  -- known to the code
4  last_seen      timestamptz NOT NULL
5);
6
7-- on startup of every service and every CronJob run
8INSERT INTO schema_consumers
9  (consumer, schema_version, last_seen)
10VALUES ('monthly-report', 5, now())
11ON CONFLICT (consumer) DO UPDATE
12  SET schema_version = EXCLUDED.schema_version,
13      last_seen = now();

A narrowing migration starts with a guard that checks the registry over a window longer than the period of the least frequent job:

1-- V6__contract_drop_customer_legacy_code.sql
2DO $$
3BEGIN
4  IF EXISTS (SELECT 1 FROM schema_consumers
5             WHERE schema_version < 5
6               AND last_seen > now() - interval '35 days') THEN
7    RAISE EXCEPTION 'contract blocked: old code still active';
8  END IF;
9END $$;
10
11ALTER TABLE customer DROP COLUMN legacy_code;

We tested this scenario on Flyway 10 and PostgreSQL 17. While the registry holds a consumer with an older version, migrate fails and the column stays untouched. Once the last consumer is updated, the same migration goes through. In MySQL, where DDL is not transactional and there are no DO blocks, put the guard in a Java callback or a Java migration — the rule is the same.

Two additions turn this from a trick into a practice:

• A naming convention. Narrowing migrations carry contract in their name, and the pipeline requires manual approval for them. Expansions go through automatically; contractions go through deliberately.
• The time window is a decision, not a constant. 35 days for a monthly cron — but what about the yearly report? The registry shows who checked in; it cannot show who never did.

And here the real questions begin. What about consumers outside our control — a BI tool, a partner? Make them register, or give them a stable view as an explicit API? How much trust do we put in a time window? We have no single answer and would like to hear yours.

Dead souls

The simplest fix for the contract problem? Don't contract. Leave legacy_code there forever; it bothers nobody.

Are you sure? Every dead column is false information about the domain model. A new team member sees it and assumes it means something. A report counts it. Someone starts writing to it "because it's there". A cluttered schema is like documentation with stale chapters: the more complete it looks, the more dangerous it is.

In our view the schema should be current in its entirety, and dead data contradicts that. The counterpoint is obvious: contracting costs effort and carries risk, and the gain from dropping one column can be illusory. Our compromise: contract as scheduled debt with a due date. The expand lands in the repository together with a "remove the old structure" task and the date by which the consumer registry should be clean. Not "some day" — a specific day.

A gallery of anti-patterns

Some have already come up. We gather them here because in practice they rarely come alone.

Every service migrates on its own. Whoever starts first decides the schema. Cure: one master, everyone else validate.

Migrations scattered across repositories. Colliding numbers, no complete history. Cure: one module with migrations and the data-access layer.

Quick manual DDL. Environments stop being copies. Cure: privileges, not good intentions.

Contract before the last consumer. The monthly report dies on the night between the 31st and the 1st. Cure: a consumer registry and a guard in the migration.

Dead souls with no deadline. Cure: debt with a due date.

And now the ones people mention less often.

An expand that isn't

"We only added a column." And yet the old code breaks. How? A few scenarios:

• the new column is NOT NULL without a default — old code that does not know it can no longer insert a row,
• a new unique index — old code keeps writing duplicates that used to be allowed,
• old code runs INSERT INTO t VALUES (…) without a column list, or maps SELECT * by position — every new column shifts its world.

An expansion is backward compatible only if old code can keep reading and writing exactly as before. The test that catches it is simple: run the previous application version against the new schema in CI.

A long migration in pod startup

A migration adds a column with a default to a table with hundreds of millions of rows. In PostgreSQL ALTER TABLE takes an ACCESS EXCLUSIVE lock; in MySQL, a metadata lock. Every query on the table waits. Meanwhile the Kubernetes liveness probe sees a pod that "has not responded" for three minutes and restarts it — mid-migration. In MySQL DDL is not transactional, so the schema is left half-way and Flyway's history holds a failed entry.

Cure: run longer migrations outside application startup (a separate Job) or give the startupProbe generous headroom. Add lock_timeout and CREATE INDEX CONCURRENTLY in PostgreSQL, and online DDL tools in MySQL (gh-ost, pt-online-schema-change). The question to ask before every migration: how many rows does this table have in production?

DDL and backfill in one file

A new column and an UPDATE filling it for 50 million rows in a single transaction. Replication falls behind, the transaction holds locks, timeouts cascade through the applications. A backfill is a separate, batched step — ideally a process in the application that can be paused and resumed.

The rest

• Editing an applied migration. The checksum changes, flyway repair "to make it pass" — and history stops being history. Applied migrations are never edited; the next one fixes mistakes.
• Two schema owners. ddl-auto=update alongside Flyway: the ORM quietly "fixes" the schema, differently in each environment. There can be only one owner.
• Java migrations that use current entities. A two-year-old migration uses a class that looks different today — and on an empty database does something other than it did then. A migration must be frozen in time: plain SQL or JDBC.
• Version collisions from parallel branches. Two branches create V2_1_30. A permanent outOfOrder=true treats the symptom. Better: a check in CI at merge time and a numbering convention that tolerates parallel work.
• Tests on a different schema. H2 with an auto-generated schema instead of migrations on the real engine: green tests, red production.
• Disabled safety catches. validateOnMigrate=false or baselineOnMigrate=true set "temporarily, to get it running" and left there for years. The latter can quietly mark an unknown database as migrated.
• Per-environment conditional migrations. "If production, then…" — and the environments stop being copies by definition.
• Objects outside migrations. Views, procedures, triggers and grants created by hand. Flyway has repeatable R__ migrations for these, and grants are part of the schema too.
• Faith in undo. A down migration gives an illusion of safety: after a contract it cannot bring deleted data back. The strategy is roll-forward; the safety net is backups with point-in-time recovery.
• An implicit API. Reports, exports and integrations read tables directly and nobody knows. The shared database has quietly become a public interface — no version, no documentation, no owner.

In short

• One migration master — ideally the backoffice, which changes along with the model anyway.
• Everyone else checks on startup (validate) and fails immediately; in Kubernetes the old version keeps running.
• Deployment order: master first, then the rest.
• Migrations in one module, together with the data-access layer.
• No manual DDL — enforced by privileges.
• Expand now, contract later — never in the same release.
• A consumer registry and a guard in every narrowing migration.
• Contract with a due date, not dead souls.

Open questions

A few questions we have no single answer to:

• Is a database shared by many services an architectural mistake to be fixed, or a reasonable compromise to be handled well?
• Where is the line between a "dead soul" and a field that might still be useful?
• How should consumers outside our control be treated — forced to register, or given a stable view as an explicit API?
• Are tools like pgroll, which move expand/contract into the database itself, the future — or one more layer of complexity?

Different experience, a counterexample, an idea we missed? Write to us — we will publish the most interesting comments alongside the article, with their authors' consent.

Sources

• Flyway: ignoreMigrationPatterns, callback events
• Liquibase: Preconditions
• pgroll: repository and documentation
• Martin Fowler / Danilo Sato: Parallel Change