The software is the easy half. The records are the project

Two different jobs that get confused

Almost every project that starts as "we need our data moved" turns out to be two projects wearing one name, and separating them early is the cheapest decision available.

Migration is a one-time move. Records leave a system that is being retired and arrive in one that is taking over. The risk is concentrated at a single moment and the question is always completeness: is everything here, is it the same, and can we prove it.

Integration is a permanent arrangement between systems that both continue to live. The risk is spread across years and the questions are different: who owns each record, what happens when both sides change the same field, and how a failure is noticed rather than discovered at month-end. The architecture for that side is on integrations, which covers the bridge patterns and failure modes in depth.

Most companies need both. What they must not do is run them as one workstream, because migration wants a freeze and integration wants continuity, and a plan that pretends otherwise fails at the cutover.

How much history to move

This is the first commercial decision, and it changes cost more than record volume does. The useful rule is to move what will be acted on and archive what will only be read.

The five passes of a migration

Every migration we run goes through the same five passes, in the same order, and every pass produces an artefact somebody can review.

1. Extract

   Prove the route out of the source before anything else: API, vendor export, direct database read, reporting interface, or structured extraction from generated output. Take a real sample, time it, and calculate the full extract from the measurement rather than from optimism. Attachments are tested separately. The source system is never edited to make extraction easier, so the pass can always be repeated from an unchanged original.

2. Profile and clean

   In a staging area, not in the source and not in the target. Count records per type, find nulls where the target requires a value, find the duplicate customers and items, normalise units, currencies, country codes and tax identifiers. Every rule is code, so it can be re-run and reviewed. Records that fail validation go to a quarantine list that a person from the business decides on; nothing is dropped quietly.

3. Map

   A written field-by-field mapping from source to target, including what happens to fields with no destination and where default values come from. This is also where the business key per entity is declared — the field combination that decides whether two rows are the same real thing — and where the identifier mapping table is created so every source key keeps a permanent link to its target key.

4. Load

   Idempotent and restartable, in dependency order: master data first, then transactions that reference it. Because the load reads the identifier mapping table, running it twice updates records rather than creating second copies. A load that cannot be re-run safely cannot be rehearsed, and a migration that cannot be rehearsed is a migration performed for the first time in production.

5. Reconcile

   Counts per record type and sums on the amount fields, taken from the source before the load and from the target after it, compared line by line. Every difference is explained, not tolerated. The reconciliation report is the deliverable the business signs, and it is the only honest answer to whether the data arrived.

Identifiers, duplicates and the source of truth

Three technical decisions cause most of the pain that appears months later, and all three are cheap to get right at the start.

The business key. For each entity, what makes two records the same real-world thing: a tax number for a company, a code for an item, an email plus a name for a contact. Declared up front, it prevents duplicates during the load and again during years of integration afterwards. Declared late, it becomes a deduplication project.

The identifier map. A permanent table linking every source identifier to its target identifier. It makes loads repeatable, it makes support questions answerable a year later, and it is what allows a record to be traced back to the system it came from when somebody disputes a figure.

The source of truth per entity. After the move, exactly one system owns each entity and the others receive it. Customers might be owned by the CRM, items by the ERP, invoices by the accounting package. Where two systems both write the same entity, the conflict rule is written down before the bridge is built, because a bridge without a declared owner produces month-end arguments that nobody can settle from the data.

Cutover, and the rollback that makes it safe

A cutover is a rehearsed procedure, not an event, and the rehearsal is not optional.

The full migration is run at least once against a copy of production, timed from beginning to end, with the reconciliation report reviewed by the people who know what the numbers should say. That rehearsal produces the real duration, which is what the cutover window is planned from. On the day, the source is frozen for the entities being moved, the final delta is extracted and loaded, reconciliation is re-run and signed off, and only then are users pointed at the new path. If reconciliation does not pass, the freeze is lifted and the company carries on in the old system, having lost a weekend rather than its records.

Where a freeze is genuinely impossible, the alternative is a parallel period with a declared source of truth per entity and a bridge carrying changes one way, retiring the old path per entity once the two agree. That is slower and it is the correct answer for operations that cannot stop. The staged version of this, including the accounting boundary, is described on the migration guide.

After the move: making failure visible

The bridges that remain need three properties, and their absence is what makes integrations fail silently for weeks. Retry with backoff, so a temporary outage on either side does not lose a record. Idempotency, so a retried message does not create a duplicate document. And monitoring that is a real alert to a real person rather than a log file nobody opens, with a reconciliation job that compares both sides on a schedule and reports drift while it is still small. The full set of failure modes is on integrations.

When you should not migrate

• The old system still fits. If the package does the job, a migration is expense without benefit. The decision framework is on packaged vs custom.
• The data is not trusted in the first place. If nobody believes the current figures, moving them produces distrusted data in a new schema. Clean first, then decide.
• Nobody will sign the reconciliation. A migration needs a person from the business who can say the numbers are right. Without that person, the project has no definition of done.
• Only one entity is really involved. Sometimes the answer is a single bridge rather than a move; that is a smaller and cheaper project and we will say so.

What a built system looks like on the other side of the move is described on custom software, and the published bands and timelines are on pricing.

Frequently Asked Questions