Automated Data Migration

200,000 XML files with constantly changing schemas, 3000 fields that needed to be mapped into 150 database tables. Just two months until the migration had to go live.

When a software company realized that their team of developers writing custom scripts wasn’t going to be able to meet their deadline, they called us.

Complex data migration, little time

The project involved working with a manufacturing company who needed to migrate thousands of critical manufacturing formulae from a legacy system into a new management platform in just a few weeks. The data was complex, was changing frequently throughout the project and there was a lot of it. While the client had had a team of developers creating separate scripts to migrate individual data structures, it quickly became apparent that the approach couldn’t scale quickly enough to handle the 200,000 files that needed to be migrated.

A more scalable data migration approach

Rather than trying to migrate so many different data structures manually, we took a different, more scalable approach and created a system that could automate the migration. The tool recognizes XML elements and attributes and automatically creates mapping documents, all without needing consultants’ input.

Despite the XML structures changing multiple times over the course of the project, the automated migration tool detected and adapted to structure changes and could successfully process all the files generically, automatically analyzing and mapping 3,000 fields to the new database. The project delivered on time and accurately and with a smaller team than originally planned.

Reusable for the future

The benefits of the automated data migration didn’t end there. A data migration is always an iterative process that continues beyond the go-live, so the fact that the framework is completely reusable and configurable means that the client can continue tackling further changes in data or structures for as long as they need.