Data Migration

Data migration can feel daunting, even though it’s a standard procedure that most organizations go through at some point. Careful planning and preparation will help to remove most of the potential pitfalls and, in most cases, allow a business to transition to a more effective and resilient way of working.

This guide offers an overview of what data migration is, examples of when it might be required, and some tips for ensuring a straightforward process.

What is a data migration?

Data migration is the act of moving data from one place to another - typically between applications, storage systems or databases.

There is one active data set before and after the migration. This is in contrast to data integration, where data is active in both or several places.

Almost all companies will eventually undertake data migration for various reasons, including application replacements or upgrades, business process changes, data volume growth and performance requirements.

Data needs to be moved to the new location and checked to ensure that it is fit for purpose in the new system. This often requires data validation, correcting problems in the source or during transportation, converting data formats and more complex data transformations such as merging values or calculating new ones.

Data migration approaches

There are three main approaches when it comes to data migration:

1. Merge two systems into a brand-new one
2. Migrate one system into another
3. Leave the systems as they are but create a common view on top of them – a data warehouse

Whichever approach you choose, several different data migration methodologies are on offer.

Data migration techniques

Big Bang Data Migration

A big bang data migration is when the entirety of your data is migrated all in a single operation. While the process may take a while, users experience a single point in time where they can no longer use old data before the new system goes live. As such, they experience the change as sudden and substantial – a 'big bang' event.

Big bang migrations typically require significant preparation periods and short downtimes, during which the system is unavailable. The ideal big bang migration has zero downtime, but guaranteeing this is usually impossible.

The overall process can be visualized like this:

Big bang data migration process

1. Design phase
   This is where you plan your project scope and goals, analyze your data samples and create your data migration plan. You can propose an architecture, schedule, and budget based on what you discover.
2. Development and testing phase
   In this stage, you implement the proposed architecture. This stage generally takes the most time. Testing on data samples is vital, as is developing and testing your migration tools (whether you’re using traditional programming languages or more specialized tools). No data is migrated during this phase.
3. The big bang
   This is when all your data gets migrated. Downtime of source and target systems is usually required to ensure data consistency.
4. User acceptance testing (UAT)
   Let users verify the migration result. If everything is ok, the source system can be turned off.

The migration can only be considered complete when data owners and all other stakeholders have confirmed that migration has been successful.

The fact that the migration itself only happens at the end of the process is both a benefit and a drawback of big bang migration. Users do not need to handle two different systems simultaneously and manage the potential awkward ‘overlap’ between two live systems.

However, as the actual data migration happens so late in the process, this increases the burden on the planning, development, and testing phases. Any problems in these areas can lead to expensive failures.

Trickle Data Migration

A trickle migration breaks the migration into smaller sub-migrations, each with its own set of goals, data, deadlines, and scope – effectively an agile approach to data migration.

Trickle migration allows stakeholders to verify the success of each individual phase, giving a continual indication of progress. Should any of the sub-processes fail, it is usually only necessary to re-run the failed process and prevent that failure from recurring on subsequent runs.

Trickle migration may require complex planning at the outset and also places a higher burden on users who need to work with two systems simultaneously while the ongoing overall migration takes place.

Big Bang vs. Trickle Data Migrations

Each methodology has its pros and cons. Investing resources and attention in the planning and execution of your migration is what will ultimately determine its success.

Advantages of a big bang data migration

• Costs are generally lower due to less training and less managing of parallel systems
• More straightforward: there’s no need to consider parallel systems
• All changes are made in one go, over a relatively short space of time, so there’s a single defined cut-off point

Disadvantages of a big bang data migration

• A high risk of expensive failure: unexpected trouble, coupled with a lack of agile project management, means problems may only be discovered after migration (at which point it may be too late to fix them)
• If the migration fails, a complete rollback is usually required
• Requires downtime: depending on your organization and the systems you’re using, downtime might not be an option

Advantages of a trickle data migration

• Less risk of expensive surprises due to compartmentalization of work and more frequent runs
• Zero downtime required as the migration is incremental
• If a single phase fails, only that single phase needs to be rolled back and repeated

Disadvantages of a trickle data migration

• More expensive – you need to maintain multiple live environments
• Needs effort to keep two systems running – not just for technical staff, but also for end users
• Complicated syncing issues may cause confusion and require extra oversight

Choosing the right data migration methodology

Deciding whether to choose a big bang or trickle migration should be made as early as possible. Ask yourself the following questions:

• What is the migration deadline?
• Can the system (or systems) experience downtime?
• Do you fully understand your data so you can plan the entire process?

There is no simple rule for choosing one over the other. However, big bang migrations are typically selected where the scope is well defined from the outset and where deadlines or other project properties mandate it.

Conversely, trickle migrations are beneficial when the migration can be easily split into several stages. They are also suitable when the scope is hard to define. In such cases, the trickle migration's phased migration approach allows you to migrate “easier” data first while dealing with more complex processes later.

Another major consideration is the experience of your team. Each migration can pose different technical and project management challenges. Teams that prefer an agile approach typically prefer trickle migrations, and teams that are more used to a waterfall methodology prefer the big bang approach.

Planning a data migration

The importance of good planning for your data migration can’t be overestimated. Poor planning can jeopardize business initiatives in one fell swoop and lead to a catastrophic loss of trust and a hefty bill to pay.

However, data migration also offers businesses the chance to dive deeply into and rethink their data. Investing time and effort into the planning process leads to a successful migration project and long-term rewards in the form of more efficient systems and greater business value.

Defining migration scope

Clearly define your goals. What would a successful migration look like? Remember that you don’t always have to move everything. In general, your main planning rule should be to find the smallest possible subset of the source data (smallest in terms of complexity rather than actual number of records) that will get you to your goal.

Estimating effort

What will the process require from you? Considerations include (but are certainly not limited to):

• How much data there is
• The age, complexity, and value of your data
• The timing of the migration (and any other projects dependent on it)
• The approach you’re taking. Automation can rapidly speed up delivery

Finding the right people

Business users are the ones who understand the data. Make sure they’re involved in the process and invested in the chosen method. This is also a chance to seek out any potential unforeseen pitfalls ahead.

Contracting external experts

If you want external help, getting an expert in both systems is ideal, but ensure they at least understand your target system. (After all, you probably already know a lot about your own source system). Take time evaluating potential partners, and make sure they combine technical expertise with an approach and manner that fits yours. Will they go the extra mile for you?

Finding the right way

Build a scalable, repeatable process from the outset. Believe us, it will save you time, effort and frustration later.

Stages of a data migration process

Data migration involves several stages:

1. Defining the scope
2. Finding teams & resources
3. Planning
4. Data discovery
5. Budget
6. Milestone start
7. Implementation
8. Milestone testing
9. Reiteration
10. Project sign-off
11. Go-live
12. Ongoing data integration
13. Contingency

Each stage has its considerations and needs to be considered to deliver a successful project.

Data migration examples

Typically, data migration occurs when upgrading existing hardware, transferring to an entirely new system, or in instances such as application replacements, business process changes, data volume growth, or to unlock better performance.

Some examples of when a data migration might be needed:

• Mergers and acquisitions: The need to unify completely separate worlds when companies combine. IT stacks and data usually need to be joined to provide one unified system for the new entity.
• Modernizing for performance: When a current system struggles to keep up with performance requirements. For example, two legacy systems create performance problems, duplicate data, and compatibility. One company saw operating costs reduced by over 40% when they moved to a new ERP system. 27 million records needed to be migrated, but the project delivered substantial efficiency gains.
• Moving to the cloud: A potentially complex process, as it usually means migrating multiple applications. Even with cloud providers’ migration tools, custom development is often required.
• Moving a data warehouse from one database to another: Migrating data like this can be more complicated than it first appears because the move is rarely a 1:1 copy. The migration often involves data cleansing exercises or more complex changes if the target database behaves differently from the source (e.g. if the target is a columnar database while the source is a traditional relational database, the data must be transformed before it is loaded).

Learn more about moving to the cloud here: Migrating Data Workloads to the Cloud

Data migration project success story

Automated data migration reduces onboarding time by 90%

A workday data migration implementation consultancy can now onboard significantly more clients, without the need for extra workforce.

• Before the project migrating each new customer took around 200 days.
• After the automated data migration framework was deployed, migration now takes just 20 days.

Automating the mapping needed to move data from the clients’ legacy systems into Workday means that the consultants now only need to spend a few days configuring the framework rather than weeks (or even months) in Excel like they had to do before.

Read the case study: Effectively Migrating Legacy Data into Workday

Project highlights

• Data loading: The migration framework can generically load data from Excel files based on configuration stored in the database
• Data validation: The solution can validate each record for any given HCM workflow. The workflows are highly customized based on module, but additional modules are simple to add in
• Reconciliation: The generic, database-driven reconciliation framework can be reused to reconcile data based upon a configurable key
• Reporting: Report on any errors with input data, transformations, validations and output data

Ask us about your data migration project, or see how CloverDX can automate and streamline your data migration by booking a demo

Data migration with CloverDX

Our mission: to help you move your data from anywhere to anywhere in a way that is streamlined, speedy, and safe.

We’ve helped many businesses with their data migration projects, whether it’s helping a client move legacy data into a new system or creating a powerful and bespoke automation framework to process huge numbers of records against a strict deadline.

Here are some of the benefits our customers have seen when using CloverDX fortheir  data migration:

• Reduce time to delivery with visual interface and ready-made components for rapid building of data workflows
• Reduce expensive surprises with repeatable and adjustable workflows
• Automate complex processes to save time, energy and frustration
• Data validation before it reaches the target system
• Creation of data quality loops within processes
• Ease of data transformation at any scale
• Full transparency of the workflow at all times

What the process of a data migration with CloverDX looks like

First things first, we do what we do with every project: sit down with you and find out what you need. It goes without saying that you want your data migration to proceed as quickly and reliably as possible. No two clients are the same, however, and we tailor the project to suit your needs.

For example:

• How tight is your deadline? Perhaps your in-house staff are overstretched, and automation could help set things in motion
• Are your data sets heavily customized? Are they in order or messy?
• What’s your bigger mission? Are you entering a new market perhaps, or aiming to onboard lots of new clients?

Whatever your answers to the above – and any other questions you have – we then create a process that’s intuitive, effective, and, above all, bespoke to your business.

Our platform helps you plan and implement your ideal data migration process. Using a visual interface with a selection of ready-made components, it’s easy to build fast and flexible data processes.

Any workflows you build are repeatable, which means you save time by not needing to rework the entire process any time your data or requirements change.

At all times, your data stays in your control, with complete visibility, transparent documentation, and version control capabilities.

In short, data migration can be done however suits your business.

We can help

Our demos are the best way to see how CloverDX works up close.

Your time is valuable, and we are serious about not wasting a moment of it. Here are three promises we make to everyone who books a demo:

• Demos tailored to you. Every business is unique. Our experts will base the demo on your unique business use case, so you can visualize the direct impact CloverDX can have on your process.
• More conversation than demonstration. Have a question? We want to hear it. Data migration can be anxiety-inducing for any business, even those who’ve undergone the process many times. Whatever concerns or reservations you have, let us hear them.
• Zero obligation. We’ve all been there. You spend some time hearing about a product or service… and then comes the hard sell. Our team doesn’t ‘do’ pushy. We prefer honest, open communication that leaves you feeling informed and confident.

Get in touch for a personalized demo.

To learn more about ways in which CloverDX makes data migration easier, just click here.

Read more about data migration

• 7 Benefits Of Using CloverDX For Your Data Migration
• Webinar: Migrating Data Workloads To Cloud
• Blog: Users: The Real Pain of a Salesforce Data Migration
• Blog: How To Get Data To/From AWS Databases
• Case Study: Effectively Migrating Legacy Data to Workday