5 Early Warning Signs of Data Migration Disaster

The best way to solve any problem is to catch it early, and when you’re planning a data migration, the ideal is to avoid the problem altogether. As migrators/integrators, we see a lot of unique projects, but they all share a common theme: their success is dependent on the team’s understanding of their data and system limitations. If those are overlooked, migration disaster can ensue. Today we’re sharing five of our top tips to help you recognize (and avoid!) the early warning signs of data migration disaster.

1. Impact on the IT Vision Has Not Been Considered

If upper management is driving the decision to migrate systems, they may not be aware of the challenges it could present to the IT department’s vision. This is particularly true when it comes to application interoperability with the overall IT design. IT departments typically institute systems that are compatible and support one another. Not often, but sometimes, IT may choose a unique but specialized product because of specific features not found in other products. It makes sense to have IT provide input on the migration to make sure they can support the new system and provide an integrated experience to users.

To secure a smooth transition, IT and upper management must cooperate to:

1. Assess the current technology stack to understand if the new system is suitable to the overall IT roadmap. 
2. Learn about the new product’s architecture and flag any potential integrations that might not work with the new product.
3. Discuss the effect of the new system on the existing technology stack. It is critical to catch dependencies that could force the upgrade of several systems to support the new one being implemented.
4. Keep an eye out for options that allow backward compatibility of technologies. These will provide the most flexibility during implementation.

2. The Current Web Service or API Doesn’t Support Full Read/Write Functionality

Modern CRM systems offer broader, more pliable architecture which can sometimes increase the complexity of logic and maintenance. To conquer this, newer data migration systems supply web service APIs that enable integrations without having to reconstruct the complex logic of the CRM. If your current system doesn’t support the framework, it is very difficult to extract data from it because it may not be able to support data in the format necessary for a successful data transfer. Various steps might be needed to extract, reformat and to process the data. The multiple steps end up costly and takes valuable time away from you.

One way to address the situation is to upgrade the current system to a newer version which supports web services & APIs. Although it is an extra step, it will bring data to a format that is more workable and update any legacy functionality to a supported functionality.

If the back-end database can be accessed via Open Database Connectivity (ODBC) or has some form of SQL query support, then it could be possible to pull the raw data via an ETL tool and migrate to a new software that is better aligned with your current IT strategy.

3. The Volume of Data is Large and Spans Multiple Product Iterations

Most businesses we work with have data in their systems that has been collected data for 10-15 years or more. In such cases, the software has likely gone through several upgrades and updates to fix different bugs and defects. Those bugs usually leave traces of bad or inconsistent data that don’t comply with the current data structure. These inconsistencies cause unpredictable issues during migration because the migration rules don’t understand how to handle the anomalies.

Avoid this issue by:

1. Ensuring there is extra time set aside to handle any unexpected obstacles.
2. Have technical staff on hand with familiarity of the older versions of the product so the background of possible data issues can be understood.
3. Plan to build additional logic to maneuver the historical data variations. This minimizes last second fixes and patches for various exceptions.

4. Your System Has Legacy Components That Don’t Support Data Export

Legacy systems are often kept running beyond their recommended lifespan because they were reliable enough to continue functioning – if it isn’t broke, don’t’ fix it, right? Another possible reason for keeping the system for so long could also be that the old system had parts which were difficult to re-engineer, indefinitely postponing the system upgrade. These situations present a particularly challenging problem. The system is so old that it becomes extremely difficult to work with, and many team members with knowledge of its functionality may have already moved on.

Hopefully, there is adequate documentation for the legacy system so the migration team can learn what its limitations are and how to best address them. Keep your eyes peeled for any software dependencies and save such software carefully so, in a worst-case scenario, the software could be recreated.

The more the software ages, the more important it becomes to maintain a relationship with the original software vendor. In many situations, they are the only ones who might have some information about the legacy system.

In addition to documenting the legacy system, the migration team must explore options for third party solutions that might offer backwards compatibility and enable pulling of data from your legacy system. It’s not always advertised, but products occasionally have the capability to go back and retrieve information from an older system.

5. Multiple Software Redesigns Accumulated Unnecessary Data Over the Years

If you’ve gone through multiple software redesigns, it’s very likely that much of the data carried over is no longer needed or used. It seems obvious that the data migration team could simply make the choice not to move the data over, but an organization’s policies could prove that the decision is much more complicated. It is important to have input from the right stakeholders and make sure all legal requirements are met.

To do this, the organization will need to initiate discussions about date retention policies as early in the process as possible. Work with your Legal and Compliance teams to get buy-in on the decisions. Get signoff from the business team on the agreed upon policies. If needed, work with data center and disaster recovery teams to archive old data for specified time periods, in case there is an unforeseeable need.