A data migration project is as exciting as it is frightening for IT teams.
It is estimated that 83% of data migration projects either fail or go over budget and over schedule (Gartner).
At the same time, according to IDC this time, data migration still accounts for 60% of all IT projects in large companies.
So it's highly likely that you'll come across a data migration project in your career.
You might as well be well prepared!
What are we talking about?
Data migration is a complex process that enables IT teams to transfer data from one information system to another.
Your organization may need to migrate and recover data, for example :
- To modernize your IS infrastructure
- Reduce the number of silos in your databases
- If you merge two entities or retrieve a new database
Why is it strategic?
Data migration and data recovery involve data, and are therefore strategic projects for the entire organization.
These are therefore particularly visible projects, and like all projects, they need to be carefully defined and planned in order to deliver their full value.
Limiting data migration to a simple transfer of data would be particularly simplistic, and that's already the first pitfall.
On a more global level, a data migration project enables an organization's entire data capital to be reviewed and a global inventory to be drawn up.
One of the keys to successful migration is ensuring that all the information in the source system has been transferred and transformed correctly. And therefore that the data transferred is reliable and usable.
You can see where we're going with this: data preparation in a data migration project is absolutely essential.
Depending on the organization, migration projects will therefore be very different (depending on whether data is already centralized or in silos, whether governance tools already exist, etc.).
However, there are a few best practices to follow to ensure a successful data migration.
So what are the absolutely essential aspects of this unique IS project?
We've deciphered for you 3 mistakes to avoid for a successful and worry-free migration.
Pitfall no. 1: document nothing
Considering data migration as a simple transfer of data is very simplistic.
Governance aspects are crucial to the success of a migration project.
We therefore advise you to document your project thoroughly, to ensure a common understanding of the project, but also to define its contours.
Here are some examples of essential documentation:
- Mapping and charting the data involved: this will enable us to define the boundaries of the project, and ascertain which data will be migrated, as well as their format and volume.
- Description of the migration process and identification of deliverables
- Data dictionary : this tool will provide a common definition of datasets, enabling us to get off to a good start.
- Data quality audit: this stage consists of assessing the quality of the data in the source system in order to define the quality enhancement processes required to migrate the data to the target system.
- Description of the transformation rules implemented: this essential step in the data migration and recovery process must be documented in order to understand the quality control processes performed.
- Processing report with matching of input and output data
Setting up or updating governance tools can be a long and tedious process.
Invenis' data experts can help you set up your migration plan and create the governance tools you need to ensure your migration project runs smoothly.
Pitfall no. 2: Neglecting data cleansing
As mentioned above, at the heart of a data migration project is the need to make all data usable.
If data is to be transferred to another system and decompiled, it must comply with standards that guarantee its use in the implementation of dedicated use cases.
It's also clear that data quality is at the heart of the migration project, and is just as crucial as the previous documentation stage.
To ensure the success of this data cleansing and quality assurance phase, data mapping and data quality audits are essential.
They will enable you to assess the work involved and ask yourself the right questions:
- What is my data capital (volume, usage, definition)?
- Where is my data? Is it siloed and scattered, or is it already centralized?
- What is the nature of the data (structured, unstructured, heterogeneous)?
- How confidential and sensitive is the data?
- How good is the data quality?
All these elements will enable us to put in place the most appropriate data quality plan. Be sure to take into account all the data you need to transform, and surround yourself with specialists for all your complex data, especially media and sensitive data requiring special attention and skills.
You will then be able to implement the appropriate quality assurance plan for your data (management of inconsistencies and incompleteness, typing errors, duplicates, formatting errors, aggregation issues, etc.). Once the data has been processed, it can be fed into the new system and used by your various teams.
Please note: data quality is everyone's business.
Don't hesitate to communicate with your business teams and your IT department during this strategic data quality phase. The audit phase will highlight the various data quality issues encountered in your databases.
While most quality defects can be resolved by processing to improve data quality (e.g. duplicate management, automatic corrections, joins and aggregation, etc.), other adjustments can be made upstream, at the time of data collection, to improve data quality at source (missing data, partial data, textual data, etc.).
Pitfall no. 3: automate nothing
Finally, to create a routine that works, we recommend that you automate recurring tasks, such as data cleansing and quality control.
Automating data cleansing and preparation means you'll always have access to data your organization can use.
To achieve this, we recommend that you anticipate the global aspect of your data quality project. In other words, your project must be able to be industrialized: your data migration project is of no value if you are only able to carry out transformations on part of your data.
You'll need to cleanse, cross-reference and reconcile all your data in order to finalize your migration project and be able to identify the match between all your input data and your output data. We recommend that you work with Data Preparation tools, which make it easy to put your data processing into production and automate your data quality pipelines.
BONUS tip : don't overestimate yourself!
As you can see, a data migration project is a high-risk undertaking.
Numerous studies point to poor data quality as the main cause of data project failure. Not all IT and Data teams are mature enough to reliably manage a data migration project. So don't hesitate to surround yourself with teams of experts in the field, and enlist the support of Data Preparation experts.
Our teams of Data experts support IT Departments in their data migration projects:
- Audits and implementation of data governance tools
- Automatic data cleansing and quality control
- Uploading data to the target system
Do you have a data migration and recovery project?
Sources :
https://www.techtarget.com/searchstorage/feature/Keys-to-a-painless-data-migration-process