Data preparation is the basis for reliable and robust processing and efficient actions, and promotes the implementation of a results-oriented data strategy.
According to the Harvard Business Review*, a data project based on unprepared data costs 100 times more than one based on clean data.
After reviewing together the 3 essential steps of data preparationdiscover in this article the criteria that define quality data.
1. Data Quality at the heart of data mining and AI
My data is of poor quality. What are the consequences?
Data preparation is a key element of data governance. It is also one of the most important challenges for data teams. According to a study conducted by Gartner*, the databases of the largest companies are made up of more than 25% of erroneous data. To deliver the most accurate results possible, you need prepared data.
To understand the real impact of data quality on your data projects, let's take an example of the application of Artificial Intelligence. The use of Machine Learning algorithms allows for the prediction and detection of abnormal behavior.
This is useful in many sectors of activity and for many business teams: predictive maintenance in industry, detection of fraudulent behavior in the banking sector and for public administrations, or in customer relations for marketing teams.
Thanks to these algorithms, manufacturers can accurately anticipate machine breakdowns and the need for maintenance.
To predict such behavior, Machine Learning algorithms will rely on multiple criteria. For this, the data must be clean and reliable. For example, if one of the criteria on the time and format is different or incorrect, it will be impossible to detect the failure of a machine.
Data Quality is a virtuous circle for companies: reliable data delivers better results.
2. Data culture: a success factor for data quality
As you can see, without reliable data, data processing will not deliver accurate results and therefore no value to business teams. In fact, 82% of CDOs say that data quality is an obstacle to their data approach.
However, the quality of the data is determined at different stages:
- Data collection
- Data preparation
Proper data collection is a major step in obtaining quality data. In the collection process, some information is sometimes filled in by humans - implying a higher risk of error.
In this case, you will need to ensure several dimensions fundamental to good human collection:
- Do the people collecting the data have the tools to do so?
- Are the fields filled in by hand?
- Is the data filled in by humans complete?
Once you have evaluated the manual collection, look at what dimensions are missing and how you can improve them. For example:
- Make sure that the necessary tools are available for manual collection
- Set up closed questions with check marks to make it easier to fill in the information
- Add an obligation to fill in certain required fields
To improve human data collection, you will need to go through a data acculturation phase. This will allow business teams to feel more involved and aware of the value of data analysis.
The data culture will have a positive impact on the whole company: data teams will collaborate better and data collection will be improved.
"The need for data acculturation was quickly identified. We need to give meaning to data internally so that everyone can understand its ambition. A company that develops is also for the benefit of its employees.
Isabelle Brochu, Director of Innovation, Tingari
Read Tingari's testimony on the data acculturation of its teams.
The good collection of data is thus an essential prerequisite for Data Preparation which will then consist in :
- Aggregate
- Inspect
- Clean
- Harmonize
- And enrich the data
It is thanks to these different steps that you will have quality data at your disposal.
3. Key indicators to measure the quality of your data
After improving the way the data is collected, you need to make sure it is of good quality.
For this, there are 4 basic indicators that will allow you to recognize poor quality data:
- Compliant: The first criterion for assessing data quality is compliance. Specifically, make sure your data meets the rules, defined constraints and current legislation such as the RGPD.
- Complete: Then check the completeness of your data. Is all the information filled in? Do you need to add new fields to complete the information?
- Correct: A data can be completed with false or inaccurate information or spelling mistakes... It is necessary to check that the data entered is correct:
- Fresh: The question of updating data is essential. A treatment cannot be valid on too old data. For example, postal addresses used for customer knowledge when some have moved. Depending on the sector of activity or the business team benefiting from the processing, the data may be updated every day, every hour or even every minute.
To improve the quality of your data, you will need to comply with these indicators.
More globally, to get more ROI from your data, Data Quality is an important criterion in the company's Data Management that should be included in your 2022 data roadmap.
Do you want a turnkey action plan to optimize the time of your data teams?
Discover the practical guide "Which roadmap for your data in 2022? " .
Sources:
https://hbr.org/2017/09/only-3-of-companies-data-meets-basic-quality-standards