For many years, survey data quality was not a concern for research buyers. Suppliers were responsible for putting in place the necessary measures to ensure that the quality of survey data met the highest standards. And for years, clients received the good quality data they paid for.
This has changed in recent years. There is a growing concern about data quality in market research, and this concern has reached clients, who very often struggle to make sense of their datasets. The issue has led to the creation of joint initiatives where vendors and clients analyse and try to come up with measures to improve data quality. And at an individual company level, more clients are taking an active role in protecting data quality, very often working with their vendors to address this issue.
The fact is that the main responsibility for research data quality still lies with the providers. But there are things that clients can do to improve data quality. Let’s look at them, but first let’s talk about the reasons for concern about the quality of research data.
The rise of fraud in market research and the role of AI
While the majority of respondents are willing to give their views and complete surveys honestly, there has been an increase in the number of completions that can be considered fraudulent. This means that the respondent is either not human, not the person they claim to be, or is deliberately giving false answers.
Although the incentives to complete surveys are usually small amounts of money in major economies, they can be significant in some developing countries. There, some people make a living by collecting as much money as possible from responding to surveys, even if it means pretending to be someone else to qualify for the job. And if they do not know where to start, tips on how to get even more incentives are regularly shared on social media. So as more and more people become aware and start taking surveys, the data looks increasingly suspicious.
Since then, all sample providers have implemented anti-fraud measures. There are now more checks when recruiting new panellists, when they sign up, when they take the survey and, of course, when they are paid. In addition, all data is scrutinised for suspicious patterns and inconsistent or impossible responses. If respondents are flagged as suspicious, their data is removed from any ongoing survey and they are contacted for an explanation, if not banned outright.
Despite these efforts, the number of completions considered fraudulent has increased, not decreased. Every time sample providers add a new layer of security, fraudsters look for ways around it. And the widespread availability of artificial intelligence has not helped. AI can now automatically respond to surveys and make them look real. And even if there are humans responding, they are now using AI to assist them. For example, if fraudsters do not speak the language of the survey, they can use AI to translate questions and then respond. They can also generate answers to open-ended questions using ChatGPT or similar solutions. AI helps fraudsters to collect incentives more efficiently – and data quality is still more of a challenge.
Why you, as a client, should care about survey data quality
If when looking to the results of your survey, they do not add up, you may have been victim of fraud. If you have come to this conclusion you will have to discard completes or even the entire dataset. This would be a waste or resources you could have used elsewhere.
However this is the best possible scenario. It would be even worse if you do not detect the fraud and use the data to make decisions. They can affect your business performance and it may have very bad consequences.
This is not a situation you want to be in. Relying solely on your vendors to fight fraud can be risky for you and your organisation. That is why I encourage our clients to take an active role in ensuring data quality. That way you have a better chance of avoiding wasted surveys and making decisions based on reliable data.
Six ways to improve data quality
The good news is that there are plenty of things you can do to combat fraud and improve data quality.
- Ask your sample provider (again). Don’t assume that your provider is in a good position and ask for a list of the quality checks they have already implemented. These should include speeding detection, geo-location, cookies to avoid duplicate respondents and more. Ask about the level of fraud detected in their panels and how this has changed over time to see if the measures are actually working. Even if you get satisfactory answers, keep raising your hand to keep your vendors on their toes to continue to improve data quality.
- Trust but verify. Check your survey results against other data sets you have access to. If you can ask for brands purchased in the category in the survey, you can compare this with retail and consumer panel data. Another very common one is to ask about social media usage and compare it with published figures. This may require some desk research to compile datasets you can use for comparison. Do this before you complete the questionnaire and you will have the data points you need to cross-reference.
- Agree with your provider on the types of questions to be used. Some quality checks work specifically with open-ended questions. Others use matrix questions. Make sure you have enough types of questions to make the most of the battery of quality checks your provider has implemented.
- Introduce trap questions. These can take various forms. They can be repeated questions to check for inconsistencies. Questions with only one correct answer allow you to eliminate participants who fail to identify the correct one. Also any ‘yes/no’ question that makes it very unlikely that the answer will be ‘yes’ (e.g. ‘Have you flown from Denver to Stockholm in the last seven days?) Any of these trap questions can help you identify fraudsters and ideally every questionnaire should have one or two. Agree with your providers which ones to introduce and check that panel members with suspicious answers are removed from your results.
- Make sure you have access to sufficient research expertise . We have talked a lot about fraud and panel quality, but data quality depends on many more factors. Having enough research expertise to oversee each project is perhaps one of the most important. This will give you confidence that the methodology you have chosen is the best for your needs, that the sample is correct, that the questionnaire is well designed and that there are no biases or potential confusions for respondents. If you do not feel confident enough, seek help.
- Test your surveys. Before sending the questionnaire to fieldwork, test your survey to make sure everything works as expected. Pay particular attention to logic jumps and branching, so that each consumer can move smoothly through the entire questionnaire. If necessary, involve more people in testing so that they can catch anything you may have missed.
Here to help
You pay for good quality data and you deserve to get it. Being aware of the problem is the first step to solving it. Most of the responsibility for tackling fraud lies with the suppliers, but there are things you can do to influence them to put data quality at the top of their list when dealing with you.
At Origin Insights, we work with many sample providers and have all these conversations on a regular basis. Whenever we use a panel, we can be confident that the quality checks are adequate. We do not work with sample providers whose data quality we do not trust. And we work hand in hand with them to design questionnaires that are excellent at detecting fraud.
We also have extensive research expertise to prevent data problems and our own protocols to ensure that surveys are thoroughly tested before going into fieldwork.
If you feel you are experiencing data issues and need help, please let us know. We will be happy to discuss and make recommendations to ensure you are on the safe side.
Mar Serrán
CEO
Want to increase the accuracy of your insights?
