Behind the Scenes of Stellar Surveys: The Data Quality Edition

Surveys have continued to grow as a popular research method for companies to gather primary insights. Yet concerns over fraud and compromised data quality have cast a shadow over the reliability of these services.

It’s critical for companies to find trusted partners to execute their market research. We’ve partnered with one of our trusted survey partners, NewtonX, to bring you a two-part series on Data Quality in the survey world to give you the tools to find the right partner for your next survey.

9 Techniques in the Survey Toolkit

The first step involves understanding the tools survey companies wield to safeguard data quality. Below we’ve defined the key tools and techniques used by top-tier survey companies:

Custom recruitment - the method of identifying, contacting, vetting, and ultimately signing-up panelists to participate in a specific project and potentially join a database. Typically companies rely on participants from previous studies who then become part of a database. While having a large database of willing participants can be useful, especially from a cost & speed perspective, it can mean that for new studies clients aren’t getting the exact right or freshest perspectives.
Profile verification - the way in which a survey company can confirm a panelist is who they say they are and that they are suited for a project. Sources can include LinkedIn profile checks, CV analysis, IP tracking, email verification, using meta-data (especially if the panelist has already completed projects with that company in the past), or even background checks.
Content knowledge checks - asking a panelist to pick correct definitions, define various acronyms, or provide other information which confirm their expertise on a topic.
Open-end validation - checking for duplicate answers, frequent misspellings, irrelevant answers, and/or gibberish entered in open-ended fields.
Conflict-answer flag - asking the same question twice or asking similar questions looking for conflicting answers.
Speeder flags - flagging any panelists who finishes a survey too quickly, e.g. in less than one-third of the median length of a survey.
Red herrings - adding a fake vendor to a list of real companies.
Straight-line flagging - flagging when a panelist selects a majority of options in a multi-select question or continually selects an individual answer on a grid.
Duplicate checks - checking for duplicates to ensure that the answers of the survey panelists are not recorded twice.

Smart Questionnaire Design for Fraud Prevention

Apart from these strategies, employing a smart questionnaire design is paramount to counter fraudulent activities. Thoughtfully crafted questions and response options can act as a preliminary defense against bad or deceptive responses. Logical traps, consistency checks, and even CAPTCHAs are proven deterrents against bots and malicious actors. Any serious survey company will help you write out a smart questionnaire to avoid any risk.

Custom Recruitment and Profile Verification: Pillars of Data Quality

One of the top ways survey companies ensure high quality data is by relying on profile verification and custom recruitment to curate a targeted, authentic participant pool. With regards to custom recruitment, companies use both manual and tech-enabled techniques to identify and onboard panelists.

Profile verification is another tool that’s essential to ensure good data quality. Expert networks tend to excel here (when they have the populations in their databases) because they have a deep set of data points on each individual they work with. That means it’s easy for them to quickly know if a certain expert has ever bought a specific product or service, or if they fit a set of criteria (e.g. CTO of a 500 person company headquartered in Germany). However this can prove dangerous if relied upon too much as the databases can become static and dated which leads to challenges sourcing niche experts at scale.

Other companies, like NewtonX, have developed a proprietary algorithm to identify and verify the right people to take surveys for their clients.

Quality is Worth Every Penny - Consider the Cost of Bad Data

As mentioned above, many best-in-class survey companies invest heavily into technology and personnel to maintain high data quality standards. But you might ask, doesn't this inflate survey costs? Oftentimes the answer is yes, but imagine if you made a critical business decision with bad data. The cost could be catastrophic.

For instance, the CTO of Cascade Insights learned the hard way what can happen when a survey panel provider doesn’t have proper quality checks. He ran a large project and once completed, discovered that most of his data had to be discarded due to poor quality after the survey panel provider outsourced the sample collection to a 3rd party. In another example, Cint, one of the largest B2C Marketplaces in the world, had a costly end of 2022 due to high amounts of fraud across their business.

Guiding Your Quest for Excellence

So, how can you practically apply these insights to select a capable survey provider? The simplest route is through Inex One! All providers on our platform undergo rigorous evaluation before being accepted. We assess their processes, technological capabilities, and compliance adherence, ensuring that clients always achieve the best results.

Most importantly, enforce data accountability when evaluating potential vendors. Request a clear overview of their data quality measures. Understand their tools, technologies, panelist recruitment methods, and verification processes. Remember, an informed decision today can prevent significant repercussions down the line.

Ready to get started? Launch your survey here, or click here to discuss your research needs with our team.