We often hear horror stories from new clients about survey programming errors. This ranges from simple spelling mistakes all the way to logic blunders. The reality is mistakes will occur from time to time but your survey programming vendor needs to do everything possible to minimize mistakes. As we all know, programming errors lead to a whole host of problems including bad data, budget overruns, missed schedules and in some instances, lost clients. The majority of mistakes can simply be prevented by having a repeatable quality control methodology that must be adhered to at all times.
It really comes down to asking your programming vendor to provide a documented approach clearly outlining how they check surveys. If they don’t have a system in place for checking then it’s up to you to determine if you can live with an arbitrary process that might lead to issues down the road.
Most surveys have the same basic elements including:
- Question text
- Question instructions
- Randomization
- Response items and pre-code values
- Scales and scale labels
- Skip logic
- Logic tied to exercises such as brand assignments and occasions
- Experimental design mapping and implementation
- Creating and calculating variables
The list can go on and on but there needs to be a way to check all of the above items visually as well as via a comprehensive data review. Most importantly, the process should not be driven by individual preference. That is, each project manager should not be applying their own personal approach or preference to conducting QC. This approach is neither repeatable nor does it guarantee every item is checked in the same way.
We often hear about project managers trying to check all survey elements at the same time which might seem to be a good way to go but often results in something being missed. A more efficient approach is to check each element of the survey individually and then leave the logic to be reviewed without having to worry about if all the items in a list are spelled correctly or if a response list was randomized or if the piping for a question is working properly. There is simply too much going on in a survey to check everything at the same time. At Brookmark, we have created a simplified way to check each individual survey element before testing logic. The final step in the qc process is a comprehensive review of automatically generated test data where all logic is examined using SPSS.
For your next study, ask your programming vendor to demonstrate their approach to survey and data review. If they cannot share this information with you, then you need to decide if you can live with a process that does not guarantee the accuracy of your survey or data.
About Brookmark Research:
Brookmark is a research operations company focused on managing the critical components of the research process including project management, survey design, programming, hosting, sample design & management, data management and reporting solutions. Brookmark provides a full outsource solution for all your research needs.