Menu Close

Online Survey Programming QC: Stop The Madness!

survey programming checklist

Ok, so you hire a survey programming vendor. They offer best-in-class programming software, a robust reporting package and a seemingly endless number of programmers available to work on your survey. You agree on the budget, schedule, etc. You deliver a complete and final survey instrument. Everything seems to be all lined up. All you need to do is review the programmed survey, submit edits if necessary and then be on your way to data collection. It all sounds so easy. Sadly, this is rarely the case. You end up spending countless hours checking the programmed survey. You struggle to keep track of your edits. You fail to accurately record the different paths/scenarios you tested (or thought you tested). You send edits, comments, suggestions and to your dismay, a few of the edits are not implemented. You recheck and recheck and send email after email. Does this sound familiar? You tell yourself there must be a better way. You swear each project will be different, yet here you are again, and again and again. We often refer to this as the QC infinite loop.

The Challenge: Lack of a QC Tool/Methodology

Most programming vendors are strong on survey programming but extremely weak on survey review. The vast majority of programming vendors do not have a methodology for conducting or managing the survey QC process. Survey vendors can no longer simply hand off a link to the client and say “here is your link, go ahead and check it.” If your vendor values you at all, they will provide you with the tools to make checking easy, convenient and repeatable. And to be blunt, if they cannot provide you with a tool for checking, it does bring into question, how they checked the survey before handing it off to you. If your current vendor does not have a QC tool and/or methodology, there is no structure. If there is no structure, you are destined to be stuck in the QC infinite loop.

The Big Question: Why Does the Client Have to Check the Survey?

As we onboard new clients, we are often faced with the question: “Why do we have to check your work – isn’t that what we are paying you for?” They are correct, they should not be checking our work, they should be checking their own work. That is, we often encourage our clients to review the programmed survey to make sure what was authored on paper matches their expectations in an online environment. This type of review is often referred to the “look and feel” test and most, if not all, of our clients want to take a look before launching the study. This often results in clients finding issues with their own logic, or they may want to rephrase questions or change scales. There is also a strong minority of our clients that want to check the entire study including question text, attributes, logic, etc. As a result, it is critical that clients be offered a tool to conduct both the look and feel test as well as the deep dive review.

Do You Have A QC Problem?

The easiest way to determine if you have a problem with survey QC is to answer the following questions? If you answer “yes” to 2 or more of these, then you need to have a discussion with your programming vendor. If you answer “yes” to most of these, then my guess would be that your vendor is not providing you with a tool to make checking easier.

– Are you spending too much time checking surveys?
– Are you finding errors in your survey programming?
– Are you finding cut and paste errors?
– Are you feeling burned out by survey checking?
– Are you having to send multiple emails with edits?
– Are you struggling to keep track of your edits?
– Is it difficult to follow the paper survey and online survey at the same time?

Your Next Survey
It’s entirely up to you but Brookmark believes that clients should not have to waste their time navigating the survey review process. If you are struggling in this area then you need to revisit the process. Find a vendor that can make your work life easier. Ask your vendor for a QC checklist and hold them accountable for the items on the list. If no list exists, then this should serve as a red flag that QC is not a top priority.

About Brookmark Research:
Brookmark is a research operations company focused on managing the critical components of the research process including project management, survey design, programming, hosting, sample design & management, data management and reporting solutions. Brookmark provides a full outsourcing solution for all your research needs.