In the online panel industry, traditionally when we think of aggregators, a negative connotation first comes to mind. After all, the term “aggregator” is usually associated as something dirty, done by those involved in some scam or another. The perception is that any form of aggregation will taint your study. It’s an easy conclusion to come to, if defaulting to the view that aggregation is nothing more than bulk mailing lists or general brokers buying un-vetted, and unmanaged third-party sample from multiple sources, adding their margin, and selling it on.
It’s true that early aggregation models/practices were seriously flawed. Many of us were burned by the early aggregators; we lost projects, money and in some cases, we lost significant business relationships. The reality is, aggregation is a necessary business practice. If you need to conduct a study online and you need something other than, or in addition to, proprietary client lists, then it most cases you’ll need to purchase sample from one or multiple panel providers.
But thanks to new tools, we now have the ability to form panel samples that are much more finely-tuned, accurate, cleaner and credible. It’s no longer a dirty word. How? In a word: vetting. Despite a vocal minority who would have consumers believe otherwise, the process of putting a panel sample together, if done right, becomes an ever more useful research tool for companies of all sizes. It’s critical to the success of the overall sample and market research industries. Survey Sampling International, Research Now and all the other “big” players will use partners (i.e., other sample companies) to help fill studies that fall short of the target number of completes.
Again, let’s be clear, we are not talking about bulk mailing lists or list brokers — we are talking about the highly recognized and credible players in the industry. In fact, I can’t think of a study that Brookmark Research or OpinionRoute conducted in the past year that didn’t require our primary sample vendor to reach out to “partners” to close the study. Sample aggregation, when done by a reputable company, is acceptable and important. It’s necessary to conduct research, producing data which yield results. It’s simple really—quality sample generally leads to quality data which leads to reliable and trustworthy results. It starts and ends there-–if the sample is bad, you may as well throw out the data. Like anything else, it’s buyer beware so do yourself and your clients a huge favor and invest some time in learning how this industry works.
After conducting thousands of studies over the past 20 years and working every major sample provider, Brookmark and OpinionRoute have learned quite a few things about this industry.
Partner versus Aggregator.
As previously stated, almost every sample provider aggregates from time to time. The truth is most proprietary panels are rather limited in their reach and quite often cannot complete a study without the support of other panels. Many sample houses with proprietary panels will say they have “partners” rather than refer to the term aggregate. Don’t be confused—using a partner is the same thing as aggregating. The key here is to make sure the partners have been vetted properly. If the sample management practices are consistent with what you determine to be acceptable, then you should be satisfied with the partner selection.
Proprietary Is Not Necessarily Better.
Several of the big sample companies boast owning their own panels. The truth is there is nothing proprietary about a panelist – just because someone subscribed to a panel does not mean they are in an exclusive relationship with that firm. The panel house may own the panel but they do not own the panelist. There is no labor contract or non-compete agreement. From a business operating standpoint, it also stands to reason that proprietary panels will often do everything they can to get as many completes possible out of their own panel (at high margins) before then scrambling to piece together the cheapest possible options available to find a way to get a job done. While many firms will argue this is not the case, practical experience suggests the bias is typically proprietary panel first and then partner sample to follow. As a result of this bias, it can be argued many proprietary panels have been misused, abused and completely exhausted in the pursuit of higher margins. It’s also conceivable, even probable, that the “partner” that is brought on second is a smaller enterprise with a more modestly used panel asset which has less bias stemming from the professional respondent.
Population Overlap Between Major Panels.
The ugly truth is the vast majority of panels “fish from the same pond”. That is, panelists are recruited from the same sorts of places (e.g., loyalty programs, corporate memberships, co-registration sites, etc.) resulting in many people signing up for multiple panels simultaneously. Industry figures suggest in major westernized regions, this overlap is incredibly high (perhaps up to 50% across certain panels) due to identical recruitment sources/approaches. Some panel firms have even aided others in their own panel recruitment. While there is nothing wrong with participating in multiple panels, it does present a problem when panel providers reach out to “partners” to help complete a study—some panelists may answer the same survey twice.
Validating The Data Quality Is The Key To Success.
When choosing sample partners, make sure they offer leading data verification and accuracy technologies. Most major panel companies will license third-party validation systems and/or integrate their own data validation systems. Firms such as Imperium offer digital fingerprinting services to identify duplicate respondents who take the same survey more than once from the same computer. TrueSample is another player offering data validation services including digital fingerprinting, third-party database validation and survey quality benchmarking. Given the significant population overlap in panels, it is critical to select sample vendors who have the resources and technology to validate your research data.
Have A Management Plan.
Can your sample partners use river sampling techniques? Can they pre-screen respondents? Are they routing? Are they blending? Are they direct emailing invites? The list goes on. You cannot afford to allow your sample vendor(s) to operate without detailed direction. Far too often, sample vendors are not proactively managed and blast sample as they see fit. This can result in several undesirable consequences including, but not limited to, bad data, missed quotas, lower incidences, missed deadline dates and increased costs. You need to be an active participant in the sample administration process to make sure data collection is done correctly.
In summary, everyone aggregates, so don’t get hung up on it. It’s more important to find someone you can trust, has innovative technology and has highly experienced project managers. Ask questions about their partners, demand transparency and proactively dictate the sample administration process.
Finally, if it’s truly an aggregators world then work with the best aggregators. They may have some proprietary panels or no panels at all but the most important thing is they know how to deliver the highest quality sample available today while making sure all technology involved supports, not compromises, your sample design.
About Brookmark Research:
Brookmark is a research operations company focused on managing the critical components of the research process including project management, survey design, programming, hosting, sample design & management, data management and reporting solutions. Brookmark provides a full outsource solution for all your research needs.