Ongoing Refinement and Adaptation are Required to Address Participant Deception in Online Nicotine and Tobacco Research Studies

Online research offers many benefits to nicotine and tobacco research, including the ability to conveniently and cheaply recruit large numbers of participants over a wide geographic area. However, online research also introduces an increased risk of participant deception and fraud.^1–3 Heffner et al. introduced valuable techniques for detecting fraud that can be applied to a wide variety of study protocols.³ We offer an example of detecting and responding to participant deception and adapting to changing patterns of fraud in an online cohort study of adolescents and young adults.

The goal of the Policy and Communication Evaluation (PACE) Vermont Study is to understand the impact of state-level policies and communication campaigns on substance use beliefs and behaviors in young Vermonters.⁴ Study measures and findings are aimed at informing state-level prevention programming.^5,6 Using a cohort design, participants aged 12–25 years are recruited via multiple methods (including online advertisements) to visit a study website and complete online eligibility screening and informed consent.⁴ Between 2020 and 2021, three periods of enrollment were open during which participants were screened through Qualtrics surveys, and then manually screened by a member of our study team. Automated survey features were employed within Qualtrics to prevent multiple screener attempts, identify potential duplicates, and detect bots⁷; screeners also included CAPTCHA verification and one of three simple math problems which served as an attention check and were randomized to ensure that the same response was not copied across surveys. In consultation with our IRB, we included language in our online informed consent about discontinuing participants for providing information in the surveys that we believed to be intentionally falsified. The systematic method of screening participants for deception developed in our pilot study⁴ was revised and refined in subsequent waves based on changes in participant response patterns.

Beyond the multiple checks built into the screening surveys themselves, we employ a multiphase screening process to reduce participant deception in our PACE Vermont cohort: First, participants complete a brief interest survey to determine initial eligibility and obtain contact information. Second, participants meeting the initial eligibility criteria receive a screener survey via text message or email to ensure valid contact information; youth participants require an additional parent survey which is linked to their response by a common identifier and provides consent for them to participate in the study. Third, eligible participants complete a payment form that includes their mailing address. Finally, the research staff creates a unified dataset across these forms to conduct a manual assessment of eligibility; coding within this dataset verifies age based on the date of birth provided and location based on IP address. Manual checks confirm eligibility, consent, and completion of the payment form as well as flagging participants based on the following criteria: (1) age (eg, providing a birth date that differs from self-reported age), (2) location outside Vermont (eg, IP address, mailing address), (3) suspicious contact information (eg, email addresses that include only numbers or names that do not correspond to contact information), and (4) inconsistency of name, location, or contact information across forms. Participants are informed about study eligibility via email after all checks are complete.

Additional manual checks were introduced in later waves. In Fall 2020, we noticed a number of respondents reporting mailing addresses that corresponded to homes listed for sale on Zillow. Hypothesizing that “bots” may be pulling valid Vermont addresses from for-sale listings, we decided to flag these participants in our fall 2020 and spring 2021 screenings. Manual checks were reviewed regularly by the study team and specific cases were resolved by discussion. After all, checks were completed, potentially deceptive participants were flagged and received an email from the study team offering the opportunity to confirm their contact information and remain in the study. Respondents who did not respond via email or whose email responses employed strange formatting, language, or did not confirm the correct contact information were deemed ineligible and removed from the study. In some cases, discrepancies in responses for youth participants were related to their lack of knowledge (eg, county of residence) and we received messages from a parent about their eligibility; we used the parent screening survey to verify the parent’s contact information and connection to the participant before deeming the participant valid and eligible.

During each enrollment period, a high proportion of our total screened respondents were determined to be ineligible (49%–67%; Table 1). However, only a fraction of likely deceptive respondents have screened out automatically through Qualtrics, either because of responses to screening questions or Qualtrics’ Expert Review Fraud Detection features (10%–46%).⁷ Most ineligible, likely deceptive respondents were screened out manually using a process that evolved at each recruitment period (52%–90%).

Our experience highlights the importance of flexible, dynamic methods of fraud detection that respond to “bot learning” and adaptation by ineligible participants. The rates of specific types of suspicious responses differed across waves of recruitment for the same cohort study, and new methods of screening were added in response to emerging patterns in the later waves. For example, the majority of ineligible respondents in summer 2020 were screened out using IP location (91%), which showed that participants were not located within the geographic bounds of the state. In later waves, this type of fraud was less common than it had been in Summer 2020 (Fall 2020: 15%, Spring 2021: 32%), while suspicious patterns in reported email addresses increased (Fall 2020: 15%, Spring 2021: 42%). In Fall 2020, 40% of participants screened out manually had addresses listed for sale on Zillow compared with 9% in Spring 2021. No single indicator of fraud was consistently useful. Instead, it was more valuable to organize screening data by response time and look for groups of responses that were unusual in similar ways and completed at roughly the same time.

Given the likelihood of both false positives and false negatives in the detection process, we conducted an ongoing assessment of the validity of survey responses and removed participants in the midst of data collection if we identified them as suspicious or deceptive. In the process of analyzing open-ended responses from our Winter 2020 survey, for example, the study team noticed patterns of unusual, nearly identical responses submitted at the same time by different respondents, some of whom had participated in multiple waves of the study. As with our recruitment approach, we contacted participants with this pattern of responses to offer them a chance to remain enrolled in the study. Few of these participants replied to the study team, and those who used a similar, unusual email structure and phrasing. As a result of these additional checks, we withdrew 75 participants from the cohort (8% of the sample) prior to the spring survey. We retain the information for these participants within our full contact list—flagged as withdrawn—to allow us to identify potential duplicates or deceptive participants in future waves of recruitment.

Our experience indicates that deception in online tobacco research is pervasive, nearer to the high estimates of previous studies,^2,3 difficult to detect with any single method, and not adequately captured by automated processes built into survey platforms. Our PACE Vermont methodology and study infrastructure have been replicated in New Jersey young adults,⁸ with consistent findings across states: 71% (4,242/5,955) of our PACE New Jersey cohort screenings were deemed ineligible and half of these ineligible respondents were identified manually (52%). Screening for participant deception is a time-intensive process that extends beyond initial screening and requires vigilant data checking at other time points. Moreover, no single method will filter out all deceptive respondents, and the methods that are used need to be responsive to the study design, participant experience, and changes in patterns of respondent behavior. Researchers designing online studies should take care to develop in-depth methods of manual fraud detection and regularly revise and evolve these methods as the study progresses to ensure data integrity. Publishing the methods used to address participant deception in online studies runs the risk of stimulating new forms of deception used to gain entry into research studies; however, this risk is balanced by the need in our field to ensure transparency and to continue to evolve and innovate our own best-practice anti-deception methods.³

Implications

Deception in online nicotine and tobacco research is pervasive, difficult to detect with any single method, and not adequately captured by automated processes built into survey platforms. Manual screening of data at recruitment and throughout data collection, the continual evolution of data quality checks, and attention to patterns of response require significant time and staffing to ensure data quality in online studies.

Funding

Research reported in this publication was supported by the National Institute on Drug Abuse (NIDA) of the National Institutes of Health under Award Number R21DA051943, the National Cancer Institute (NCI), and Food and Drug Administration (FDA) Center for Tobacco Products (CTP) under U54CA229973, and a contract from the New Jersey Department of Health (NJDOH).

Acknowledgments

The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or the State of New Jersey.

Conflict of Interest

The authors have no conflicts to disclose.

References