When to Text? How the Timing of Text Message Contacts in Mixed-Mode Surveys Impacts Response

Abstract

1. INTRODUCTION

Using multiple modes of contact is especially important in encouraging response to surveys and increasing response rates over a single contact mode (Dillman et al. 2014). As researchers look to new technologies to help curtail rising survey costs while maintaining high-quality, representative samples, texting will likely be a key contact mode in web and mixed-mode survey designs. Given that smartphone ownership has expanded greatly (Perrin 2021), and the use of texting is widespread, text messaging has become more feasible as a survey contact mode.

Text outreach can be an important survey innovation to reach people who may not respond via other contact modes and reduce the burden in responding to surveys. Texting a link to a web survey or a phone number to call can reduce respondent burden by providing a direct way to initiate a web or phone survey from the smartphone where the text is received, thus making it easier to respond. For many surveys, texting would likely not be used alone but in combination with other modes of outreach, such as mail, email, and outbound phone (i.e., interviewers making outgoing calls to sample members). However, there is little research on how best to sequence text messages with other modes of contact to maximize survey response and minimize nonresponse, while maintaining overall data quality.

Our research focuses on text messaging as an emerging contact mode in mixed-mode surveys and aims to address these gaps in the literature. This paper presents the design and results of a full factorial experiment that varies the sequencing of text messages with mailing contacts and the time of the day text messages were sent. The experiment was conducted in the second wave of a mixed-mode nationally representative longitudinal survey, the 2022 National Survey of Fishing, Hunting, and Wildlife-Associated Recreation (FHWAR). Data collection was conducted via web and inbound phone (i.e., interviewers receiving incoming calls from sample members), offering an ideal opportunity to research how best to sequence text messaging with other mail contacts in mixed-mode recruitment designs and whether the time a text message is sent has an impact on response.

Our research focuses on understanding whether sending a text reminder early versus later in the field period or sending text messages in the morning versus afternoon improves completion rates, response speed, and nonresponse, as well as how a text reminder compares to a text invite in these areas. The experimental design and analysis were stratified by Cooperative Respondents, those who completed the first survey wave after receiving a text message, and Other Respondents, a group that contained all other respondents who completed the first survey wave and screener respondents who did not respond to the first wave. The screener was a short survey sent to all address-based sample (ABS) members that contained questions about household members’ participation in fishing, hunting, and wildlife-watching activities, basic demographic characteristics, and contact information. Since all respondents participated in the screener, the design allowed for a robust nonresponse analysis between the screener and second-wave respondents. Our research also examines whether sending text messages has any negative impacts on web response time and item nonresponse rate, two recognized indicators of data quality (Groves 1989; de Leeuw et al. 2003; Andreadis 2021).

Our research questions include:

1. What is the association between sequencing of text messages and time of day texts were sent and the overall completion rate or the mode or device of response?

2. What is the association between sequencing of text messages and time of day texts were sent and the number of days respondents took to complete the survey?

3. What is the association between sequencing of text messages and time of day texts were sent and data quality, as measured by web survey response time and item nonresponse rates?

4. What is the association between sequencing of text messages and time of day texts were sent and sample composition and nonresponse?

2. BACKGROUND

2.1 Survey Response Theory and Mode Preference

Various theoretical perspectives have been developed to illuminate an individual’s decision making about whether to respond to a survey. These provide theoretical models for explaining participants’ internal psychology and cognitive processes in a way that can be leveraged by survey researchers to inform recruitment and contact strategies, including the use of text messaging. Our research is grounded in leverage-saliency and social exchange theory, as well as what is known presently from the survey research literature (Cook et al. 2000; Dillman et al. 2014; Dillman 2017; Daikeler et al. 2022; Cabrera-Álvarez and Lynn 2022; and others described below).

Leverage-saliency theory proposes that people place different levels of importance on various survey features and that survey designers can emphasize the survey features that might animate a response and deemphasize features that might discourage a response (Groves et al. 2000). From this perspective, text messaging may be a key survey feature that improves access for some sample members over traditional contact modes. Social exchange suggests that respondents are cognizant of the potential costs (e.g., expending time, mental effort) and benefits (e.g., answering interesting questions, helping others, receiving an incentive) of survey participation (Dillman et al. 2014). This means that survey researchers can promote cooperation by establishing trust with participants and communicating that the potential benefits of survey participation outweigh the costs (Dillman et al. 2014). From this perspective, texting may have a positive effect on participation by reducing burden and making it easier for an individual to respond, especially when the text invites the recipient to a web or phone survey that can be completed on the same device. However, texting could also have a negative effect if text messages are not viewed as legitimate or are considered a nuisance because of over-contacting.

Research on mixed-mode designs needs to consider how mode and device preferences may impact people’s willingness to respond. Survey researchers have identified multiple factors that can influence people’s response mode preferences (Olson et al. 2012; Smyth et al. 2014). We group and expand these factors into three broad areas: (i) media or device factors, including access, familiarity and ease of use, and perceived privacy and confidentiality related to the specific mode of contact or data collection; (ii) respondent factors, including time demands, external distractions, and concerns about presentation of self; and (iii) survey-specific factors, including perceived legitimacy, ease of visual and aural processing, and cognitive burden. People’s evaluation as to the importance of each factor will vary, influencing to different degrees whether they decide to respond to the survey request. The media and device factors may be especially relevant when exploring text outreach in surveys.

2.2 Literature on Mixed-Mode Contact and Recruitment

Utilizing multiple modes of contact, such as mail, email, text, telephone, and in-person, can be critical to getting potential participants to respond to a survey. Each mode has strengths and limitations, such as accessibility, cost, suitability for population(s) of interest, and technical and legal considerations. In general, the aim of pairing two or more contact modes is to reduce survey costs, reduce recruitment or collection time, and/or minimize total survey error (Dillman et al. 2014). Overall, recruitment strategies should be carefully crafted so contacts work together in a holistic manner, where each contact is connected with similar branding and visual design but has its own distinct approach, messaging, and plea so they appeal to different aspects that may encourage response.

Existing survey research literature has shown that, in general, the number of contacts is positively associated with response rate (Cook et al. 2000; Daikeler et al. 2022). The more invitations or contacts sent, the greater the response rate, with suggested maximum numbers that vary by mode (Cook et al. 2000; Lundquist and Särndal 2013; Moore et al. 2016). Multiple contact modes can be used to improve data quality by improving response rates and decreasing coverage error without increasing other sources of error (Dillman et al. 2014). Mixed-mode contact approaches have increased response rates and representativeness in general population studies (de Leeuw 2005; Beebe et al. 2005, 2012). Similar findings have been observed in certain special populations, such as in surveys of clinicians (Beebe et al. 2007, 2018; Dykema et al. 2013) and students (Millar and Dillman 2011; Patrick et al. 2022). However, a limitation of this literature is that the effects of multiple contact modes are often considered in conjunction with the effects of multiple data collection modes (e.g., Beebe et al. 2012). For example, Beebe et al. (2012) found that, while multiple contact and data collection modes were associated with increased response rates in a general population health survey, they did not necessarily reduce nonresponse bias. This suggests that survey researchers should continue to investigate the relationship between mixed-mode contact approaches and nonresponse, among other dimensions of data quality.

2.3 Literature on Text Messages as a Contact Mode

Texting is an emerging technology for survey contact. For the past several years, some survey researchers have gathered cell phone numbers and the consent to text survey participants, where the participant agrees to be texted for the purposes of the study. This approach has been used in panel and longitudinal study designs, so texting can be used for follow-up attempts to contact participants for future surveys. More recently, survey researchers are also using unconsented texting, where cell phone numbers are matched to addresses in address-based studies or used directly in listed samples or random digital dial studies, but the participant has not agreed to be texted for the purposes of the study. Depending on the approach used, various types of software can facilitate sending texts or more manual approaches can be used. Both approaches incur costs, such as staff time to write and send individual text messages or service fees for batch text message platforms.

The optimal time to introduce text messaging into a contact strategy is understudied, but existing research suggests that text messaging may be more impactful at the reminder stage, rather than at the prenotification or invitation stage (Bošnjak et al. 2008; DuBray 2013; De Bruijne and Wijnant 2014; Cabrera-Álvarez and Lynn 2022). One possible explanation, based on social exchange theory, is that text messages may not convey the same degree of trust or legitimacy as other contact modes, such as mail. Thus, text messaging may be most effective after some rapport has been established through other modes. Cabrera-Álvarez and Lynn (2022) compared the impact of adding text messages to various parts of the contact strategy for a mixed-mode survey of panelists in the United Kingdom. The authors found that a consented text message reminder led to the greatest increase in response rate, even more than when text messages were used for both the invitation and reminder. Two European studies, one with German students and the other with a panel in the Netherlands, found that consented text invitations did not improve response over email invitations (Bošnjak et al. 2008; De Bruijne and Wijnant 2014).

DuBray (2013) experimented with unconsented text prenotifications for the Centers for Disease Control and Prevention’s Behavioral Risk Factor Surveillance System telephone survey, finding no significant impact. In Australia, Dal Grande et al. (2016) also experimented with unconsented text prenotifications and found opposite results; respondents who received an unconsented text prenotification had a significantly higher response rate than those who did not.

Extant literature on text messaging is limited in certain respects. For example, existing studies tend to rely on non-experimental designs in surveys (Bošnjak et al. 2008; De Bruijne and Wijnant 2014; Cabrera-Álvarez and Lynn 2022) and many of these studies involve non-US samples (Bošnjak et al. 2008; De Bruijne and Wijnant 2014; Cabrera-Álvarez and Lynn 2022). We also note that few existing studies had the necessary sample sizes to look at effects of text messaging within subgroups, such as historically hard-to-survey groups. The collective literature finds promising associations with text contacts and survey completion, yet little in terms of conclusive evidence as to optimal text timing or frequency. Informed by leverage salience and social exchange survey response theories and research on factors that impact mode preferences, our study seeks to contribute new knowledge from an experiment to understand whether text reminders sent early versus later in the field period are more effective, and how they compare to a text invite, as well as whether the time of day text messages are sent impacts survey participation, response speed, and data quality. Our work furthers the research on how best to include text messaging with other contact modes in mixed-mode recruitment designs.

3. METHODS

3.1 Data

The experiment uses data collected by NORC at the University of Chicago for the 2022 National Survey of FHWAR. The FHWAR, conducted about every five years since 1955, is a nationally representative longitudinal survey sponsored by the US Fish and Wildlife Service that measures fishing, hunting, and wildlife-watching participation and expenditure by people over 16 years old living in the United States. The 2022 FHWAR consisted of a screener wave and three subsequent data collection waves. For each wave, sample members received a mailed letter with a $1 incentive directing them to complete the questionnaire online or over the phone with a live interviewer by calling a toll-free number. Nonrespondents received two reminder postcards and finally a self-administered paper questionnaire with a letter explaining the survey. In waves 1, 2, and 3, the research team experimented with adding text message and email outreach to this contact strategy.

The screener was administered from January 15 to April 15, 2022. The screener sample included an ABS sample and one drawn from AmeriSpeak, a nationally representative probability-based panel. The ABS sample was developed from the November 2021 US Postal Service’s Delivery Sequence File (DSF), including only city-style residential addresses and PO Box addresses that were flagged as Only Way to Get Mail (OWGM). Drop delivery and vacant households were removed. The ABS sample allowed for oversampling counties with high hunting participation identified by hunting license lists. The sample was stratified by state. The contact strategy for the screener varied depending on the sample source. AmeriSpeak panel members were contacted via web, text, and phone contacts, while ABS sample members received up to four contact attempts as described previously in addition to the text messages outlined in detail in section 3.2.

Table 1 shows the number of completes and response rates for the screener and the first two waves of the survey. In total, 42,340 households completed the screener. By mode, 32,928 (78 percent) households completed by web, 2,341 by phone (6 percent), and 7,071 by paper survey (17 percent). By sample source, 30,854 ABS sample members completed the screener (an American Association for Public Opinion Research (AAPOR) RR3 response rate of 11 percent), and 11,486 panel members completed the screener (an AAPOR RR3 response rate of 9 percent). The screener obtained household rostering and demographic characteristics, potential participation in fishing, hunting, and wildlife-watching activities, contact information, and a request to consent to receive text messages. Question wording for key screener items used in the analysis is provided in appendix A in the supplementary data online. Based on their screener data, household and household members were sampled into one of three groups for subsequent waves—fishing, hunting, or wildlife-watching (see table S1 in the supplementary data online for comparison of characteristics across the three groups). Questionnaires and outreach materials were tailored to each group.

Wave 1 data collection was conducted between May 4 and August 29, 2022. The wave 1 sample consisted of all households identified during screening and used the contact strategy described earlier. Wave 1 questionnaires asked respondents about activities they had participated in since January 1, 2022. In total, 16,609 surveys were completed in wave 1. This included 14,227 (86 percent) households, which completed by web, 620 (4 percent) by phone, and 1,762 (11 percent) by paper survey. By sample source, 10,403 ABS sample members (an AAPOR RR3 response rate of 4 percent) and 6,206 panel sample members (an AAPOR RR3 response rate of 8 percent) completed the wave 1 survey.

Wave 2 data collection, which is the focus of this paper, was conducted September 7 to December 23, 2022. A supplemental sample of panel members was also included in wave 2 to improve the precision of the results. In addition to the four contact attempts used in previous waves, sample members who had consented to receive text messages also received text messages as part of the contact strategy. This design is described in the following section. Wave 2 panelists could complete the survey via web and phone mode. For ABS sample members, web and inbound phone were offered concurrently in the initial mailings, with web mentioned in the text messages, and paper offered to nonrespondents later in the field period when the self-administered questionnaire was sent. Wave 2 questionnaires asked respondents about activities they had participated in since completing the wave 1 survey. In total, 16,968 surveys were completed in wave 2. This included 15,279 (90 percent) households that completed by web, 728 (4 percent) by phone, and 961 (6 percent) by paper survey. By sample source, 10,357 ABS sample members (an AAPOR RR3 response rate of 4 percent) and 6,611 panel members (an AAPOR RR3 response rate of 7 percent) completed the wave 2 survey. See appendix C in the supplementary data online for wave 2 Preferred Reporting Items for Complex Sample Survey Analysis (PRICSSA).

3.2 Experimental Design

During wave 2 of the 2022 FHWAR, text messaging was integrated as an additional contact mode for ABS sample members who had consented to receiving text messages during the screener. The text messages let the respondent know the survey was ready and provided a direct link to the web survey to make it easier for people to complete it. See appendix D in the supplementary data online for the text messages used.

The experiment consisted of varying the sequencing of text messages—whether the text was sent as an invitation, early reminder, or late reminder—as well as the time of day the text messages were sent, either morning or afternoon. Morning text messages were sent at 10:00 am Central Time and afternoon text messages were sent at 5:00 pm Central Time. These times were chosen based on their appropriateness across major US time zones (i.e., not too early or late for respondents in Eastern or Pacific time zones) while still delineating two discrete periods of the day (i.e., morning and afternoon). The sample was divided into two groups based on their response to wave 1. The Cooperative Respondents group consisted of respondents who completed wave 1 after receiving a text message (n = 2,315). The Other Respondents group was much larger (n = 15,694) and consisted of the remaining wave 1 completes, as well as screener respondents who did not respond in wave 1. Cooperative Respondents were older, less likely to be Hispanic, more likely to be Black, more likely to have participated in fishing, and lived in smaller households compared to Other Respondents (see table S2 in the supplementary data online for comparison of characteristics across the two groups).

Text messages were sent in batches using Twilio, a communications platform. Note that not all text messages were delivered successfully: 1,011 texts went undelivered through blocking by a phone company and Twilio stopped service because of the high number of STOP requests received. Part of the reason for the high number of STOP requests may have been the length of time between when respondents gave consent and when they received the first text message. The blocking affected all experimental conditions equally and is an important operational consideration when sending text messages for survey contacts. Few differences were found between those with blocked texts and those who received texts, where Black respondents were less likely to be blocked (⁠$χ23=7.87, p=.05$⁠) and those from single-person households were blocked at higher rates (⁠$χ23=12.69, p=.005$⁠). Sample members with undelivered text messages were excluded from the subsequent data analysis.

As shown in table 2, Cooperative Respondents were randomly assigned to one of the four conditions formed by crossing two factors—sequencing (early text reminder vesus late text reminder) and time of text message (morning versus afternoon)—in a 2×2 factorial design. Other Respondents were randomly assigned to one of the six conditions formed by crossing two factors—sequencing (text invitation versus early text reminder versus late text reminder) and time of text message (morning versus afternoon)—in a 3×2 factorial design. Day of the week was not part of the experimental design as we wanted to text on specific days that best fit within the overall contact schedule.

Table 3 shows the contact attempts for each group. All sample members received the standard FHWAR invitation letter with a $1 incentive provided in the envelope. Nonrespondents received two postcard reminders and a self-administered questionnaire with a letter explaining the survey. The Cooperative Respondents group received an additional two text message contacts (a text invitation before the letter invitation and a text reminder) for a total of six contacts. The timing of the text reminders varied, with the early text reminder group receiving the reminder between the invitation letter and the first postcard and the late text reminder group receiving the reminder between the first and second postcards. The Other Respondents group received one text message contact in addition to the standard FHWAR contact strategy, for a total of five contacts. As with the Cooperative Respondents group, the timing of the text messages for the Other Respondents group varied. The text invite group received a text message before the letter invitation, the early text reminder group received a text between the letter invitation and first postcard, and the late text reminder received a text after the final mailing.

3.3 Methodological and Statistical Approach

In this study, we explored the effects of the sequencing (text invite, early text reminder versus late text reminder) and time of day that text messages were sent (morning versus afternoon) on completion rates, response speed, data quality, and nonresponse.

In these formulas, $P^c$ is the proportion of completes for condition c, $Var^(P^c)$ is the variance of the estimate, and n is the total sample allocated.

We also examined the effects of the sequencing and time conditions on the response speed in terms of days to complete the web survey or paper questionnaire. The days to completion were analyzed via unweighted t-tests. In addition, we compared the conditions on data quality measures, including web response time (i.e., the number of minutes it took for the respondent to complete the web survey) and item nonresponse for web completes and paper completes (unweighted). The durations for the web response times were trimmed so that respondents with a response time in the lower 1 percent of the distribution were set to be 1 percent and respondents with a response time in the upper 1 percent of the distribution were set to be 99 percent to diminish the effects of extreme values on analysis when conducting unweighted t-tests. Rates of item nonresponse were calculated as the number of survey items for which the respondent did not provide a valid response category, either through a “Don’t Know,” “Refused,” or “Skip,” over the total number of survey items. Rates of item nonresponse were calculated for all questions a respondent was asked in the survey, which ranged from 29 questions up to 218 questions, with 46 questions asked on average. Finally, sample composition was compared for each condition by analyzing distributions of demographics and key survey indicators and testing for significance using chi-squared tests.

4. RESULTS

4.1 Completion Rates

AAPOR response rates could not be calculated from a lack of detailed dispositions from the text messages. Table 4 presents the web, paper, phone, and overall completion rates by sequencing condition. Among Cooperative Respondents, there were no statistically significant differences between the early and late text reminder groups on web, paper, phone, or overall completion rates. Similarly, among Other Respondents, there were no statistically significant differences between the text invite, early text reminder, or late text reminder groups on web, paper, phone, or overall completion rates. However, those who received the text invite or early text reminder were more likely to complete the web survey using a smartphone than those who received the late text reminder (⁠$χ22=36.38, p<.0001$⁠). As expected, completion rates were significantly higher for Cooperative Respondents than for Other Respondents as the Cooperative Respondents had a history of responding after receiving a text in wave 1.

Table 5 presents the web, paper, phone, and overall completion rates by time of text message. There were no statistically significant differences between text messages sent in the morning and afternoon on completion rates for Cooperative Respondents or Other Respondents. There were also no significant differences on the device used to complete the web survey by time of texts.

Figure 1 shows the base-weighted cumulative overall completion rates by week for each condition. The solid vertical lines represent the text invitation, early text reminder, and late text reminder, respectively, and the dotted lines represent the four mailings to sampled persons, including the letter invitation, two postcard reminders, and the letter and paper questionnaire.

Figure 1.

Overall Cumulative Completion Rates by Contact for Sequencing Conditions.

Open in new tabDownload slide

Regarding Cooperative Respondents, as shown in figure 1, the completion rate at week 1 was similar between the early and late text reminder groups since both groups received the text invitation, but diverged in weeks 3 and 4, when more sampled persons who had received the early text reminder responded. The early text reminder group had a significantly higher completion rate at week 4 than the late text reminder group (⁠$χ21=4.11, p=.04$⁠). The difference diminished and became non-significant after those in the late text reminder group received their text reminder. For the rest of the field period, the early text reminder group had a slightly higher completion rate though the difference was not statistically significant. For this group, the text reminders seemed to be more effective than the text invitation at increasing completion rates, with the early text reminder being somewhat more effective than the late text reminder, at least early in the field period. The combination of mail contacts and texting led to a consistent increase in response after each contact. Since all groups received the same mailings, differences observed for weeks when a text was sent to one group and not another can likely be attributed to texting.

Regarding Other Respondents, as shown in figure 1, the text invite group had a significantly higher completion rate than the early and late text reminder groups at week 1 (⁠$χ22=177.08, p<.0001$⁠). However, the pattern changed quickly after the letter invitation was mailed. The early text reminder group continued to have higher completion rates than the other two groups for the rest of the field period. Particularly, the early text reminder group had a significantly higher completion rate than the Text Invite group (⁠$χ21=11.37, p=.0007$⁠) at week 2. The differences between these groups become nonsignificant afterward, with the early text reminder group having slightly higher completion rates than the text invite group. The late text reminder group had significantly lower completion rates than both other groups from week 2 to 8, suggesting that including an additional contact such a via text message is more effective than no text contacts with this group. The differences between the three conditions diminished after sample members received the late text reminder in that group.

The findings suggest that a text invitation is effective at improving response, but its effects diminish quickly. An early text reminder appears to be more effective than a late text reminder at increasing completion rates. Though not tested specifically, it seems that the most effective contact strategy would include the text invitation, the early text reminder, and the late text reminder, in addition to the mail contacts.

4.2 Response Speed

We also examined how quickly respondents completed the survey. Table 6 presents the mean number of days to complete the web and paper surveys. For Cooperative Respondents, there were no significant differences between the early and late text reminder groups. For Other Respondents, however, there were significant differences on the mean number of days to complete the web survey (⁠$F2, 5713=88.16, p<.0001$⁠), as well as the mean number of days to complete the paper survey (⁠$F2, 465=116.99 p<.0001$⁠). The mean number of days to complete the web survey for the early text reminder group was 13.4 days as compared to 18.2 days for the text invite group and 21 days for the late text reminder group. Respondents in the early text reminder group also completed the paper survey faster than those in the text invite group. These findings suggest that the early text reminder was the most effective in terms of getting sample members to complete the survey request as quickly as possible.

Table 6 also includes the mean number of days to complete the web and paper survey by time of text message. For Cooperative Respondents, there were no significant differences between the early and late text reminder groups on the mean number of days to complete the web or paper survey. For Other Respondents, those who received the text message in the afternoon completed the web survey a day faster than respondents who received the text message in the morning (⁠$t5684.4=2.63, p=.009$⁠). There were no significant differences on the mean number of days to complete the paper survey between morning and afternoon.

4.3 Data Quality

We examined two data quality measures: item nonresponse rates for web and paper completes and response time for web completes. If sequencing or time of text messaging are negatively associated with data quality through either higher rates of item nonresponse or speeding, it would suggest that any benefits of text messaging may need to be balanced with their impact on data quality. As shown in table 7, neither sequencing nor time of text messaging had significant effects on web response time or item nonresponse to the web or paper modes. These results are encouraging in that using text as a contact mode does not appear to impact these data quality measures.

4.4 Sample Composition and Nonresponse

To explore the effects of sequencing and time of text messaging on nonresponse, we compared the sample composition across conditions for demographic and selected survey variables. Overall, the sample composition was very similar with only a few significant differences found when comparing conditions (see tables S3–S6 in the supplementary data online). The oldest age group, 55 and older, made up a higher proportion of completes for the early text reminder groups for both Cooperative and Other Respondents. The Other Respondents also had a higher proportion of African American or Black respondents in the afternoon text group. Importantly, no significant differences were found in the key substantive indicators in the study, participation in hunting, fishing, or wildlife activities, implying these minimal demographic differences did not affect survey results.

5. CONCLUSIONS AND DISCUSSION

To explore the effectiveness of texting as a mode of contact, we conducted an experiment that varied when text messages were sent during the study period and the time of day the text messages were sent to examine effects on response rates, response speed, data quality, and nonresponse. We found that the text invite did create an initial increase in response, but its effects diminished quickly. This lends some evidence to the hypothesis based on social exchange theory that electronic contact strategies (i.e., email and text) may be more trusted and attended to after a postal letter that establishes the legitimacy of the research ask (Dillman et al. 2014), even in longitudinal studies like this one where respondents previously consented to receiving text messages. An early text reminder was more effective than a late text reminder and raised the completion rate early in the field period, reducing the sample that needed to be sent additional follow-up mailings. The early text reminder also resulted in survey participants completing the survey more quickly. The mean number of days to complete the web survey was lower for those that received the early text reminder than those who were sent the text invite or the late text reminder.

Some of the benefits of the text reminder may arise because additional contacts, regardless of mode, tend to improve overall response rates. It is possible that a similar effect could have been observed by sending an additional mail contact. However, texting also provides additional benefits that additional mail contacts may not, including: (i) text messages can be sent at a lower cost than a mailed letter or postcard, (ii) text messages may be attended to by people who may not receive or read mail at their address, and (iii) text messages can encourage a more timely response, especially when sample members are provided a direct link to a web survey that they can instantly complete on their own.

The early and late text reminders not only helped encourage web response but also helped increase response rates to the paper survey, relative to the text invitation. These findings are consistent with other studies that found contact via one mode can help drive response to the survey by another mode (Dillman et al. 2014; Dillman 2017). The mean number of days to complete the paper survey was also about 10 days shorter for those who received the text reminder compared with the text invite.

Sample composition did not significantly vary across experimental conditions, indicating that including text message contacts did not significantly impact the overall demographic representation of the completed sample. Data quality measures, including mean response time on the web and item nonresponse for web and paper did not significantly vary across the conditions, suggesting that the text reminders did not negatively impact data quality. Sending the text in the morning versus afternoon did not have a significant impact on overall response, nonresponse, or other data quality measures, though sample members in the Other Respondents group were more likely to complete the survey a day faster when sent afternoon texts rather than morning texts.

There are multiple limitations of this study that should be explored through additional research and experimentation. First, some form of texting was used in each of our conditions because we found texting had been helpful in wave 1, so we did not test a group that did not receive a text invitation or reminder. This could be important for future studies to further understand potential benefits of texting versus no texting. Similarly, we did not directly test a group that received both a text invitation and reminder or multiple reminders, which we see as important for future experiments. Further, this experiment was conducted within a longitudinal study where consent to text was provided as part of the initial screener response. People may be more receptive to texts after they have already participated in an initial study than when they have not previously participated or consented to receive text messages. Future research should explore the use of unconsented texting in cross-sectional survey designs. Further studies could also explore whether certain days of week are better for sending text messages than others. Finally, the text messages in this experiment focused on encouraging web response and used specific messaging for this study. Future research could explore encouraging response to other modes, such as calling in to complete the survey with an interviewer and experimenting with the messaging used to encourage respondents to complete the survey.

Overall, our research suggests that text messages can be an important contact mode in mixed-mode survey designs. Sending text messages for survey reminders is likely more effective than sending a text invitation, especially for studies where no other outreach has been made to establish trust and legitimacy, and text messaging does not appear to negatively impact data quality. Further research should compare a contact strategy with texting versus no texting, examine whether sending multiple text reminders improves response more than a single text reminder, and experiment with different message content and strategies, including the use of unconsented texting in cross-sectional study designs. As text messaging continues to grow as a contact mode for mixed-mode study designs, it will be important to understand how to best integrate texting with other contact modes and how these design choices impact response in different survey populations and study designs.

SUPPLEMENTARY MATERIALS

Supplementary materials are available online at academic.oup.com/jssam.

The authors would like to thank our NORC at the University of Chicago colleagues David Sterrett and Emily Alvarez for their expertise on the National Survey of Fishing, Hunting, and Wildlife-Associated Recreation data collection, Kate Hobson for guidance on text message implementation, Semilla Stripp for support with analysis, and Nola du Toit for developing the data visualizations. This study design and analysis was not preregistered in an independent, institutional registry.

This work was supported by NORC at the University of Chicago as an internal research and development initiative. The experiment uses data collected by NORC at the University of Chicago for the U.S. Fish and Wildlife Service [F20AP00134].

REFERENCES