A New Question Sequence to Measure Voter Turnout in Telephone Surveys: Results of an Experiment in the 2006 ANES Pilot Study

Abstract

Survey respondents typically report having voted at a rate higher than the nation in fact turned out on election day. This may be the result of errors people make when trying to remember whether they voted and of motivated misreporting due to social desirability bias. This paper explores whether a new sequence of questions designed to reduce both types of errors reduces reports of turnout in telephone interviews. An experiment embedded in the 2006 American National Election Studies Pilot Study indicates that the question sequence did do so, and that it resulted in turnout estimates more consistent with official records than did the simple, direct traditional ANES question.

Introduction

Post-election surveys typically yield higher voter turnout estimates than do government records (e.g., Clausen 1968; Traugott and Katosh 1979), and validation studies suggest that some respondents claimed to have voted when county election records showing voter registrations and turnout suggest that they did not (e.g., Clausen 1968; Parry and Crossley 1950; Traugott and Katosh 1979). Higher estimates of turnout from surveys may result from (1) missing government records for respondents who voted; (2) lower survey participation among nonvoters than among voters; (3) increased turnout caused by interviewing respondents before an election and thereby boosting their interest in politics and/or their sense of civic duty; (4) errors in methods used to calculate rates of actual turnout, such as using the voting age population as the denominator rather than the voting-eligible population (Burden 2000; Clausen 1968; McDonald and Popkin 2001; Presser, Traugott, and Traugott1990; Traugott 1989); (5) misreporting by respondents motivated to portray themselves favorably (Aarts 2002; Andolina et al. 2003; Blais et al. 2004; Brockington and Karp 2002; Holbrook, Green, and Krosnick 2003; Holbrook and Krosnick 2010; Lutz 2003; Lyons and Scheb 1999; although see Locander, Sudman, and Bradburn 1976; Presser 1990); and (6) accidentally misremembering having voted when one did not (Belli et al. 1999; Belli, Moore, and Van Hoewyk 2006; Stocké 2007; although see Abelson, Loftus, and Greenwald 1992; Belli, Traugott, and Rosenstone 1994).

Past studies have explored whether these last two factors can be reduced by replacing the simple direct question (SDQ) asked in many past American National Election Studies (ANES) surveys with the short or long “experimental questions” shown in table 1. These experiments have sometimes found that the experimental questions yielded lower turnout estimates than did the SDQ, but not always (see table 2).¹

The current study assessed whether a new multi-question sequence (MQS; see the bottom of table 1) lowers turnout estimates in telephone interviews. This approach sought to achieve the same goals as prior experimental questions but with design improvements, based on the following logic:

• 1) Some respondents may feel a societal expectation that they vote, so during a survey (especially an oral interview), such respondents who did not vote may prefer not to acknowledge their failure to conform to this social norm and be motivated to misreport having voted (see Karp and Brockington 2005).

• 2) Allowing respondents to report that they intended to vote in the most recent election and/or that they often voted in previous elections may allow people to convey an image of being responsible citizens and therefore to feel less pressure to report that they voted when they did not.

• 3) Respondents who intended to vote in the most recent election but did not, and respondents who voted in previous elections but did not vote in the most recent one, might experience “source confusion,” mislabel memories of their intentions or prior behaviors as actually voting in the most recent election (e.g., Johnson, Hashtroudi, and Lindsay 1993; see also Hill and Hurley 1984, c.f. Presser and Traugott 1992), and misreport having voted when they did not.

• 4) Source confusion may be more likely as the time interval between the election and the survey interview grows (because memories may become less vivid over time),² and more common among people who are interested in politics (who are especially likely to have intended to vote and to have voted often in the past).³

• 5) Inducing respondents to think about whether they intended to vote in the most recent election and whether they voted in previous elections may help respondents separate their memories of intentions and past behavior from their memories of behavior in the most recent election, thereby minimizing source confusion and increasing reporting accuracy.

• 6) It is undesirable to offer answer choices that are not mutually exclusive when asking closed-ended questions that require respondents to select just one answer (Bradburn, Sudman, and Wasink 2004). Respondents who thought about voting but did not and who usually voted in past elections could legitimately select any of three of the answer choices offered by the long or short experimental questions, and it is not clear which of these the respondents should choose.

• 7) Offering double-barreled answer choices (such as the second and third answer choices in the long and short experimental questions) should be avoided, because some respondents may find themselves wanting to endorse part of an answer choice but not another part of it.

• 8) Any question measuring participation in an election that does not explicitly assess “early voting” done before election day may fail to accurately assess turnout, since this is increasingly common (Gronke, Galanes-Rosenbuam, and Miller 2007).

• 9) To reduce cognitive burden and administration time, it is preferable to decompose a long, complex, and cognitively difficult question into a series of short, easy-to-answer questions.

The MQS (1) decomposed the long experimental question into several shorter questions to reduce cognitive burden, (2) allowed respondents to report one attribute at a time, and (3) offered mutually exclusive and exhaustive sets of answer choices. The MQS was intended to alert respondents to the possibility of memory errors, encourage respondents to think carefully when answering, and reduce social desirability pressures by providing respondents with the opportunity to report that they usually voted and/or that they had planned to vote (Belli et al. 1999). Finally, the MQS allowed respondents to report that they voted through absentee or early voting and to report that they were not sure whether they voted.

We tested whether the MQS reduced estimates of turnout relative to the SDQ, whether the MQS was especially effective at reducing turnout reports from people who were interviewed a long time after the election and who were most involved in politics (measured using past voting behavior, strength of party identification, political efficacy, and an index of attention to and interest in politics), and whether the MQS produced more accurate estimates of turnout rates in various states.

Data

This test was carried out in the ANES 2006 Pilot Study. An area probability sample of 1,211U.S. adults eligible to vote was interviewed face-to-face before the 2004 presidential election (RR = 66.1 percent),⁴ and 1,066 of these people were reinterviewed after that election (reinterview rate = 88.0 percent). Between November 13, 2006, and January 7, 2007, 675 of the respondents from the 2004 pre-election wave were reinterviewed by telephone (reinterview rate = 56.3 percent; eleven deceased respondents were excluded; cumulative response rate = 37.2 percent).

The 675 respondents interviewed in 2006 were randomly assigned to be asked either the SDQ or the MQS (see Supplementary Data, the Appendix in this article, and the online appendix [supplementary data online] for all question wordings, coding of variables, and distributions of responses to the MQS).⁵

Results

The two groups of respondents who were randomly assigned to be asked the SDQ or the MQS did not differ significantly in terms of six demographics and nine political attributes (see the online appendix). However, compared with the people who were asked the SDQ, people who were asked the MQS were significantly more likely to have said that they did not vote in 2000 (26.8 percent versus 19.4 percent), more likely to be Hispanic (5.6 percent versus 1.8 percent), and less likely to be White (75.2 percent versus 81.9 percent).⁶ We therefore controlled for these variables in our analysis.

Of respondents who were asked the SDQ, 81 percent said they voted, whereas 72 percent of respondents who were asked the MQS said so, a significant difference (b = –0.50, p < .01; see equation I in table 3).⁷ This effect was marginally significant when controlling for turnout in 2000, race, and ethnicity (b = –0.37, p = 0.06; see equation II in table 3), and maintained its strength when controlling for all demographic variables (b = –0.48, p < .05; see equation III in table 3).

Among the respondents who were asked the MQS and who said that they did not vote in 2006, 42 percent said that they had usually voted in the past, and 52 percent said that they had planned to vote in 2006. These sizeable numbers are consistent with the argument that the MQS reduced turnout reports by allowing respondents to report having performed admirable behaviors and/or by helping respondents avoid source confusion.

We next tested whether the question-wording effect was stronger among respondents more likely to experience memory errors. None of the moderators we tested significantly interacted with the question-wording manipulation to influence turnout reports (time between election day and the survey interview: b = –0.03, SE = 0.76, n.s.; an index of interest in and attention to politics: b = 0.18, SE = 0.91, n.s.; reported turnout in 2000: b = 0.31, SE = 0.48, n.s.; strength of party identification: b = 0.48, SE = 0.63, n.s.; political efficacy: b = 0.61, SE = 1.03, n.s.).⁸ Furthermore, the seven demographic variables did not moderate the question-wording effect (see the online appendix [supplementary data online]).

To compare the accuracy of the MQS and the SDQ, we computed the association of turnout estimates yielded by each method with the actual turnout rates in twenty-seven states with sufficient survey sample sizes (obtained from the United States Election Project, elections.gmu.edu; see table 4). The average absolute difference between the state turnout estimate generated using the SDQ responses and the actual turnout rate for that state was significantly larger than the average absolute difference between state turnout estimate produced by the MQS and the actual turnout rate (40.27 versus 26.13 percentage points; t(26) = 4.97, p = .001). The SDQ yielded a higher turnout rate than the MQS for twenty of the twenty-seven states (74 percent), significantly more than would be expected by chance alone (p = .01). Furthermore, the correlation of actual state-level turnout rates with survey state-level estimates of turnout was substantially stronger using the MQS (r = 0.45) than when using the SDQ (r = 0.28). These state-by-state results constitute a series of replications and are consistent with the hypothesis that the MQS yielded more accurate measurements than did the SDQ. However, the 2004 ANES sample was not designed to yield a representative sample of the population of each state, and the numbers of respondents interviewed in most states were very small, so this analysis should be viewed with caution.

Discussion

These results suggest that during the 2006 telephone interviews, turnout reports were more accurate when obtained with the new MQS than when obtained using the SDQ. The fact that the question-wording manipulation did not have more impact among people interviewed longer after the election or among respondents higher in political involvement (who are more likely to experience source confusion) is consistent with several past studies (Belli et al. 1999; Belli, Moore, and Van Hoewyk 2006; Duff et al. 2007)⁹ and suggests that the MQS did not reduce reported turnout by reducing memory errors relative to the SDQ. Therefore, future research should explore the mechanism for the principal effect observed here.

All past studies showing that experimental question wordings reduced reported turnout (relative to the SDQ) have involved telephone interviewing (Belli et al. 1999; Belli, Moore, and Van Hoewyk 2006; Duff et al. 2007), as did ours. Most past ANES surveys have involved face-to-face interviewing, so we should hesitate before concluding that the ANES face-to-face interviewing would benefit from use of the new MQS. Specifically, some past studies suggest that relative to telephone interviewing, face-to-face interviewing entails less social desirability response bias and inspires more cognitive effort to generate accurate answers (e.g., de Leeuw and van der Zouwen 1988; Holbrook, Green, and Krosnick 2003; Johnson, Hougland, and Clayton 1989; c.f. Colombotos 1965; Rogers 1976; Wiseman 1972). Therefore, if the new MQS did improve the accuracy of telephone turnout reports by reducing social desirability pressures and/or source confusion, the same benefits may not be observed during face-to-face data collection. Future research is needed to explore this.

Turnout estimates from the MQS were still consistently higher than official rates, perhaps because the MQS did not completely eliminate memory errors and/or social desirability bias. This overestimation may also have occurred for other reasons, such as greater nonresponse among non-voters than among voters and higher real turnout rates among respondents due to pre-election interviews (Greenwald et al. 1987; Traugott and Katosh 1979; though see Greenwald et al. 1988; Mann 2005; Smith, Gerber, and Orlich 2003).

Two additional caveats to the present findings:

• – These data did not permit testing whether the MQS reduced turnout reporting relative to the SDQ for each individual respondent, and it is possible that the MQS may have caused underreporting of turnout by some respondents.¹⁰

• – The study provides no direct evidence regarding the mechanism by which the MQS reduced reported turnout rates.

Conclusion

Because the MQS yielded lower estimates of turnout than did the SDQ and also reduced cognitive burden and obtained more information about respondents’ voting behavior (e.g., past voting behavior, voting intentions, and early and absentee voting), the new series of questions seems to be a viable alternative to the SDQ and a possible alternative to previous experimental questions (although we could not test this directly). We look forward to future studies comparing the new multi-question sequence to other experimental questions.

Appendix: Question Wordings and Coding of Variables for Core Survey Variables

Some of the measures we used were administered during the 2006 interviews, and others were administered during the 2004 pre- or post-election interviews. Data from the 2004 and 2006 interviews were linked using respondents’ case ID numbers. The year in which a variable was measured is indicated in parentheses after the variable name, along with the variable name from the ANES codebook. Unless otherwise specified, respondents with missing data on a given variable were excluded from analyses in which that variable was used.

Core Survey Variables

Turnout question version (2006): In the survey, 336 respondents, selected randomly, were asked the ANES simple direct turnout question (see row 1 of table 1; v06p776). The other 339 respondents were asked a series of questions measuring turnout (see row 4 of table 1; v06p777–v06p779b). Using answers to these questions, turnout in 2006 was coded 1 for respondents who said they voted (in response to the simple direct question) or were sure they voted (in response to the new questions) and 0 for respondents who said they did not.¹¹ A variable was coded 1 for respondents who were asked the new questions and 0 for respondents who were asked the simple direct question.

Past voting behavior (2004): Respondents were asked, “In 2000, Al Gore ran on the Democratic ticket against George W. Bush for the Republicans and Ralph Nader as the Reform Party candidate. Do you remember for sure whether or not you voted in that election?”[v043002]¹² People who said they had voted were coded 1, and people who said they had not were coded 0.¹³

Days since the election (2006): The numbers of days between the election and the respondent’s 2006 post-election interview was coded to range from 0 (for interviews done six days after the election, the shortest observed time interval) to 1 (for interviews done fifty-nine days after the election, the longest interval). [v06p201a and v06p201b]

Political involvement (2006): Half of the respondents were randomly assigned to be asked the following three questions: (1) “How interested are you in information about what’s going on in government and politics? Extremely interested, very interested, moderately interested, slightly interested, or not interested at all?” [v06p630] (2) How closely do you pay attention to information about what’s going on in government and politics? Extremely closely, very closely, moderately closely, slightly closely, or not closely at all?” [v06p631] (3) How often do you pay attention to what’s going on in government and politics? All the time, most of the time, about half the time, once in a while, or never?” [v06p632] The other half of the respondents were randomly assigned to be asked the following two questions: (1) “Some people don’t pay much attention to political campaigns. How about you? Would you say that you have been VERY MUCH interested, SOMEWHAT interested, or NOT MUCH interested in the political campaigns this year?” [v06p633] and (2) “Some people seem to follow what’s going on in government and public affairs most of the time, whether there’s an election going on or not. Others aren’t that interested. Would you say you follow what’s going on in government and public affairs most of the time, some of the time, only now and then, or hardly at all?” [v06p634] Responses to all questions were coded to range from 0 (least involved) to 1 (most involved) and averaged into an index of political involvement. Respondents with missing data for any question were coded as missing for the index. Respondents were assigned to the two political involvement conditions orthogonally from assignment to turnout question-wording condition, and these two variables were unassociated.

Strength of party identification (2006): Half of the respondents were randomly assigned to be asked: “Generally speaking, do you think of yourself as a Republican, a Democrat, an Independent, or what?” (the order of the first two response options was varied across respondents). The other half of the respondents were randomly assigned to be asked: “As of today, do you think of yourself as a Republican, a Democrat, an Independent, or what?” (the order of the first two response options was varied across respondents). Then all respondents were asked the appropriate follow-up questions: [IF REPUBLICAN] “Would you call yourself a strong Republican or a not-very-strong Republican?” [IF DEMOCRAT] “Would you call yourself a strong Democrat or a not-very-strong Democrat?” [IF INDEPENDENT OR SOMETHING ELSE] “Do you think of yourself as closer to the Republican Party or to the Democratic Party?” [summary variable: v06p680] (the order of these response options was varied across respondents and consistent with the order of the initial party identification question). Responses were coded 1 for respondents who identified as strong Republicans or Democrats, 2/3 for respondents who identified as weak Republicans or Democrats, 1/3 for respondents who initially reported that they were independent or something else but reported that they were closer to one party or the other in the follow-up question, and 0 for respondents who said they were independents or something else and did not lean toward either major party. Assignment to the two question-wording conditions was orthogonal to assignment to turnout question-wording condition.

Political efficacy (2006): Half of the respondents were randomly assigned to be asked the following two questions: (1) “I’d like to read you a few statements about public life. I’ll read them one at a time. Please tell me how strongly you agree or disagree with each of them. ‘Public officials don’t care much what people like me think.’ Do you agree strongly, agree somewhat, neither agree nor disagree, disagree somewhat, or disagree strongly?” [v06p650] and (2) “‘People like me don’t have any say about what the government does.’ Do you agree strongly, agree somewhat, neither agree nor disagree, disagree somewhat, or disagree strongly?” [v06p651] The other half of the respondents were randomly assigned to be asked the following two questions: (1) “How much do public officials care what people like you think? A great deal, a lot, a moderate amount, a little, or not at all?” [v06p652] and (2) “How much can people like you affect what the government does? A great deal, a lot, a moderate amount, a little, or not at all?” [v06p653] Responses to each question were coded to range from 0 (lowest efficacy) to 1 (highest efficacy) and averaged together to form a political efficacy index. Respondents with missing data for any question were coded as missing for the index. Assignment to the two question-wording conditions was orthogonal to assignment to the turnout question-wording condition.

References