The Impact of Mail, Web, and Mixed-Mode Data Collection on Participation in Establishment Surveys

The first experimental group (i.e., control group) was conducted using the standard IAB-JVS mode design, that is, a concurrent mail–web mixed-mode design with paper questionnaire and login information to the web questionnaire included in both contact attempts (invitation and post-due-date reminder). The majority of establishments (N = 109,924) were allocated to this group.

The other three experimental groups consisted of a sequential web-to-mail mixed-mode design, a single-mode web design, and a single-mode mail design. For the sequential web-to-mail design, the invitation letter referred to the web survey, whereas the post-due-date reminder offered establishments the additional option of responding via the enclosed mailed questionnaire. For the single-mode mail design, both the invitation and post-due-date reminder contained a printed version of the questionnaire without the possibility of web completion. Except for the experimental manipulations, all field procedures were administered identically in all groups.

4.1.3 Establishment history panel

The forthcoming analysis uses data from the Establishment History Panel (BHP), an administrative database of all establishments in Germany, to investigate (proxy) nonresponse bias and correlates of survey participation (Ganzer et al. 2022). The BHP is an annual and cross-sectional aggregation of employee records to the reference date (June 30). As we use the 2020 BHP, we have administrative data for respondents and nonrespondents about one quarter before the IAB-JVS field period started. The response indicator from the IAB-JVS can be linked to the BHP via a unique establishment identifier for 127,338 establishments. Because of bankruptcies, changes in ownership or legal forms, or due to mergers or splits, the linkage was unsuccessful in 3.85 percent of cases. These observations are not included in the analysis of nonresponse bias (RQ2) and survey participation (RQ3) but are part of the response rate (RQ1) and cost (RQ4) analyses.

The BHP contains variables regarding basic establishment characteristics, including establishment size, industry, region, and various (aggregate) employee characteristics, such as the average age of employees, the share of fixed-term employees, and the average wage of employees. Since these variables are likely to be correlated with several IAB-JVS survey variables, such as the number of newly hired employees, number of vacancies, and number of hirings with fixed-term contracts, nonresponse bias in the BHP variables can serve as reasonable proxies for nonresponse bias in the survey variables. This strategy to estimate proxy nonresponse biases using administrative data has been applied in many other methodological studies (e.g., Eckman and Haas 2017; Kreuter et al. 2010; Küfner et al. 2022). All used BHP variables are categorized into roughly equal-sized groups with the exception of binary variables. The variables do not have any missing values. Supplement B and C in the supplementary data online provide an overview of the BHP variables and summary statistics for each administrative variable by mode design group.

4.2 Methods

4.2.1 Response rates

To address the first research question (RQ1), we report and compare the response rate of each mode design. In this analysis, a respondent is defined as an establishment that answers at least two essential Eurostat questions on the number of employees and job vacancies and submits their answers by clicking on the “submit” button in the web survey or by mailing the paper questionnaire back to the survey institute. Both essential questions are placed at the beginning of the questionnaire (see Supplement A.2 in the supplementary data online for the exact wording of these questions). Response rates are computed using the AAPOR RR1 definition (American Association for Public Opinion Research 2016), which is simply the number of respondents divided by the full sample (see Supplement D.1 in the supplementary data online for the corresponding formula).

4.2.2 Nonresponse bias

To assess the impact of mode designs on nonresponse bias (RQ2), bias estimates are calculated for multiple BHP variables. Nonresponse bias is defined as the difference between the estimate of interest for respondents and the estimate of interest for the full sample, which are proportions for the categorized BHP variables:

{\hat{NR bias}}_{i} = {\hat{Y}}_{i, r} - {\hat{Y}}_{i, n}

(1)

where

{\hat{Y}}_{i, r}

denotes the estimator for the i^th statistic of interest based on the respondents and

{\hat{Y}}_{i, n}

is the estimator based on the full sample.

Further, we also construct and compare measures of absolute bias and average absolute bias for summarized comparisons. Absolute nonresponse bias is defined as:

\hat{Abs . {NR bias}_{i}} = | \hat{{NR bias}_{i}} |

(2)

and average absolute nonresponse bias is defined as:

\hat{Avg . abs . NRbias} = \frac{\sum_{i = 1}^{K} \hat{Abs . {NR bias}_{i}}}{K},

(3)

with K being the total number of statistics of interest considered.

The average absolute nonresponse bias measure is computed separately across all statistics of interest for the establishment characteristic variables, the (aggregate) employee characteristic variables, and across all BHP variables (see also table S3 in the supplementary data online for an overview). To prevent the lower sampling variance of the concurrent mail–web mixed-mode design, due to its disproportionately larger sample size, from influencing the average absolute bias results, we use a repeated downsampling approach to ensure this design is analyzed with the same sample size as the other mode designs (around 7,500). The resulting resamples are also used for computing confidence intervals for the average absolute nonresponse bias estimates in this design. To estimate confidence intervals for the other mode designs, we use bootstrapped standard errors based on 500 replicates and a normal approximation. For the downsampling as well as the bootstrapping approach, we take industry and establishment size as strata into account. We compute only the overall estimates of average nonresponse bias for each mode design group, but no mode-specific estimates of bias for the individual modes used in the mixed-mode designs. As a sensitivity check, we also compute the median absolute nonresponse bias for the mode design groups.

4.2.3 Modeling survey participation

We model subgroup participation in multiple ways. First, we use logistic regression to model the likelihood of participation by establishment subgroup (size and industry) for each of the four mode designs. This analysis shows whether there are subgroup differences in overall response propensities between the different mode designs. For the mixed-mode designs, we make no distinction between whether an establishment participated via web or mail and are only interested in the overall response propensities for each design. No hypotheses are tested with this analysis.

To address the research question (RQ3) and test the hypotheses about which establishment subgroups are more likely to participate via web or mail (see section 3.2), we use two separate approaches. First, we model the likelihood of participation in single-mode web and single-mode mail designs by combining both groups and examining the predicted probabilities of response for different establishment characteristics (establishment size and industry) in the different mode designs with the covariates fixed at their global means. A logistic regression model of the response indicator (1 = response; 0 = nonresponse) is fitted with interaction terms between the experimental mode group and the establishment characteristics. To test for differences in the predicted probabilities between the single-mode web and the single-mode mail designs with respect to the covariates, we show the results of a Wald test (see Mize 2019). The following formula is used to predict the probability of participation based on the logistic regression:

P r (R_{k} = 1) = \frac{1}{1 + exp (- (α + β x_{k}^{⊤} + ζ {[x_{k} m_{k}]}^{⊤} + γ z_{k}^{⊤}))}

(4)

where Pr is the probability of the response indicator R_k for the k^th establishment (R_k = 1 = response, R_k = 0 = nonresponse),

x_{k, t}

is a vector of variables (establishment characteristics and experimental mode indicator), and

β

the corresponding vector of coefficients,

(x_{k, t} m_{k, t})

is a vector of establishment characteristics interacting with the experimental mode indicator,

ζ

the corresponding vector of coefficients, and

z_{k, t}

is a set of additional control variables with

γ

as the vector of corresponding coefficients.

Second, we examine which subgroups are more likely to participate via web or mail in the concurrent mail–web mixed-mode design. Here, we fit a multinomial logistic regression model where the dependent variable is web response, mail response, and nonresponse (reference group). Based on the multinominal regression, we estimate predicted probabilities for industry and establishment size with the covariates fixed at their global means. To assess the statistical significance of the difference between the predicted web and mail probabilities, we test the difference between web and mail using a Wald test. We formulate the multinomial logistic regression model to predict the probabilities in the following way:

P r (R_{k} = i) = \frac{exp (α_{i} + β_{i} x_{k}^{⊤} + γ_{i} z_{k}^{⊤})}{1 + \sum_{j = 2}^{3} exp (α_{j} + β_{j} x_{k}^{⊤} + γ_{j} z_{k}^{⊤})} for i = 2, 3

(5)

where Pr is the probability of the response indicator R_k for the k^th establishment (R_k = 1 = nonresponse, R_k = 2 = web response, R_k = 3 = mail response),

x_{k}

is a vector of variables of interest for the k^th establishment,

β_{i}

the corresponding vector of coefficients for the i^th response outcome, and

z_{k}

is a vector of additional control variables for the k^th establishment, with

γ_{i}

as their corresponding vector of coefficients for the i^th response outcome.

In both models, we include categorized foundation year as a control variable to account for the tenure of the establishment.

Design weights are incorporated into the analysis of response rates, nonresponse bias, and survey participation to account for unequal probabilities of selection. We also account for stratification when estimating linearized standard errors for the response rate comparison and survey participation models. All computations were conducted in Stata 17 (StataCorp 2021).

4.2.4 Survey costs

To assess the impact of mode designs on survey costs (RQ4), we consider the following variable costs: postage, printing of invitation and reminder letters and paper questionnaires, envelopes, and data entry. Since exact costs are not available or cannot be published due to contractual regulations, the analysis is based on assumed costs. These assumed costs come from consultations with the survey institute, online research, and experiences from other surveys. Variable costs account for only a portion of the total costs of each survey design and are typically more pronounced for mail modes. Costs related to survey management, set-up costs, and data processing are not included (see table S54 in the supplementary data online for a list of mode-related fixed costs). Nevertheless, variable costs are an important component of survey costs and provide some indications of the cost-effectiveness of the different mode designs. In the forthcoming analysis, we report the per-respondent costs for each experimental group.

5. RESULTS

5.1 Response Rates

Figure 2 presents the design-weighted response rates for each experimental group. A tabular version with absolute numbers and unweighted response rates are shown in Supplement D in the supplementary data online. Comparing the mode designs, there is neither a substantial nor a statistically significant difference between the response rates of the concurrent mail–web mixed-mode design (15.0 percent), the sequential web-to-mail mixed-mode design (14.5 percent), the single-mode web design (14.6 percent), and the single-mode mail design (13.5 percent). Hence, there is no support for hypotheses M1, M2, M3, and M4. Both the response rate of the single-mode web design (14.6 percent) and the web take-up rate in the sequential web-to-mail mixed-mode design (7.9 percent) are higher than the web take-up rate of the concurrent mixed-mode design (4.9 percent), which lends further support to the performance of “push-to-web” strategies in establishment surveys and supports M5. In line with M6, the response rate of the single-mode mail design (13.5 percent) exceeds the mail take-up rate of the concurrent mixed-mode design (10.1 percent).

Figure 2.

Response Rate (weighted) and 95 percent Confidence Interval, by Mode Design. Source: IAB-JVS 2020.

5.2 Nonresponse Bias

Figure 3 shows the average absolute nonresponse bias across all BHP administrative variables. Tables and figures for the average absolute nonresponse bias and the median absolute nonresponse bias are shown separately for the establishment and employee characteristic variable groups in Supplement E in the supplementary data online. Moreover, nonresponse biases for individual variables are presented in Supplement E.3 in the supplementary data online.

Figure 3.

Average Absolute Nonresponse Bias Estimates and 95 percent Confidence Intervals, by Mode Design, for All BHP Administrative Variables.

Overall, the results show rather low levels of aggregate nonresponse bias. The average absolute nonresponse bias across all BHP administrative variables is less than 3 percent for all mode designs. Further, there are no statistically significant differences between the mode designs. Similarly, there are no substantial differences in average absolute bias between the mode designs with respect to the establishment characteristic variables (e.g., size, industry, region) or the (aggregate) employee characteristic variables (e.g., average age of employees, share of female employees) when both variable groups are examined separately. Thus, the different mode designs yield respondents that are generally comparable with respect to establishment and workforce characteristics. Similar conclusions hold for the median absolute nonresponse bias results.

For individual variables, a few notable biases can be observed. The raw nonresponse biases for establishment size categories are particularly large in the single-mode mail design with the largest bias occurring for the smallest establishment size group (7.25 percent). Specifically, establishments with less than 10 employees are overrepresented by 7.25 percentage points, which is higher than in the other mode design groups. The single-mode web (–1.79 percent) and the single-mode mail (–2.08 percent) designs have smaller negative biases for the service industry compared to the sequential web-to-mail (–7.94 percent) and concurrent mail/web mixed-mode designs (–5.32 percent), indicating that the service industry is more accurately represented by respondents in the single-mode designs. We observe a strong and significant nonresponse bias for establishments founded after 2010 in the mail-only group (–9.12 percent), meaning that these younger establishments are underrepresented in the respondent pool. Moreover, participating establishments with the highest proportion of high-educated employees in the sequential web-to-mail mixed-mode design are overrepresented by 6.58 percentage points.

In summary, the results do not show strong differences between the different mode designs with respect to aggregate nonresponse bias across all administrative variables. Lastly, the biases of individual variables show only few meaningful differences between the mode designs. For instance, the single-mode mail design overrepresents the smallest establishments and underrepresents the youngest establishments to a greater extent than the other mode designs.

5.3 Predictors of Survey Participation by Mode

Figure 4 shows the predicted probabilities of survey participation for each of the four mode designs by establishment characteristics, number of employees (panel a) and industry (panel b) based on a logistic regression model of survey participation. We do not observe any substantial differences in response propensities between the different mode designs with respect to these characteristics, suggesting a similar pattern of survey participation regardless of which survey design is used.

Figure 4.

Predicted Probabilities and 95 percent Confidence Intervals of Participation, by Mode Design. Source: IAB-JVS 2020.

To test our mode-specific hypotheses, we take a closer look at the single-mode web and mail designs. Figure 5 shows the predicted probabilities of survey participation in the single-mode web and mail designs for the establishment characteristics based on the logistic regression model of survey participation. With respect to establishment size, larger establishments (except for the largest establishments with more than 250 employees) are more likely to participate in a web survey than in a mail survey compared to smaller establishments. This higher likelihood of participation is statistically significant and thus supports H1a. The agriculture, construction, public administration, information/communication, and finance/insurance industries show no reliable relationship with either mode, thus yielding no support for H2a, H3a, H4a, H5a, and H6a, respectively.

Figure 5.

Predicted Probabilities and 95 percent Confidence Intervals of Survey Participation in Single-Mode Web and Single-Mode Mail Designs, by Establishment Characteristics. Source: IAB-JVS 2020. Notes: The additional column shows the difference between the predicted probability of mail and web participation and the corresponding result of a Wald test. Significance Levels: ***p < 0.001; **p < 0.01; *p < 0.05.

Figure 6 shows the predicted probabilities and confidence intervals from the multinomial regression model of participation in the web and mail modes of the concurrent mixed-mode design with nonresponse as the reference category. The corresponding regression tables are provided in Supplement F in the supplementary data online. The results show that the predicted probabilities for mail participation are higher than for web participation for the different establishment characteristics (except for the information/communication industry). For instance, larger establishments are more likely to participate by mail than by web in the concurrent mixed-mode design, which contradicts H1b. However, the difference between the predicted probabilities of web and mail shrinks with increasing establishment size, from 6.0 percentage points (mail: 11.0 percent; web: 5.0 percent) for the smallest establishments to 0.7 percentage points (mail: 2.8 percent; web: 2.1 percent) for the largest establishments. This implies that smaller establishments have a higher likelihood to participate by mail, while larger establishments are almost equally likely to participate by web or mail.

Figure 6.

Predicted Probabilities and 95 percent Confidence Intervals of Survey Participation by Mail and Web in the Concurrent Mail–Web Mixed-Mode Design, by Establishment Characteristics. Source: IAB-JVS 2020. Notes: The additional column shows the difference between the predicted probability of mail and web participation and the corresponding result of a Wald test. Significance Levels: ***p < .001; **p <.01; *p < .05.

With respect to industry participation, establishments in the agricultural and construction industries have a higher probability of participating by mail than by web, yielding support for H2b and H3b, respectively. There is no statistically significant difference in the predicted probabilities of web and mail participation for establishments in the public administration, information/communication, and the finance/insurance industries. Hence, there is no support for hypotheses H4b, H5b, and H6b. However, it is interesting that these three industries are the only ones where the predicted probabilities of mail participation are not significantly higher than web participation. This could be a sign that these industries are more open to choosing the web mode compared to other industries.

5.4 Respondent Composition by Mode and Mode Design

As a side analysis, table 1 shows the composition of respondents by mode of participation within each mode design. Here, we can explore whether the compositions of web and mail respondents differ from each other conditional on certain mode designs and whether that depends on whether two modes are used or just one. We point out a few notable patterns and speculate on their causes. For the establishment size subgroups, we can see that the smallest establishments are overrepresented in the mail mode in all three of the mail mode designs, indicating that the mail mode brings in more of these respondents. Conversely, the web mode tends to bring in larger establishments at a greater rate than mail.

Table 1.

Respondent Composition by Mode and Mode Design (in Column Percentages)

	Mail–web (all)	Mail–web (web)	Mail–web (mail)	Web-to-mail (all)	Web-to-mail (web)	Web-to-mail (mail)	Single-mode (web)	Single-mode (mail)	Full (Sample)
Number of employees
1–9	72.9 (71.8, 73.9)	69.7 (68.3, 71.2)	74.4 (73.0, 75.7)	73.6 (70.7, 76.3)	69.3 (66.7, 71.8)	78.9 (72.5, 84.2)	72.1 (68.8, 75.1)	77.0 (73.3, 80.3)	69.9 (69.0, 70.8)
10–19	14.9 (13.8, 15.9)	14.5 (13.2, 16.1)	15.0 (13.7, 16.4)	12.9 (10.2, 16.1)	12.9 (10.5, 15.8)	12.8 (8.0, 19.9)	13.5 (10.6, 17.1)	12.3 (9.1, 16.3)	14.9 (14.2, 15.6)
20–49	8.2 (7.9, 8.4)	10.1 (9.5, 10.6)	7.2 (6.9, 7.6)	8.5 (7.7, 9.5)	11.5 (10.0, 13.1)	4.9 (3.7, 6.4)	10.2 (9.1, 11.3)	7.4 (6.6, 8.2)	9.3 (9.0, 9.7)
$\geq$ 50	4.1 (4.0, 4.2)	5.7 (5.5, 5.8)	3.4 (3.3, 3.5)	5.0 (4.6, 5.5)	6.3 (5.6, 7.1)	3.4 (2.6, 4.3)	4.2 (3.9, 4.6)	3.3 (3.0, 3.7)	5.9 (5.7, 6.1)
Industry
Agriculture/forestry	3.5 (3.5, 3.5)	3.1 (3.1, 3.1)	3.7 (3.7, 3.7)	2.8 (2.6, 3.0)	1.9 (1.5, 2.4)	3.8 (3.7, 4.0)	2.2 (2.2, 2.3)	4.6 (4.6, 4.6)	2.7 (2.4, 3.0)
Mining/energy/waste	0.7 (0.7, 0.7)	0.9 (0.9, 0.9)	0.6 (0.6, 0.6)	0.6 (0.5, 0.6)	0.7 (0.6, 0.8)	0.4 (0.4, 0.5)	0.8 (0.7, 0.8)	0.6 (0.6, 0.7)	0.9 (0.8, 0.9)
Manufacturing	8.5 (8.4, 8.5)	9.2 (9.0, 9.3)	8.1 (8.1, 8.2)	9.0 (9.0, 9.1)	9.7 (9.3, 10.2)	8.1 (7.8, 8.5)	9.1 (8.9, 9.2)	8.1 (8.0, 8.2)	8.2 (7.9, 8.5)
Construction	12.7 (12.5, 12.8)	11.0 (10.9, 11.1)	13.5 (13.3, 13.7)	11.9 (11.9, 12.0)	14.7 (14.1, 15.3)	8.5 (6.8, 10.5)	11.8 (11.8, 11.9)	11.6 (11.6, 11.7)	11.0 (10.3, 11.8)
Trade/car-repair	15.0 (15.0, 15.0)	14.3 (14.2, 14.4)	15.3 (15.3, 15.4)	11.2 (11.2, 11.2)	5.2 (3.1, 8.9)	18.7 (17.8, 19.6)	16.7 (16.7, 16.8)	14.0 (13.9, 14.0)	18.9 (17.8, 20.0)
Transportation/storage	3.6 (3.6, 3.6)	3.4 (3.4, 3.4)	3.6 (3.6, 3.7)	4.9 (4.9, 4.9)	5.3 (4.8, 5.8)	4.5 (4.3, 4.7)	3.9 (3.9, 3.9)	5.2 (5.2, 5.3)	3.8 (3.5, 4.0)
Information/communication	2.5 (2.5, 2.6)	4.2 (4.2, 4.2)	1.7 (1.7, 1.7)	2.6 (2.6, 2.6)	2.7 (2.5, 2.8)	2.5 (2.4, 2.6)	2.0 (2.0, 2.0)	2.2 (2.2, 2.2)	3.1 (2.9, 3.3)
Finance/insurance	2.5 (2.5, 2.5)	3.2 (3.2, 3.2)	2.1 (2.1, 2.1)	1.4 (1.4, 1.4)	1.6 (1.5, 1.6)	1.2 (0.4, 3.2)	1.0 (0.9, 1.3)	3.3 (3.2, 3.3)	3.0 (2.8, 3.2)
Business-related services	20.1 (20.0, 20.3)	21.8 (21.8, 21.9)	19.3 (19.1, 19.6)	21.9 (21.9, 22.0)	29.5 (28.4, 30.7)	12.4 (10.8, 14.1)	22.0 (21.9, 22.1)	21.9 (21.7, 22.0)	18.7 (17.9, 19.5)
Other services	29.4 (29.4, 29.5)	26.5 (26.4, 26.5)	30.9 (30.8, 31.0)	32.3 (32.2, 32.4)	27.3 (25.3, 29.5)	38.5 (36.9, 40.1)	29.0 (28.9, 29.2)	27.3 (27.1, 27.5)	28.4 (27.5, 29.4)
Public administration	1.4 (1.4, 1.5)	2.3 (2.3, 2.3)	1.0 (1.0, 1.0)	1.4 (1.3, 1.4)	1.3 (1.0, 1.7)	1.5 (1.2, 1.8)	1.4 (1.4, 1.5)	1.1 (1.1, 1.2)	1.4 (1.3, 1.5)

	Mail–web (all)	Mail–web (web)	Mail–web (mail)	Web-to-mail (all)	Web-to-mail (web)	Web-to-mail (mail)	Single-mode (web)	Single-mode (mail)	Full (Sample)
Number of employees
1–9	72.9 (71.8, 73.9)	69.7 (68.3, 71.2)	74.4 (73.0, 75.7)	73.6 (70.7, 76.3)	69.3 (66.7, 71.8)	78.9 (72.5, 84.2)	72.1 (68.8, 75.1)	77.0 (73.3, 80.3)	69.9 (69.0, 70.8)
10–19	14.9 (13.8, 15.9)	14.5 (13.2, 16.1)	15.0 (13.7, 16.4)	12.9 (10.2, 16.1)	12.9 (10.5, 15.8)	12.8 (8.0, 19.9)	13.5 (10.6, 17.1)	12.3 (9.1, 16.3)	14.9 (14.2, 15.6)
20–49	8.2 (7.9, 8.4)	10.1 (9.5, 10.6)	7.2 (6.9, 7.6)	8.5 (7.7, 9.5)	11.5 (10.0, 13.1)	4.9 (3.7, 6.4)	10.2 (9.1, 11.3)	7.4 (6.6, 8.2)	9.3 (9.0, 9.7)
$\geq$ 50	4.1 (4.0, 4.2)	5.7 (5.5, 5.8)	3.4 (3.3, 3.5)	5.0 (4.6, 5.5)	6.3 (5.6, 7.1)	3.4 (2.6, 4.3)	4.2 (3.9, 4.6)	3.3 (3.0, 3.7)	5.9 (5.7, 6.1)
Industry
Agriculture/forestry	3.5 (3.5, 3.5)	3.1 (3.1, 3.1)	3.7 (3.7, 3.7)	2.8 (2.6, 3.0)	1.9 (1.5, 2.4)	3.8 (3.7, 4.0)	2.2 (2.2, 2.3)	4.6 (4.6, 4.6)	2.7 (2.4, 3.0)
Mining/energy/waste	0.7 (0.7, 0.7)	0.9 (0.9, 0.9)	0.6 (0.6, 0.6)	0.6 (0.5, 0.6)	0.7 (0.6, 0.8)	0.4 (0.4, 0.5)	0.8 (0.7, 0.8)	0.6 (0.6, 0.7)	0.9 (0.8, 0.9)
Manufacturing	8.5 (8.4, 8.5)	9.2 (9.0, 9.3)	8.1 (8.1, 8.2)	9.0 (9.0, 9.1)	9.7 (9.3, 10.2)	8.1 (7.8, 8.5)	9.1 (8.9, 9.2)	8.1 (8.0, 8.2)	8.2 (7.9, 8.5)
Construction	12.7 (12.5, 12.8)	11.0 (10.9, 11.1)	13.5 (13.3, 13.7)	11.9 (11.9, 12.0)	14.7 (14.1, 15.3)	8.5 (6.8, 10.5)	11.8 (11.8, 11.9)	11.6 (11.6, 11.7)	11.0 (10.3, 11.8)
Trade/car-repair	15.0 (15.0, 15.0)	14.3 (14.2, 14.4)	15.3 (15.3, 15.4)	11.2 (11.2, 11.2)	5.2 (3.1, 8.9)	18.7 (17.8, 19.6)	16.7 (16.7, 16.8)	14.0 (13.9, 14.0)	18.9 (17.8, 20.0)
Transportation/storage	3.6 (3.6, 3.6)	3.4 (3.4, 3.4)	3.6 (3.6, 3.7)	4.9 (4.9, 4.9)	5.3 (4.8, 5.8)	4.5 (4.3, 4.7)	3.9 (3.9, 3.9)	5.2 (5.2, 5.3)	3.8 (3.5, 4.0)
Information/communication	2.5 (2.5, 2.6)	4.2 (4.2, 4.2)	1.7 (1.7, 1.7)	2.6 (2.6, 2.6)	2.7 (2.5, 2.8)	2.5 (2.4, 2.6)	2.0 (2.0, 2.0)	2.2 (2.2, 2.2)	3.1 (2.9, 3.3)
Finance/insurance	2.5 (2.5, 2.5)	3.2 (3.2, 3.2)	2.1 (2.1, 2.1)	1.4 (1.4, 1.4)	1.6 (1.5, 1.6)	1.2 (0.4, 3.2)	1.0 (0.9, 1.3)	3.3 (3.2, 3.3)	3.0 (2.8, 3.2)
Business-related services	20.1 (20.0, 20.3)	21.8 (21.8, 21.9)	19.3 (19.1, 19.6)	21.9 (21.9, 22.0)	29.5 (28.4, 30.7)	12.4 (10.8, 14.1)	22.0 (21.9, 22.1)	21.9 (21.7, 22.0)	18.7 (17.9, 19.5)
Other services	29.4 (29.4, 29.5)	26.5 (26.4, 26.5)	30.9 (30.8, 31.0)	32.3 (32.2, 32.4)	27.3 (25.3, 29.5)	38.5 (36.9, 40.1)	29.0 (28.9, 29.2)	27.3 (27.1, 27.5)	28.4 (27.5, 29.4)
Public administration	1.4 (1.4, 1.5)	2.3 (2.3, 2.3)	1.0 (1.0, 1.0)	1.4 (1.3, 1.4)	1.3 (1.0, 1.7)	1.5 (1.2, 1.8)	1.4 (1.4, 1.5)	1.1 (1.1, 1.2)	1.4 (1.3, 1.5)

Source.—IAB-JVS 2020.

Table 1.

Respondent Composition by Mode and Mode Design (in Column Percentages)

	Mail–web (all)	Mail–web (web)	Mail–web (mail)	Web-to-mail (all)	Web-to-mail (web)	Web-to-mail (mail)	Single-mode (web)	Single-mode (mail)	Full (Sample)
Number of employees
1–9	72.9 (71.8, 73.9)	69.7 (68.3, 71.2)	74.4 (73.0, 75.7)	73.6 (70.7, 76.3)	69.3 (66.7, 71.8)	78.9 (72.5, 84.2)	72.1 (68.8, 75.1)	77.0 (73.3, 80.3)	69.9 (69.0, 70.8)
10–19	14.9 (13.8, 15.9)	14.5 (13.2, 16.1)	15.0 (13.7, 16.4)	12.9 (10.2, 16.1)	12.9 (10.5, 15.8)	12.8 (8.0, 19.9)	13.5 (10.6, 17.1)	12.3 (9.1, 16.3)	14.9 (14.2, 15.6)
20–49	8.2 (7.9, 8.4)	10.1 (9.5, 10.6)	7.2 (6.9, 7.6)	8.5 (7.7, 9.5)	11.5 (10.0, 13.1)	4.9 (3.7, 6.4)	10.2 (9.1, 11.3)	7.4 (6.6, 8.2)	9.3 (9.0, 9.7)
$\geq$ 50	4.1 (4.0, 4.2)	5.7 (5.5, 5.8)	3.4 (3.3, 3.5)	5.0 (4.6, 5.5)	6.3 (5.6, 7.1)	3.4 (2.6, 4.3)	4.2 (3.9, 4.6)	3.3 (3.0, 3.7)	5.9 (5.7, 6.1)
Industry
Agriculture/forestry	3.5 (3.5, 3.5)	3.1 (3.1, 3.1)	3.7 (3.7, 3.7)	2.8 (2.6, 3.0)	1.9 (1.5, 2.4)	3.8 (3.7, 4.0)	2.2 (2.2, 2.3)	4.6 (4.6, 4.6)	2.7 (2.4, 3.0)
Mining/energy/waste	0.7 (0.7, 0.7)	0.9 (0.9, 0.9)	0.6 (0.6, 0.6)	0.6 (0.5, 0.6)	0.7 (0.6, 0.8)	0.4 (0.4, 0.5)	0.8 (0.7, 0.8)	0.6 (0.6, 0.7)	0.9 (0.8, 0.9)
Manufacturing	8.5 (8.4, 8.5)	9.2 (9.0, 9.3)	8.1 (8.1, 8.2)	9.0 (9.0, 9.1)	9.7 (9.3, 10.2)	8.1 (7.8, 8.5)	9.1 (8.9, 9.2)	8.1 (8.0, 8.2)	8.2 (7.9, 8.5)
Construction	12.7 (12.5, 12.8)	11.0 (10.9, 11.1)	13.5 (13.3, 13.7)	11.9 (11.9, 12.0)	14.7 (14.1, 15.3)	8.5 (6.8, 10.5)	11.8 (11.8, 11.9)	11.6 (11.6, 11.7)	11.0 (10.3, 11.8)
Trade/car-repair	15.0 (15.0, 15.0)	14.3 (14.2, 14.4)	15.3 (15.3, 15.4)	11.2 (11.2, 11.2)	5.2 (3.1, 8.9)	18.7 (17.8, 19.6)	16.7 (16.7, 16.8)	14.0 (13.9, 14.0)	18.9 (17.8, 20.0)
Transportation/storage	3.6 (3.6, 3.6)	3.4 (3.4, 3.4)	3.6 (3.6, 3.7)	4.9 (4.9, 4.9)	5.3 (4.8, 5.8)	4.5 (4.3, 4.7)	3.9 (3.9, 3.9)	5.2 (5.2, 5.3)	3.8 (3.5, 4.0)
Information/communication	2.5 (2.5, 2.6)	4.2 (4.2, 4.2)	1.7 (1.7, 1.7)	2.6 (2.6, 2.6)	2.7 (2.5, 2.8)	2.5 (2.4, 2.6)	2.0 (2.0, 2.0)	2.2 (2.2, 2.2)	3.1 (2.9, 3.3)
Finance/insurance	2.5 (2.5, 2.5)	3.2 (3.2, 3.2)	2.1 (2.1, 2.1)	1.4 (1.4, 1.4)	1.6 (1.5, 1.6)	1.2 (0.4, 3.2)	1.0 (0.9, 1.3)	3.3 (3.2, 3.3)	3.0 (2.8, 3.2)
Business-related services	20.1 (20.0, 20.3)	21.8 (21.8, 21.9)	19.3 (19.1, 19.6)	21.9 (21.9, 22.0)	29.5 (28.4, 30.7)	12.4 (10.8, 14.1)	22.0 (21.9, 22.1)	21.9 (21.7, 22.0)	18.7 (17.9, 19.5)
Other services	29.4 (29.4, 29.5)	26.5 (26.4, 26.5)	30.9 (30.8, 31.0)	32.3 (32.2, 32.4)	27.3 (25.3, 29.5)	38.5 (36.9, 40.1)	29.0 (28.9, 29.2)	27.3 (27.1, 27.5)	28.4 (27.5, 29.4)
Public administration	1.4 (1.4, 1.5)	2.3 (2.3, 2.3)	1.0 (1.0, 1.0)	1.4 (1.3, 1.4)	1.3 (1.0, 1.7)	1.5 (1.2, 1.8)	1.4 (1.4, 1.5)	1.1 (1.1, 1.2)	1.4 (1.3, 1.5)

	Mail–web (all)	Mail–web (web)	Mail–web (mail)	Web-to-mail (all)	Web-to-mail (web)	Web-to-mail (mail)	Single-mode (web)	Single-mode (mail)	Full (Sample)
Number of employees
1–9	72.9 (71.8, 73.9)	69.7 (68.3, 71.2)	74.4 (73.0, 75.7)	73.6 (70.7, 76.3)	69.3 (66.7, 71.8)	78.9 (72.5, 84.2)	72.1 (68.8, 75.1)	77.0 (73.3, 80.3)	69.9 (69.0, 70.8)
10–19	14.9 (13.8, 15.9)	14.5 (13.2, 16.1)	15.0 (13.7, 16.4)	12.9 (10.2, 16.1)	12.9 (10.5, 15.8)	12.8 (8.0, 19.9)	13.5 (10.6, 17.1)	12.3 (9.1, 16.3)	14.9 (14.2, 15.6)
20–49	8.2 (7.9, 8.4)	10.1 (9.5, 10.6)	7.2 (6.9, 7.6)	8.5 (7.7, 9.5)	11.5 (10.0, 13.1)	4.9 (3.7, 6.4)	10.2 (9.1, 11.3)	7.4 (6.6, 8.2)	9.3 (9.0, 9.7)
$\geq$ 50	4.1 (4.0, 4.2)	5.7 (5.5, 5.8)	3.4 (3.3, 3.5)	5.0 (4.6, 5.5)	6.3 (5.6, 7.1)	3.4 (2.6, 4.3)	4.2 (3.9, 4.6)	3.3 (3.0, 3.7)	5.9 (5.7, 6.1)
Industry
Agriculture/forestry	3.5 (3.5, 3.5)	3.1 (3.1, 3.1)	3.7 (3.7, 3.7)	2.8 (2.6, 3.0)	1.9 (1.5, 2.4)	3.8 (3.7, 4.0)	2.2 (2.2, 2.3)	4.6 (4.6, 4.6)	2.7 (2.4, 3.0)
Mining/energy/waste	0.7 (0.7, 0.7)	0.9 (0.9, 0.9)	0.6 (0.6, 0.6)	0.6 (0.5, 0.6)	0.7 (0.6, 0.8)	0.4 (0.4, 0.5)	0.8 (0.7, 0.8)	0.6 (0.6, 0.7)	0.9 (0.8, 0.9)
Manufacturing	8.5 (8.4, 8.5)	9.2 (9.0, 9.3)	8.1 (8.1, 8.2)	9.0 (9.0, 9.1)	9.7 (9.3, 10.2)	8.1 (7.8, 8.5)	9.1 (8.9, 9.2)	8.1 (8.0, 8.2)	8.2 (7.9, 8.5)
Construction	12.7 (12.5, 12.8)	11.0 (10.9, 11.1)	13.5 (13.3, 13.7)	11.9 (11.9, 12.0)	14.7 (14.1, 15.3)	8.5 (6.8, 10.5)	11.8 (11.8, 11.9)	11.6 (11.6, 11.7)	11.0 (10.3, 11.8)
Trade/car-repair	15.0 (15.0, 15.0)	14.3 (14.2, 14.4)	15.3 (15.3, 15.4)	11.2 (11.2, 11.2)	5.2 (3.1, 8.9)	18.7 (17.8, 19.6)	16.7 (16.7, 16.8)	14.0 (13.9, 14.0)	18.9 (17.8, 20.0)
Transportation/storage	3.6 (3.6, 3.6)	3.4 (3.4, 3.4)	3.6 (3.6, 3.7)	4.9 (4.9, 4.9)	5.3 (4.8, 5.8)	4.5 (4.3, 4.7)	3.9 (3.9, 3.9)	5.2 (5.2, 5.3)	3.8 (3.5, 4.0)
Information/communication	2.5 (2.5, 2.6)	4.2 (4.2, 4.2)	1.7 (1.7, 1.7)	2.6 (2.6, 2.6)	2.7 (2.5, 2.8)	2.5 (2.4, 2.6)	2.0 (2.0, 2.0)	2.2 (2.2, 2.2)	3.1 (2.9, 3.3)
Finance/insurance	2.5 (2.5, 2.5)	3.2 (3.2, 3.2)	2.1 (2.1, 2.1)	1.4 (1.4, 1.4)	1.6 (1.5, 1.6)	1.2 (0.4, 3.2)	1.0 (0.9, 1.3)	3.3 (3.2, 3.3)	3.0 (2.8, 3.2)
Business-related services	20.1 (20.0, 20.3)	21.8 (21.8, 21.9)	19.3 (19.1, 19.6)	21.9 (21.9, 22.0)	29.5 (28.4, 30.7)	12.4 (10.8, 14.1)	22.0 (21.9, 22.1)	21.9 (21.7, 22.0)	18.7 (17.9, 19.5)
Other services	29.4 (29.4, 29.5)	26.5 (26.4, 26.5)	30.9 (30.8, 31.0)	32.3 (32.2, 32.4)	27.3 (25.3, 29.5)	38.5 (36.9, 40.1)	29.0 (28.9, 29.2)	27.3 (27.1, 27.5)	28.4 (27.5, 29.4)
Public administration	1.4 (1.4, 1.5)	2.3 (2.3, 2.3)	1.0 (1.0, 1.0)	1.4 (1.3, 1.4)	1.3 (1.0, 1.7)	1.5 (1.2, 1.8)	1.4 (1.4, 1.5)	1.1 (1.1, 1.2)	1.4 (1.3, 1.5)

Source.—IAB-JVS 2020.

Regarding the industry subgroups, the mail mode brings in a higher percentage of construction sector respondents than web in the concurrent mixed-mode design, but the opposite is true for the sequential mixed-mode design, suggesting that this subgroup, if given a choice, generally prefers to respond via mail but is just as likely to participate using either web or mail depending on which mode is offered first. This is also reflected in both single-mode designs, which bring in similar shares of construction respondents. The opposite pattern is seen for the information/communication sector. The concurrent design brings in more web than mail respondents, while the sequential and single-mode designs bring in similar shares of respondents in both modes.

Interestingly, the mail and web modes bring in relatively similar proportions of trade/car-repair respondents in the single-mode and concurrent designs, but this group is strongly underrepresented in the web mode of the sequential mixed-mode design. This may be driven by late respondents, who are automatically allocated to the mail nonresponse follow-up phase, rather than a strong reluctance to participate in the web mode. A similar pattern occurs for the business-related services sector, where there is a strong overrepresentation of web respondents in the sequential design, but comparable representation between both modes in the other designs. In contrast to the previous example, this could be driven by an early respondent effect, where these respondents are highly cooperative and answer promptly regardless of which mode is offered. However, these are only speculations as it is difficult to draw definitive conclusions between the sequential and non-sequential designs due to the additional selection step in the former.

5.5 Survey Costs

Table 2 shows a summary of all analyzed costs per mode design and Supplement table S55 in the supplementary data online provides more details on the specific costs associated with each mode design. The analysis shows that the single-mode mail design leads to the highest costs per respondent (35.20 €), which includes costs of each contact attempt and data entry. This is followed by the concurrent mail–web mixed-mode design (29.69 €), which includes costs for postage and printing, but fewer data entry costs. Due to the higher number of web respondents and less expensive mailings, the sequential web-to-mail mixed-mode (21.55 €) and the single-mode web (13.93 €) designs are the least expensive mode designs. These results support hypothesis M7 and highlight the potential cost savings of switching to a push-to-web design in establishment surveys.

Table 2.

Survey Costs by Experimental Group

	Mail–web	Web-to-mail	Web-only	Mail-only
Invitation	192,367.00 €	6,608.80 €	6,593.84 €	13,135.50 €
Post-due-date reminder	174,217.15 €	12,181.75 €	6,123.92 €	11,877.25 €
Re-postage mail	12,389.15 €	492.90 €	0.00 €	1,124.50 €
Data entry	15,816.11 €	631.03 €	0.00 €	1,567.66 €
Total costs	394,835.01 €	19,914.48 €	12,717.76 €	27,804.91 €
Costs per sampled unit	3.59 €	2.65 €	1.70 €	3.70 €
Costs per respondent unit	29.69 €	21.55 €	13.93 €	35.20 €

	Mail–web	Web-to-mail	Web-only	Mail-only
Invitation	192,367.00 €	6,608.80 €	6,593.84 €	13,135.50 €
Post-due-date reminder	174,217.15 €	12,181.75 €	6,123.92 €	11,877.25 €
Re-postage mail	12,389.15 €	492.90 €	0.00 €	1,124.50 €
Data entry	15,816.11 €	631.03 €	0.00 €	1,567.66 €
Total costs	394,835.01 €	19,914.48 €	12,717.76 €	27,804.91 €
Costs per sampled unit	3.59 €	2.65 €	1.70 €	3.70 €
Costs per respondent unit	29.69 €	21.55 €	13.93 €	35.20 €

Note.—Postage mail package: 1.55 €; postage web package: 0.80 €; print mail package: 0.20 €; print web package: 0.08 €; postage for return mail respondent: 1.55 €; data entry mail respondent: 1.98 €; data entry web respondent: 0.00 €.

Table 2.

Survey Costs by Experimental Group

	Mail–web	Web-to-mail	Web-only	Mail-only
Invitation	192,367.00 €	6,608.80 €	6,593.84 €	13,135.50 €
Post-due-date reminder	174,217.15 €	12,181.75 €	6,123.92 €	11,877.25 €
Re-postage mail	12,389.15 €	492.90 €	0.00 €	1,124.50 €
Data entry	15,816.11 €	631.03 €	0.00 €	1,567.66 €
Total costs	394,835.01 €	19,914.48 €	12,717.76 €	27,804.91 €
Costs per sampled unit	3.59 €	2.65 €	1.70 €	3.70 €
Costs per respondent unit	29.69 €	21.55 €	13.93 €	35.20 €

	Mail–web	Web-to-mail	Web-only	Mail-only
Invitation	192,367.00 €	6,608.80 €	6,593.84 €	13,135.50 €
Post-due-date reminder	174,217.15 €	12,181.75 €	6,123.92 €	11,877.25 €
Re-postage mail	12,389.15 €	492.90 €	0.00 €	1,124.50 €
Data entry	15,816.11 €	631.03 €	0.00 €	1,567.66 €
Total costs	394,835.01 €	19,914.48 €	12,717.76 €	27,804.91 €
Costs per sampled unit	3.59 €	2.65 €	1.70 €	3.70 €
Costs per respondent unit	29.69 €	21.55 €	13.93 €	35.20 €

6. QUALITATIVE INSIGHTS

To augment the quantitative results, qualitative interviews were conducted to understand how establishments view web and mail modes when deciding whether to participate in a voluntary survey. To this end, we conducted 46 short structured interviews and 12 semi-structured in-depth qualitative interviews with interviewees recruited from participants and non-participants of the 2020 fourth quarter mode design experiment discussed above. The short structured interviews aimed to gather information on establishment mode preferences and the perceived advantages and disadvantages of web, mail, and telephone modes. Selected establishments were balanced across establishment size, industry, region, and experimental groups (see table S56 in the supplementary data online for an overview of the sample characteristics). Human resources representatives and managers within the establishments responsible for responding to the IAB-JVS served as interviewees. These interviews were embedded in routine questionnaire pretests and carried out via telephone from February to May 2022 by trained interviewers.

The in-depth interviews were conducted to understand the impact of mode on response processes and the decision to participate. The sample, consisting of eight interviews with respondents and four with nonrespondents, was balanced in terms of establishment size, industry, region, and experimental group as outlined in table S56 in the supplementary data online. The interviews, conducted by the authors of this article between March and May 2022, lasted between 31 and 55 minutes. Using a semi-structured interview guide (see Supplement H.4 in the supplementary data online for the complete interview guidelines), sessions were held via video or telephone. To counteract potential recall issues concerning specific response decisions and processes from the IAB-JVS conducted 1.5 years ago, we introduced the relevant mode design scenarios when needed. More methodological details are provided in Supplement H in the supplementary data online.

For both qualitative study arms, most respondents (26 out of 46 short interviews, and 11 out of 12 in-depth interviews) preferred the web mode over the mail mode, or were indifferent. Interestingly, four establishments that responded by mail in the mixed-mode groups stated a preference for web surveys in the qualitative interviews. A possible explanation for this paradoxical preference is that these establishments see the advantages of participating by web, but in a real-world situation, it seems easy for them to grab a pen and fill in the paper questionnaire, or to use the words of one interviewee: “When I have this questionnaire in front of me on paper, I tend to be the person who fills it out on paper. If I had received the questionnaire by e-mail via a link, I probably wouldn’t have printed it out and filled it in, but would have submitted it online.”(see table S59, Quote No. 1 in the supplementary data online).

We identified two channels where establishments stated that web or mail modes could influence their decision to participate. First, the appearance and length of a mailed questionnaire can impact the participation decision in a positive (e.g., formal and reputable) or negative (e.g., overwhelming or too long) way. Length, in particular, was reported as having a negative effect on participation in mail surveys because it is more salient than in a web survey. As one interviewee put it: “Length is a deterrent, yes, of course. That means that if I have a twenty-page questionnaire somewhere, […] the will to drop out suddenly increases very exponentially.” (see table S57, Quote No. 1 in the supplementary data online). Second, respondents perceive responding to a web survey as less burdensome than to a mail survey in general (“It [the web questionnaire] would go faster and would be easier, easy in terms of effort”; see table S57, Quote No. 2 in the supplementary data online).

The biggest cited advantage of web questionnaires was the flexibility to complete the questionnaire at their convenience. This was mentioned for both mail and web modes, but was more strongly associated with the web mode: “You can just organize it yourself. […] Online I can say: O.k. I’ll put that aside now and take it at 4:00 p.m. and work on it then.” (see table S57, Quote No. 3 in the supplementary data online). For both qualitative study arms, web questionnaires were reported to be faster to complete than mail questionnaires. Respondents provided three main reasons for this. First, the return of web questionnaires does not require cumbersome postal returns: “I think the general willingness to participate is generally higher with an online survey, because you simply save yourself the trouble of sending it back and so on.” (see table S57, Quote No. 4 in the supplementary data online). Second, the internal routing of the mail questionnaire can be replaced with a brief email and other kinds of cooperation are facilitated (e.g., screen-sharing): “But also—as I said—the internal back and forth, you’re quicker at it [with web questionnaires]. And with that, there is also acceptance [for the survey request], somehow. Because anything that requires less effort within the company has great advantages in terms of getting results.” (see table S57, Quote No. 5 in the supplementary data online).

Third, because using computers and web apps is part of many HR managers’ regular routine, the entire response process is regarded as being quicker: “Yes, online just goes quickly. […] I log in, that’s what I do [working with web applications] most of the time, the threshold to participate there is relatively low.” (see table S57, Quote No. 6 in the supplementary data online). Moreover, the response process of a web survey is seen as easier, because it is easier to correct answers and there are no worries about unreadable handwriting: “No, actually [online] is much better for me […] I always doubt that you can read my handwriting then.” (see table S57, Quote No. 7 in the supplementary data online). Another cited advantage of the web questionnaire is the facilitated use of internal documents, making it easier to search, copy, and paste from internal management systems. In addition, web surveys are seen as more modern and more sustainable than mail surveys. Nicely summarized by the following statement: “Especially with the sustainability mindset that is overtaking us all, online is the most efficient, cost-effective, and easiest method.” (see table S57, Quote No. 8 in the supplementary data online).

Regarding the advantages of mail surveys, some respondents of the short interviews noted that having a paper questionnaire on the desk has a reminding effect and could thereby increase the likelihood to participate. Other reported advantages of mail surveys were that establishments get an easy overview of all questions before starting the answering process and could discuss the questionnaire in a team meeting more easily. Some respondents also stated the advantage of writing notes on a paper questionnaire and considering their answers based on those notes: “I’m actually also more of a haptic person. I have to be able to take notes all the time, assess questions and answers, and think things through.” (see table S58, Quote No. 1 in the supplementary data online). Establishments also appreciate that the mail survey can be copied and filed in their records for future reference: “In addition, I can make a copy of it—and I often do this […]—and file it in our correspondence. […] I know that I can’t do that with an online survey.” (see table S58, Quote No. 2 in the supplementary data online). One cited disadvantage, common to both modes, is the burden of having to proactively contact the survey institute in the case of misunderstandings or ambiguities. Further cited advantages and disadvantages of web and mail modes from the short interviews are summarized in table 3. Additional quotes from the in-depth interviews are provided in Supplement H.3 in the supplementary data online.

Table 3.

Advantages and Disadvantages of Web and Mail Modes

Web		Mail
Advantages	Disadvantages	Advantages	Disadvantages
Time and local flexibility (23) Speed (11) Easy use of internal documents (7) Easy handling processes (6) Easy to enter/correct (4) Sustainability (1)	One-way communication prevents clarification of questions or communication of additional information (4) Cumbersome log-in and handling processes (4) Low memory capacity (3) Error-proneness of answering the questionnaire (2) Data privacy concerns (2)	Time flexibility (12) Easy use of internal documents (7) High memory capacity (4) Better distribution within the establishment (2) Questionnaire easy to handle (2) Comparability with previous surveys / knowledge documentation (1) Speed (1) More intensive engagement with questionnaire (1)	Long processing time (9) Cumbersome handling (8) Demanding return (6) Unsustainable (6) One-way communication prevents clarification of questions or communication of additional information (6) Does not fit so well in operating procedure (2) Error-proneness of answering the questionnaire (1)

Web		Mail
Advantages	Disadvantages	Advantages	Disadvantages
Time and local flexibility (23) Speed (11) Easy use of internal documents (7) Easy handling processes (6) Easy to enter/correct (4) Sustainability (1)	One-way communication prevents clarification of questions or communication of additional information (4) Cumbersome log-in and handling processes (4) Low memory capacity (3) Error-proneness of answering the questionnaire (2) Data privacy concerns (2)	Time flexibility (12) Easy use of internal documents (7) High memory capacity (4) Better distribution within the establishment (2) Questionnaire easy to handle (2) Comparability with previous surveys / knowledge documentation (1) Speed (1) More intensive engagement with questionnaire (1)	Long processing time (9) Cumbersome handling (8) Demanding return (6) Unsustainable (6) One-way communication prevents clarification of questions or communication of additional information (6) Does not fit so well in operating procedure (2) Error-proneness of answering the questionnaire (1)

Note.—Number of mentions in parentheses. Establishments were asked about their mode preference and the perceived advantages and disadvantages of web, mail and telephone interviews. Telephone interviews are not in the focus of this article and hence are not displayed here.

Source.—46 qualitative interviews 2022.

Table 3.

Advantages and Disadvantages of Web and Mail Modes

Web		Mail
Advantages	Disadvantages	Advantages	Disadvantages
Time and local flexibility (23) Speed (11) Easy use of internal documents (7) Easy handling processes (6) Easy to enter/correct (4) Sustainability (1)	One-way communication prevents clarification of questions or communication of additional information (4) Cumbersome log-in and handling processes (4) Low memory capacity (3) Error-proneness of answering the questionnaire (2) Data privacy concerns (2)	Time flexibility (12) Easy use of internal documents (7) High memory capacity (4) Better distribution within the establishment (2) Questionnaire easy to handle (2) Comparability with previous surveys / knowledge documentation (1) Speed (1) More intensive engagement with questionnaire (1)	Long processing time (9) Cumbersome handling (8) Demanding return (6) Unsustainable (6) One-way communication prevents clarification of questions or communication of additional information (6) Does not fit so well in operating procedure (2) Error-proneness of answering the questionnaire (1)

Web		Mail
Advantages	Disadvantages	Advantages	Disadvantages
Time and local flexibility (23) Speed (11) Easy use of internal documents (7) Easy handling processes (6) Easy to enter/correct (4) Sustainability (1)	One-way communication prevents clarification of questions or communication of additional information (4) Cumbersome log-in and handling processes (4) Low memory capacity (3) Error-proneness of answering the questionnaire (2) Data privacy concerns (2)	Time flexibility (12) Easy use of internal documents (7) High memory capacity (4) Better distribution within the establishment (2) Questionnaire easy to handle (2) Comparability with previous surveys / knowledge documentation (1) Speed (1) More intensive engagement with questionnaire (1)	Long processing time (9) Cumbersome handling (8) Demanding return (6) Unsustainable (6) One-way communication prevents clarification of questions or communication of additional information (6) Does not fit so well in operating procedure (2) Error-proneness of answering the questionnaire (1)

Source.—46 qualitative interviews 2022.

7. DISCUSSION

This study evaluated the impacts of an experiment comparing a concurrent mail–web mixed-mode, a sequential web-to-mail mixed-mode, a single-mode web, and a single-mode mail design on survey participation in a large-scale establishment survey. The findings are useful for survey practitioners seeking to maximize participation rates and minimize nonresponse bias and costs in voluntary establishment surveys. The main findings can be summarized as follows. The experiment did not reveal any substantial differences in response rates between the four mode designs. Similarly, the four mode designs did not show meaningful differences with respect to aggregate (proxy) nonresponse bias. However, there were a few differences in subgroup participation between the mode designs. In line with the literature (Kaiser 2001; Dickey and Riberas 2007; Jones and Phipps 2010; Thompson et al. 2015), larger establishments were more likely to participate in the single-mode web survey design than in the single-mode mail survey design. In the concurrent mixed-mode design, all establishment size classes were more likely to choose the mail mode, but the difference in the likelihood of participation between the web and mail modes was smallest for the largest establishments. We found establishments in the agriculture/forestry and construction industries to be more likely to participate via mail in the concurrent mail–web mixed-mode design, but this result did not appear in the single-mode design comparison. Against our expectations, there was no strong evidence that establishments in the information/communication, finance/insurance, and public administration industries have stronger preference for participating via web than mail (or vice versa) in a concurrent mail–web mixed-mode or in a single-mode design.

Lastly, the cost analysis showed that push-to-web designs, such as a single-mode web and a sequential web-to-mail design can achieve substantial per-respondent cost savings of more than 50 and 25 percent, respectively, compared to a concurrent mail–web mixed-mode design. A single-mode mail design, in contrast, resulted in 19 percent higher costs per respondent than the concurrent mail–web mixed-mode design. However, we note that these cost results account for variable costs only, which are typically more pronounced in mail modes than in web modes. Future research could examine whether cost savings can be achieved when fixed costs, including software and programming the web instrument, are considered.

The accompanying qualitative study provided additional insights into how establishments weigh the pros and cons of web and mail modes when deciding whether to participate in a voluntary survey. In general, the participants preferred web over mail. While most participants did not explicitly state that mode has an impact on their willingness to participate, they did repeatedly mention that web surveys require less effort to respond compared to mail. Other key advantages of the web mode that were cited include sustainability, modernness, and facilitated handling of the questionnaire, which were important factors for the establishments. The fact that participants qualitatively preferred the web mode contrasts with the quantitative findings, where the mail mode was chosen at a higher rate than web in the concurrent mixed-mode design. This conflicting finding could be driven by at least two factors. First, the qualitative interviews were conducted 1.5 years after the IAB-JVS mode design experiments and overlapped with the COVID-19 pandemic when establishments were becoming more accustomed to working online. Thus, attitudes toward online surveys may have changed during this time. This may explain why for some of the same establishments their reported mode preference differed from their actual mode choice. Second, the paper questionnaire was sent along with the invitation to the concurrent mixed-mode design. As one qualitative participant noted, it was perhaps easier for many establishments to simply fill out the paper questionnaire on the spot rather than expend the effort of logging into the website, or they possibly perceived the paper questionnaire to be the priority mode given that they were contacted by postal mail rather than by email. This may also explain why there were similar discrepancies in the quantitative study, where some establishment subgroups were more likely to participate via mail than web in the concurrent design, but more likely to participate via web than mail in the single-mode designs. We suspect that had we reversed the delivery mode used in the concurrent design by emailing the survey invitation with a link to the web survey and attaching a PDF version of the questionnaire, which the establishment could optionally print and mail-back to the survey institute, this would have resulted in a higher web take-up rate compared to the mail mode and made the results more consistent across quantitative and qualitative studies. However, we can only speculate on the causes of these discrepancies and their remedies and suggest that they be investigated further in future work.

Forgoing mail questionnaires entirely and adopting a web-only mode design, as some National Statistical Institutes have done (see section 2.1), did not yield any negative impacts in our study. We attribute this result to several factors. First, the transformative impact of the COVID-19 pandemic on establishment practices, such as the prevalence of remote and hybrid working arrangements, has likely increased the burden associated with sharing paper questionnaires within establishments. Second, the pandemic has stimulated advances in IT infrastructure and skills among establishments and employees, potentially fostering a greater willingness among previously reluctant establishments to respond online. And lastly, ongoing improvements to web questionnaires (e.g., delegation functions) are reducing the perceived burden of responding online.

This study fills important gaps in the literature regarding the impact of self-administered mode designs on voluntary establishment survey participation. The lack of recent experimental evidence in this area is particularly notable given recent changes in Internet availability and usage by businesses (OECD 2023), but also increased work-from-home and flexible working patterns adopted by establishments in response to (and, in many cases, continuing beyond) the COVID-19 pandemic. Thus, we believe our findings are applicable to the post-pandemic situation, though further research is needed to confirm. Our study further adds to the literature by shining light on the effectiveness of various self-administered mode designs in a general population sample of establishments, which complements previous studies that have focused on more specific establishment populations (Erikson 2007; Downey et al. 2007; Bremner 2011; Harris-Kojetin et al. 2013; Ellis et al. 2013; Millar et al. 2018). An additional strength of the study is the comprehensive examination of several outcomes of interest: response rates, nonresponse bias, subgroup participation, and survey costs across different mode designs, which is rare in the establishment literature. The utilization of a mixed-methods approach combining quantitative experiments with qualitative interviews is another unique feature of this study, as it allowed for an examination of the pros and cons of using web and mail modes from the establishments’ perspective and thus shed light on the reasons for their participation decisions. Future methodological research might consider using a similar mixed-methods approach to analyze the drivers of establishment survey participation more generally and examine them from different perspectives.

In conclusion, the findings suggest using “push-to-web” designs for self-administered establishment surveys, implemented as either a single-mode web survey or a sequential web-to-mail mixed-mode survey. These designs can yield similar response rates and levels of nonresponse bias compared to concurrent mail–web mixed-mode designs, but with significantly lower per-respondent costs. This can also be viewed as an endorsement for the growing shift toward using web surveys as a supplement, or replacement, of mail surveys by researchers and statistical agencies.

Supplementary Materials

Supplementary materials are available online at academic.oup.com/jssam.

We thank the “Data- and IT-Management” Department at the IAB for data provision and the colleagues from Pro-IAB for recruiting and conducting the qualitative interviews. We are grateful to Economix Research & Consulting for implementing the experiment. We thank Mocja Bavdaž, Mario Bossler, Nicole Gürtzgen, Alexander Kubis, Martin Popp, and Marieke Volkert for helpful comments and suggestions. We thank Franka Vetter for valuable research assistance. We are thankful for comments by participants at the 2021 European Survey Research Association Conference, the 2021 European Sociological Association Conference, the 2022 Federal Computer Assisted Survey Information Collection Workshop, the 2022 European Conference on Quality in Official Statistics, the 2022 Business Data Collection Methodology Workshop, the 2022 German Online Conference, the 2023 workshop of the Method Section of the German Sociological Association, and an internal IAB seminar. This study design and analysis was not preregistered.

REFERENCES

American Association for Public Opinion Research

(

2016

Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys

(9th ed.),

Alexandria, VA

: AAPOR.

Google Preview

Bavdaž

(

2010

), “

The Multidimensional Integral Business Survey Response Model

,”

Survey Methodology

–

Bavdaž

Snijkers

Sakshaug

J. W.

Brand

Haraldsen

Kurban

Saraiva

Willimack

D. K.

(

2020

), “

Business Data Collection Methodology: Current State and Future Outlook

,”

Statistical Journal of the IAOS

741

–

756

Bossler

Gürtzgen

Kubis

Küfner

Lochner

(

2020

), “

The IAB Job Vacancy Survey: Design and Research Potential

,”

Journal for Labour Market Research

–

Bossler

Gürtzgen

Kubis

Küfner

Popp

(

2022

), “The IAB Job Vacancy Survey: Establishment Survey on Labour Demand and Recruitment Processes, Waves 2000 to 2019 and Subsequent Quarters 2006 to 2021,” FDZ-Datenreport 08/2022, Research Data Centre of the Federal Employment Agency, Nürnberg.

Bremner

(

2011

), “An Investigation Into the Use of Mixed Mode Data Collection Methods for UK Business Surveys,” in Proceedings of the BLUE-ETS Conference on Burden and Motivation in Official Business Surveys, Heerlen: Statistics Netherlands, pp.

217

–

220

Buiten

Snijkers

Saraiva

Erikson

A.-G.

Born

(

2018

), “

Business Data Collection: Toward Electronic Data Interchange. experiences in Portugal, Canada, Sweden, and The Netherlands with Edi

,”

Journal of Official Statistics

419

–

443

Christianson

Tortora

(

1995

), “Issues in Surveying Businesses: An International Survey,” in

Business Survey Methods

Cox

eds. B.

Binder

D. A.

Chinnappa

B. N.

Christianson

Colledge

M. J.

Kott

P. S.

Hoboken, NJ

: John Wiley & Sons,

237

–

256

Google Preview

Cohen

McCarthy

Rosen

Wiatrowski

(

2006

), “

Internet Collection at the Bureau of Labor Statistics: An Option to Report Data

,”

Monthly Labor Review

129

–

De Leeuw

E. D.

(

2005

), “

To Mix or Not to Mix Data Collection Modes in Surveys

,”

Journal of Official Statistics

233

–

255

De Leeuw

E. D.

(

2018

), “

Mixed-Mode: Past, Present, and Future

,”

Survey Research Methods

–

Dickey

N. J.

Riberas

Z. T.

(

2007

), “Do Economic and Demographic Characteristics Differ Between Web and Mail Respondents to the 2005 Census of Agriculture Content Test?” in Proceedings of the Third International Conference on Establishment Statistics, Montréal, Canada: American Statistical Association, pp.

375

–

381

Dillman

D. A.

(

2017

), “

The Promise and Challenge of Pushing Respondents to the Web in Mixed-Mode Surveys

,”

Survey Methodology

–

Dillman

D. A.

Messer

B. L.

(

2010

), “Mixed-Mode Surveys,” in

Handbook of Survey Research

, eds.

Mardsen

P. V.

Wright

J. D.

Bingley, UK

Emerald Group Publishing Limited

, pp.

551

–

574

Google Preview

Dillman

D. A.

Phelps

Tortora

Swift

Kohrell

Berck

Messer

B. L.

(

2009

), “

Response Rate and Measurement Differences in Mixed-Mode Surveys Using Mail, Telephone, Interactive Voice Response (IVR) and the Internet

,”

Social Science Research

–

Downey

McCarthy

(

2007

), “Encouraging the Use of Alternative Modes of Electronic Data Collection: Results of Two Field Studies,” in Proceedings of the Third International Conference on Establishment Statistics, Montréal, Canada: American Statistical Association, pp.

517

–

524

Eckman

Haas

G.-C.

(

2017

), “

Does Granting Linkage Consent in the Beginning of the Questionnaire Affect Data Quality?

,”

Journal of Survey Statistics and Methodology

535

–

551

Egeln

Gottschalk

Lubczyk

(

2023

), “Technischer anhang zur befragung 2022,” Technical report, ZEW and IAB.

Ellis

Aspinwall

Heinrich

Ginder

McDonald

H. S.

Noonan

(

2013

), “The Effects of Pushing Web in a Mixed-Mode Establishment Data Collection,” in Proceedings of the 68th AAPOR Conference, Boston, MA: AAPOR, pp.

4437

–

4450

Erikson

(

2007

), “Effects of Offering Web Questionnaires As An Option in Enterprise Surveys: The Swedish Experience,” in Proceedings of the Third International Conference on Establishment Statistics.

Ganzer

Schmucker

Stegmaier

Stüber

(

2022

), “Establishment History Panel 1975-2020,” FDZ-Datenreport 03/2022, Research Data Centre of the Federal Employment Agency, Nürnberg.

Gensicke

Tschersich

Kohaut

Mielewczyk

Künzel

(

2022

), “Technical Report on the IAB Establishment Panel—Wave 29 (2021),” FDZ-Methodenreport 07/2022, Research Data Centre of the Federal Employment Agency, Nürnberg.

Giesen

(

2007

), “The Response Process Model As A Tool For Evaluating Business Surveys,” in Proceedings of the Third International Conference on Establishment Statistics, Montréal, Canada: American Statistical Association, pp.

871

–

880

Gleiser

Sakshaug

J. W.

Volkert

Ellguth

Kohaut

Möller

(

2022

), “

Introducing Web in a Mixed-Mode Establishment Survey: Effects on Nonresponse

,”

Journal of the Royal Statistical Society: Series A

185

891

–

915

Haas

G.-C.

Eckman

Bach

(

2021

), “

Comparing the Response Burden between Paper and Web Modes in Establishment Surveys

,”

Journal of Official Statistics

907

–

930

Haraldsen

Snijkers

Roos

Sundvoll

Vik

Stax

H.-P.

(

2011

), “Utilizing Web Technology in Business Data Collection: Some Norwegian, Dutch and Danish Experiences,” in Proceedings of the Conference on New Techniques and Technologies for Statistics.

Hardigan

P. C.

Succar

C. T.

Fleisher

J. M.

(

2012

), “

An Analysis of Response Rate and Economic Costs between Mail and Web-Based Surveys among Practicing Dentists: A Randomized Trial

,”

Journal of Community Health

383

–

394

Harris-Kojetin

Sengupta

Hobbs

Greene

Eicheldinger

(

2013

), “Obtaining Participation Via Less Expensive Modes: Experiments In A Mixed-Mode Establishment Survey,” in Proceedings of the 68th AAPOR Conference.

Jones

Phipps

(

2010

), “Using Establishment Characteristics to Predict Respondent Mode Preferences in the Occupational Employment Statistics Survey,” in Statistical Survey Paper at the U.S. Bureau of Labour Statistics.

Jones

Ryan

Lanyon

A. J.

Apostolou

Price

König

Volkert

Sakshaug

J. W.

Mead

Baird

Elliott

McLaren

C. H.

(

2023

), “Producing Official Statistics during the Covid -19 Pandemic,” in

Advances in Business Statistics, Methods and Data Collection, eds

Snijkers

Bavdaz

Bender

Jones

MacFeely

Sakshaug

J. W.

Thompson

K. J.

van Delden

Hoboken, NJ

: John Wiley & Sons, pp.

225

–

264

Kaiser

(

2001

), “Differences in Response Patterns in a Mixed Mode—Online/Paper & Pencil Business Survey,” ZEW Discussion Paper 01-50, Centre for European Economic Research, Mannheim.

Klausch

Hox

J. J.

Schouten

(

2013

), “

Measurement Effects of Survey Mode on the Equivalence of Attitudinal Rating Scale Questions

,”

Sociological Methods & Research

227

–

263

König

Sakshaug

J. W.

Stegmaier

Kohaut

(

2021

), “

Trends in Establishment Survey Nonresponse Rates and Nonresponse Bias: Evidence from the 2001-2017 IAB Establishment Panel

,”

Journal of Official Statistics

931

–

953

Kreuter

Müller

Trappmann

(

2010

), “

Nonresponse and Measurement Error in Employment Research: Making Use of Administrative Data

,”

Public Opinion Quarterly

880

–

906

Küfner

Sakshaug

J. W.

Zins

(

2022

), “

Analysing Establishment Survey Non-Response Using Administrative Data and Machine Learning

,”

Journal of the Royal Statistical Society Series A

185

310

–

342

Langeland

J. L.

(

2019

), “The Use of Email in Establishment Surveys,” PhD thesis, University of Maryland, College Park.

Medway

R. L.

Fulton

(

2012

), “

When More Gets You Less: A Meta-Analysis of the Effect of Concurrent Web Options on Mail Survey Response Rates

,”

Public Opinion Quarterly

733

–

746

Millar

M. M.

Schmuhl

Page

Genovesi

A. L.

Ely

Hemingway

Olson

L. M.

(

2018

), “

Improving Response to an Establishment Survey through the Use of Web-Push Data Collection Methods

,”

Mathematical Population Studies

168

–

179

Mize

T. D.

(

2019

), “

Best Practices for Estimating, Interpreting, and Presenting Nonlinear Interaction Effects

,”

Sociological Science

–

117

OECD

(

2023

), “ICT Access and Usage by Businesses.” Data extracted from OECD.Stat.

Office for National Statistics

(

2022

), “Business Insights and Conditions Survey (BICS) QMI,” Available at https://www.ons.gov.uk/economy/economicoutputandproductivity/output/methodologies/businessinsightsandconditionssurveybicsqmi.

Olson

Smyth

J. D.

Horwitz

Keeter

Lesser

Marken

Mathiowetz

N. A.

McCarthy

J. S.

O’Brien

Opsomer

J. D.

Steiger

Sterrett

Suzer-Gurtekin

Z. T.

Turakhia

Wagner

(

2021

), “

Transitions from Telephone Surveys to Self-Administered and Mixed-Mode Surveys: AAPOR Task Force Report

,”

Journal of Survey Statistics and Methodology

381

–

411

Olson

Smyth

J. D.

Horwitz

Keeter

Lesser

Marken

Mathiowetz

McCarthy

O’Brien

Opsomer

Steiger

Sterrett

Suzer-Gurtekin

Z. T.

Turakhia

Wagner

(

2019

), “Report of the AAPOR Task Force on Transitions From Telephone Surveys to Self-Administered and Mixed-Mode Surveys,” Task force report, American Association for Public Opinion Research.

Sakshaug

J. W.

Vicari

Couper

M. P.

(

2019

), “

Paper, E-mail, or Both? Effects of Contact Mode on Participation in a Web Survey of Establishments

,”

Social Science Computer Review

750

–

765

Snijkers

Jones

(

2013

), “Business Survey Communication,” in

Designing and Conducting Business Surveys

, eds.

Snijkers

Haraldsen

Jones

Willimack

D. K.

Hoboken, NJ

: Wiley Online Library, pp.

359

–

430

Snijkers

Geurden-Slis

Goris

Burger

van den Hombergh

(

2018

), “The Effect of Response Measures in Business Surveys,” in Proceedings of the Workshop on Statistical Data Collection.

Snijkers

Göttgens

Hermans

(

2011

), “Data Collection and Data Sharing at Statistics Netherlands: Yesterday, Today, Tomorrow,” in Proceedings of 59th Plenary Session of the Conference of European Statisticians (CES).

StataCorp

(

2021

), “Stata Statistical Software: Release 17,” College Station, TX: StataCorp LLC.

Thompson

K. J.

Oliver

Beck

(

2015

), “

An Analysis of the Mixed Collection Modes for Two Business Surveys Conducted by the us Census Bureau

,”

Public Opinion Quarterly

769

–

789

U.S. Bureau of Labor Statistics

(

2020

), “Survey of Occupational Injuries and Illnesses: Data Sources.” Available at https://www.bls.gov/opub/hom/soii/data.htm.

U.S. Bureau of Labor Statistics

(

2023

), “Household and Establishment Survey Response Rates.” Available at https://www.bls.gov/osmr/response-rates/.

U.S. Census Bureau

(

2022a

), “How Do I Get Started.” Available at https://www.census.gov/topics/business-economy/information/response/get-started.html#par_textimage_10.

U.S. Census Bureau

(

2022b

), “Methodology: Methodology.” Available at https://portal.census.gov/pulse/data/#methodology.

U.S. Census Bureau

(

2023a

), “Economic Census Methodology: 2017.” Available at https://www.census.gov/programs-surveys/economic-census/year/2017/economic-census-2017/technical-documentation/methodology.html.

U.S. Census Bureau

(

2023b

), “Frequently Asked Questions—About the Economic Census.” https://www.census.gov/programs-surveys/economic-census/year/2022/about/faq/faq-general.html.

Willimack

D. K.

McCarthy

J. S.

(

2019

), “Obstacles and Opportunities for Experiments in Establishment Surveys Supporting Official Statistics,” in

Experimental Methods in Survey Research: Techniques That Combine Random Sampling with Random Assignment

, eds.

Lavrakas

Traugott

Courtney

Holbrook

de Leeuw

West

Bingley, UK

Wiley Online Library

, , pp.

309

–

325

Willimack

D. K.

Nichols

(

2010

), “

A Hybrid Response Process Model for Business Surveys

,”

Journal of Official Statistics

Wolf

Christmann

Gummer

Schnaudt

Verhoeven

(

2021

), “

Conducting General Social Surveys as Self-Administered Mixed-Mode Surveys

,”

Public Opinion Quarterly

623

–

648