Farmers’ attitudes toward economic experimentation

Abstract

Economic experiments have gained popularity in Agricultural Economics. However, challenges in recruitment and acceptance of experimental methods among farmers persist. Surveying 406 Swedish farmers, we explore farmers’ attitudes toward different types and features of economic experiments. Only a quarter of participants accept a randomized controlled trial on agri-environmental schemes where a control group is excluded from payment eligibility. When the randomization varies the size of payments, acceptance drops to only 7 per cent. Despite being standard practice in lab-in-the-field experiments, farmers strongly reject behavior-contingent payments. Engaging farmers during study design and sharing results with them can ameliorate recruitment challenges.

1. Introduction

Economic experiments have gained popularity in Agricultural Economics, because they offer clean causal research designs to evaluate the effectiveness of different policies and programs, and can provide valuable insights into the factors that influence farmers’ decision-making (Colen et al. 2016; Palm-Forster and Messer 2021), making them an important complement to causal inference for observational data (Wuepper and Finger 2023; Henningsen et al. 2024). The use of randomized controlled trials (RCTs) in the evaluation of agricultural policies and programs is rising rapidly in low-income countries, but remains largely limited in high-income countries (Behaghel et al. 2019). In high-income countries, there is a paucity of RCTs in the field of agricultural policy (Behaghel et al. 2019; Morawetz and Tribl 2020; Chabé-Ferret et al. 2023; Alif et al. 2024), which are costly to run and face acceptance barriers among farmers (Behaghel et al. 2019; Morawetz and Tribl 2020) and policy-makers (Höhler et al. 2023). At the same time, the experimental literature offering behavioral insights into farmers’ decision-making in high-income countries based on (lab-in-the-field) experiments and stated preferences studies is growing (Colen et al. 2016; Kuhfuss et al. 2016; Dessart et al. 2019; Palm-Forster and Messer 2021; Schulze et al. 2023), but also here researchers report difficulties in recruiting farmers for such experiments (Lefebvre et al. 2021; Rosch et al. 2021; Weigel et al. 2021; Baaken et al. 2024).

While there is a rapid uptake of modern causal inference methods among agricultural economists (Wuepper and Finger 2023; Henningsen et al. 2024), experiments allow the researchers to deliberately change a treatment. A better understanding of the acceptance of experiments may help to facilitate recruitment, leading to higher and more representative participation, and thereby helping to inform more accurately policies and programs aimed at promoting sustainable and efficient farming practices. Awareness about and consideration of farmers’ concerns over (elements of) experimental practices can help to build a better relationship between researchers, farmers, and program managers with the ultimate goal of more and better evaluations of policies. Especially in the field of Agricultural Economics, where researchers are often working closely and repeatedly with close-knit agricultural communities—including farmers, but also extension workers and local policy officers—, researchers’ actions and attitudes can either strengthen or damage relationships and thereby generate or limit future opportunities for research (Palm-Forster and Messer 2021; Höhler et al. 2023). Consideration of farmers’ attitudes can increase the chances of future collaboration and also help to ensure that research is relevant and responsive to the needs and concerns of farmers.

While there is a small literature on the ethics of field experiments (List 2008; Glennerster and Powers 2016), these studies discuss the ethics of experimental practices from a normative point of view, on which there is not necessarily consensus and which do not automatically correspond to what is perceived as ethical by those participating. In this paper, we empirically assess the opinion of the farmers, providing practical insights on which types and features of experimental practices are more or less accepted and which ones are seen as especially problematic. We do this based on a survey experiment and a survey among 406 Swedish farmers. By means of a recruitment experiment that varies incentives, we also test who selects into our study.

Our contributions to the literature are three-fold. First, through the survey experiment, we study the acceptance of different formats of an RCT, contributing to the small empirical literature on the acceptance of RCTs among farmers (Behaghel et al. 2019; Morawetz and Tribl 2020) and the overall literature on the acceptance of RCTs and randomization (Haushofer et al. 2019; Meyer et al. 2019; Heck et al. 2020; Mislavsky et al. 2020; Corduneanu-Huci et al. 2021; Mazar et al. 2023; DellaVigna et al. 2024; Dur et al. 2025). We investigate how the eligibility to participate (full exclusion from participation vs. having either a randomly assigned high or low payment) and the respondents’ outcome after randomization (being among the “lucky” or “unlucky” ones) affects acceptance. Second, through statements of an item battery, we explore attitudes toward specific details in the implementation of lab-in-the-field experiments (Rosch et al. 2021; Höhler et al. 2023). Third, we test whether different payment modes for motivating farmers to take part in research activities result in different recruitment rates (Weigel et al. 2021) or in differences in self-selection into participation. Overall, our study provides insights into the challenges of recruitment and acceptance of economic experiments among farmers, and points out practical improvements for the design of future experiments.

2. Types of economic experiments, acceptance of economic experiments, and recruitment challenges

The literature provides various definitions of economic experiments and field experiments. We follow Harrison and List (2004) and Colen et al. (2016) to distinguish (1) stated preferences studies and survey experiments, (2) laboratory experiments, (3) lab-in-the field experiments, (4) and RCTs. First, stated preferences studies such as discrete choice experiments and survey experiments use randomization to study the acceptance of policies and programs for instance in environmental economics (Mariel et al. 2021). Participants make hypothetical choices with no financial consequences for themselves. Second, laboratory experiments use monetary incentives to study economic decisions under induced value theory (Smith 1976). Participants are typically undergraduate students and tasks are abstractions of reality. Third, in contrast, lab-in-the field experiments target non-student subjects, such as farmers. Abstract lab-in-the field experiments are sometimes called artefactual field experiments (Harrison and List 2004), whereas experiments that introduce additional context are sometimes called framed field experiments (Harrison and List 2004). In both cases, participants do not make decisions in their natural environment, which distinguishes lab-in-the field experiments from field experiments or so-called RCTs. While there is a growing literature of stated preferences studies and lab-in-the-field experiments among European farmers, RCTs with farmers are still very rare in Europe (Lefebvre et al. 2021).

2.1 Potential of RCTs for EU agricultural policy design and outcome-based agricultural policy

The Common Agricultural Policy (CAP) of the European Union provides almost 60 billion Euros of subsidies per year to ten million farms across Europe (European Commission 2022). The CAP is progressively channeling its budget toward environmental objectives, including landscape preservation and biodiversity (European Commission 2022), with a growing interest in voluntary schemes such as agri-environmental contracts, possibly including outcome-based payments (Hasler et al. 2022). These schemes, often also termed result-based, operate by compensating farmers based on the measurement of specific environmental indicators.¹ Upon meeting indicator-based performance criteria, farmers become eligible for full payments or top-ups.

The implementation and evaluation of outcome-based schemes is intricate, facing challenges in design, execution, and monitoring. Farmers may perceive these schemes as riskier due to their reliance on results and imperfect indicators (Šumrada et al. 2021; Tanaka et al. 2022; Canessa et al. 2023). To evaluate outcome-based schemes, randomly assigning control groups with no payments or lower payments is desirable. This approach would facilitate the differentiation of outcomes stemming from incentives. While the CAP's evaluation has traditionally leaned on EU-wide simulation models, surveys, qualitative analyses, and stakeholder interviews, it is increasingly apparent that these methods have limitations in establishing causal relationships between policy interventions and outcomes (Colen et al. 2016). The European Commission's comprehensive monitoring and evaluation framework underscores the need for improved evaluation methods that can confidently attribute outcomes to specific policy interventions (European Commission 2017). Consequently, researchers advocate the incorporation of experimental methodologies such as economic experiments and RCTs into the CAP evaluation toolkit (Colen et al. 2016).

RCTs are often considered to be the gold standard in the clinical trial literature (Bothwell et al. 2016). Randomization closes causal backdoors that would otherwise emerge from confounds or self-selection into treatment. In many contexts, RCTs are an effective means to overcome the fundamental evaluation problem. For example, if a voluntary agri-environmental policy was offered to all farmers in a country, one could not distinguish general trends from policy impacts, and thus could not assess whether additional environmental benefits are generated from the policy. Several RCTs with farmer participants have been organized in lower-income countries (Duflo et al. 2011; Fafchamps and Minten 2012; Giné et al. 2012; Bulte et al. 2014; Bold et al. 2022), but applications in the field of agricultural policy remain limited in high-income countries and would often focus on behavioral interventions, such as social norm nudges (e.g. Chabé-Ferret et al. 2023). The use of RCTs is less obvious for more traditional CAP measures, such as direct income support or market-based measures. Yet, the voluntary enrollment of farmers in the more recent agri-environmental measures, with different potential payment schemes, generates opportunities for randomization in the implementation or gradual roll-out of such programs, allowing to compare a treatment and a control group and to provide robust causal impact evaluation (Colen et al. 2016).

2.2 Challenges in enhancing the use of experimental methods: acceptance and recruitment

While there has been an expansion of the application of various types of economic experiments aiming to understand farmer decision-making and the role of behavioural factors therein (Dessart et al. 2019; Schaub et al. 2023; Wuepper et al. 2023), the implementation of RCTs for the evaluation of the CAP has not taken off. RCTs, while potentially valuable for assessing CAP measures, face significant challenges that have hindered their application thus far. There is a general reluctance to run experiments within organizations (Ferraro et al. 2023). Obstacles include the long timelines for designing and implementing experiments, the possible long delays in full effects to materialize, the legal and regulatory constraints, and ethical concerns over differential treatments. The inconsistency of applying policies exclusively to a subset of European farmers, deviates from prevailing EU norms and may encounter regulatory opposition (Lefebvre et al. 2021).² The exclusion of farmers from payment schemes or policy measures in a random manner raises fairness and ethical concerns (Baele 2013), leading to potential limitations in the acceptance of RCTs among farmers and policymakers. A limited number of studies, focusing on other policy domains, have investigated which features may make RCTs more acceptable. Meyer et al. (2019) and Heck et al. (2020) find a general aversion to experiments, while Mislavsky et al. (2020); Mazar et al. (2023), and Dur et al. (2025) challenge this, and find that experiments are at least as acceptable as the intervention or policy change they aim to test. Especially in the context of the CAP, which is already negatively viewed by many European farmers, this may result in experiments in this domain being viewed with skepticism by farmers, and lead to hesitance among policymakers as well.

Only few studies have investigated how different RCT designs may affect the acceptability among participants. We know of only one study evaluating the acceptance of an RCT in the context of the CAP: Morawetz and Tribl (2020) explore the acceptance of an innovative RCT model known as “upRCT,” contrasting it with a standard RCT design. While the standard RCT involves excluding a group from a measure, the upRCT involves replacing conditional payments with unconditional ones. In other words, an upRCT evaluates policy impacts by removing the monitoring and enforcement component of a novel policy where the up (unconditional payment) group serves as the control. Austrian farmers engaged in a “refrain from silage” agri-environmental payment scheme were surveyed to gauge the acceptance of this RCT design. By presenting hypothetical scenarios wherein groups are excluded from payment or receive unconditional payments in the following year, the study reveals that general acceptance is low, but that the acceptance rate for upRCT is approximately twice that of the conventional RCT.

The implementation of lab-in-the-field experiments typically poses fewer practical and ethical concerns compared to RCTs, as lab-in-the-field experiments typically operate in disconnection from actual programs or policy interventions. RCTs are more impacting as they affect participants’ real life, possibly involving risks and generating real costs or benefits both immediately and in the future. The drivers of decision-making factors, on the contrary, can often be studied from lab-in-the-field experiments without having consequences that carry beyond the duration of the experiment. Nevertheless, also for these types of economic experiments, understanding farmers’ acceptance of different features of experimental practice is important.

Several researchers have raised that the pool of agricultural producer participants is typically limited (Höhler et al. 2023) and that recruiting a sufficiently large and representative sample of farmers in economic experiments is challenging and expensive (Palm-Forster et al. 2019; Lefebvre et al. 2021; Rosch et al. 2021). Weigel et al. (2021) tested different recruitment strategies and found that regular mail invitations and higher monetary incentives enhance recruitment success among US farmers, but overall results were discouraging. For instance, none of several thousand farmers invited online, accepted the invitation to participate in an experimental study. Gajic et al. (2012) examine response rates under different payment schemes, including low lottery prices to a large subset of participants vs. high lottery prices to a small subset, finding a higher rate of completed surveys for the latter while involving much lower logistic and financial costs. Yet, the complexity of participation motivations involves factors beyond invitation methods and also pertains to details of the study design (Höhler et al. 2023). When farmers disapprove of certain experimental practices this will likely affect willingness to participate in (future) experiments and may break trust. While incentive payments and the prospect of real monetary stakes may motivate farmers’ participation, potential unequal outcomes of “winners” and “losers” might also generate ethical concerns and deter potential participants. Besides, also the involvement of stakeholders and collaboration between researchers and public and private agricultural organizations may increase trust and facilitate recruitment (Höhler et al. 2023).

3. Methods and data

Our methods follow the three objectives of the paper. First, to evaluate how different aspects of RCT design affect the acceptance of RCTs, we run a survey experiment with four treatment arms. Note that this is similar to a stated preferences study, as farmers do not face consequences for their decisions. Note also that the content of the preference is on the acceptance of an RCT though. Second, we explore the acceptance of specific experimental practices in lab-in-the-field experiments (including different forms of performance-based payments) by asking respondents to indicate their willingness to participate in a public goods game. While we base this on a simple survey question, the focus here is on the acceptance of lab-in-the-field experiments and their features. Hence, we focus on the acceptance of both RCTs and lab-in-the-field experiments. Third, to test whether different payment modalities result in different response rates or in selection bias, we randomly assigned participants to two types of e-mail invitations, each offering a different payment structure for motivating participation in the study. This part of the study qualifies as an RCT.

The online survey was sent out to 8,944 randomly selected email addresses of Swedish farmers registered at the Swedish Statistics Authority. About 60 per cent of the registered farms had provided an email address at the time of the study, across all farm sizes (see Table A4 in the Appendix for a comparison of the number of farms with and without an email address by size category). In that way, we aimed to target a large and representative sample of Swedish farmers. A financial incentive was offered to encourage participation in the survey, as suggested by Weigel et al. (2021). Data collection was carried out for approximately two weeks in January and February 2022.

Our survey instrument consisted of four parts: (1) the main survey experiment presenting four different scenarios of a hypothetical RCT discussed in this paper, (2) an item battery on the acceptance of various experimental practices in a commonly replicated lab-in-the-field experiment, (3) an item battery on the acceptance of nudges (based on Reisch and Sunstein 2016), and (4) a set of demographic questions on the respondents and their farms. The first three parts were presented in random order. Part (3) of the survey is the subject of a separate analysis (see Colen et al. 2024) and is not further discussed here.

3.1 Stated-preferences study on farmers’ acceptance of RCTs

In the main survey experiment, respondents are presented with a scenario of an RCT to evaluate the effectiveness of an agri-environmental scheme that provides payments to farmers to engage in environmentally friendly farming practices. The experiment consisted of four treatments in a between-subjects design, that is, each respondent received one scenario. We varied two factors with two levels each (2 × 2 design): the RCT design with respect to eligibility (EXCLU: full exclusion from participation vs. PAY: having either a randomly assigned high or low payment) and the respondents’ position (LUCK: being among the eligible or high-paying participants vs. UNLUCK: being among the non-eligible/low-paying participants).

In the full exclusion scenario (EXCLU), participants were presented with a hypothetical scenario in which one group of farmers would be randomly excluded from participating in the scheme to test its effectiveness. In contrast, in the second RCT scenario (PAY), survey participants were presented with a case in which one group of farmers would receive a higher payment while the other group of farmers would receive a lower payment. Hence, unlike in EXCLU, everyone can participate, but the payment levels are used to evaluate differences in enrollment and (cost) effectiveness.

The scenarios were either framed as the respondent being among the “lucky” (LUCK) participants (allowed to participate in EXCLU or receiving the higher payment in PAY) or among the “unlucky” (UNLUCK) participants (not being allowed to participate in EXCLU or receiving the lower payment in PAY). The scenarios are presented in Tables A1 and A2 of the Appendix.

Our outcome of interest is the discrete response to the question: “Do you find this approach to evaluate the effectiveness of the new agri-environmental scheme acceptable?” which respondents could answer with “Yes”, “Don't know/undecided”, and “No”. There also was a follow-up asking about reasons for the acceptance or non-acceptance. We consider the EXCLU RCT as being more restrictive, as it excludes one group from participation, and therefore expect it to be less accepted among farmers. In line with the findings of Haushofer et al. (2019), we expect farmers in the unlucky group to be less accepting. We therefore formulate the following hypotheses for the survey experiment:

  

3.2 Stated-preferences study on farmers’ acceptance of lab-in-the-field experiments

In this part of the survey, we presented a public goods game scenario to farmers and used it to ask about a series of statements regarding the details of implementing the public goods game (see Table A3 and the survey for details on the scenario). Hence, this part is informative on farmers’ acceptance of lab-in-the-field experiments. The public goods game which was originally developed by Isaac et al. (1984) has become one of the most replicated economic experiments (see e.g. Zelmer (2003) for an early meta-analysis). It is used to measure free-riding vs. pro-social behavior and also found its way into lab-in-the-field experiments in the Agricultural Economics literature (Bouma et al. 2020; Rommel, Schulze et al. 2023; Liu et al. 2024,). We use it here as an example of a typical experiment with strategic interdependence. Typically, these types of experiment are more difficult to implement with farmers. Our benchmark willingness to participate may hence present a lower bound.

In the standard version of the public goods game, each player of a group of participants is initially endowed with an equal amount of money or tokens. In the second step of the game, participants can contribute to the group account using a share from 0 per cent to 100 per cent of their initial endowment. The contributions of all players to the group account are then added up and multiplied with a number greater than one but smaller than the number of players (which creates a social dilemma). Finally, the thereby increased sum of the group account is equally redistributed to the players. In a one-shot game, the Nash equilibrium under standard preferences is to contribute zero, whereas the social optimum is to contribute all of the initial endowment to the group account.

Participants were then asked to indicate the probability of being willing to participate in a public goods game experiment on a seven-point Likert scale for a total of eight statements. The scale ranged from “Much less likely to participate” to “Much more likely to participate”. Three out of eight statements are about the design process (were farmers involved in the design) and consequentiality of the study (are results communicated to policy-makers; is a summary of results sent to participants). Another four statements relate to payments being given to participants (all participants receive a small payment; only some randomly selected participants receive a larger payment), and these payments being linked to actions (payments being dependent on the participants’ own decisions within the game; dependent also on decisions of other participants). Such behavior-linked payments and the interdependence of payments of participants are core principles of game theory experiments. The final statement relates to false information (the experimenter giving false information on contributions of other participants, to test responses to own contributions).

3.3 RCT on the effect of payment modalities on the recruitment of farmers

To test for differential response and learn about possible selection bias as a result of the type of financial incentive offered, we randomly offered two types of incentives: half of the email invitations were sent out offering a 60 SEK (Swedish crowns, approximately 5 Euros/US Dollars) payment. The other half were offered a 600 SEK (approximately 50 Euros/US Dollars) lottery ticket with a 10 per cent chance of receiving it (representing the same expected value and same expected utility under risk-neutrality). While these incentives may be relatively low relative to farmers’ opportunity cost, they are common in similar surveys. Survey respondents were given the option to enter their email address after answering the question to later receive the payment or participate in the lottery, respectively. For ethical reasons, we chose amounts with the same expected value and did not include a non-incentivized control condition.

4. Results

4.1 Summary statistics

Of the invited 8,944 farmers, 671 started the survey, 568 answered at least one question, and 406 completed the survey till the end (= 4.5 per cent response rate). This response rate is substantially higher than for some experiments that used the same sampling approach (e.g. Rommel, Sagebiel et al. (2023) who achieved less than 2 per cent). However, it is lower than in a recent survey that has used frequent text message reminders and was on an issue that more directly addressed farmers (Oyinbo and Hansson 2024). Mail surveys are uncommon in Sweden, and the authors are not aware of any recent mail surveys.

The median response time was approximately 14 min among those who completed the full survey. Summary statistics for the respondents are provided in Table 1. We decided not to force answers to each question, resulting in some questions left unanswered by some of the participants, resulting in different sample sizes for the different parts of our analysis. In order not to reduce the sample size for the different parts of our analysis, we opted for not homogenizing the final sample used, and instead, we use all available data and report the corresponding sample size for each part of the analysis.

The summary statistics show that our sample consists mostly of male farmers (82 per cent), with an average age of 59. The farmers in our sample have on average 100 ha of land (arable, grassland and forest land, leased or owned) and 45 livestock units. Forty five per cent of farmers report earning more than 250,000 SEK in farm income. Farm income corresponds on average to 35 per cent of total income, and consists for 27 per cent of CAP payments, on average. Twenty one per cent of farms is at least partly, or in transition to, organic. Thirty eight per cent of farmers have experience with implementing agri-environmental schemes.

The summary statistics reported are similar to other samples from the same registry (e.g. Rommel, Sagebiel et al. 2023) and point to a slight overrepresentation of large farms as compared to the registry. Comparison of our sample to a representative sample of the Swedish farming population from 2016 (European Commission 2021) shows that our sample is representative in terms of gender and age, but confirms a bias toward larger farms (see Table A5 in Appendix).

4.2 Results of stated-preferences study on farmers’ acceptance of RCTs

Table 2 displays the share of respondents who replied with “Yes” to the question whether the presented hypothetical RCT scenarios was considered acceptable.

A first observation is that overall acceptance is low (15.52 per cent) across all conditions. In contrast to our expectation and hypothesis, the EXCLU scenario is much more likely to be accepted than the PAY scenario. The difference between the scenarios is statistically significant (z = 4.82; P > 0.001; two-sided two-sample test of proportions). A similar and statistically significantly higher rate of acceptance of the EXCLU over the PAY scenario is found within the subsamples of “lucky” (z = 3.33; P > 0.001) and “unlucky” respondents (z = 3.48; P > 0.001). We do not find large or statistically significant differences between being in the LUCK or UNLUCK treatment (neither within EXCLU and PAY nor when pooling). Thus, we also do not find support in our data for the second hypothesis.

Table 3 presents coefficient estimates from four binary logistic regression models with the dependent variable being one if the respondent accepts the RCT design. Model (1) includes only the treatment variables (three dummies with the reference category EXCLU = 1; LUCK = 1) and includes the full sample (N = 406). Model (2) is equal to Model (1) but includes only those observations for which also personal characteristics and farm characteristics are complete (N = 304), for comparison with Models (3) and (4). Model (3) adds personal characteristics of the respondent (gender and age), and model (4) adds farm characteristics. The results confirm the analysis above. There is a larger and statistically significant negative effect of the PAY scenario (as compared to EXCLU), but whether the respondent ends up in the lucky or unlucky group, does not matter for acceptance. None of the personal or farm-specific co-variates have a large or statistically significant effect on acceptance.

4.3 Results of stated-preferences study on farmers’ acceptance of lab-in-the-field experiments

Responses regarding the stated probability to participate in the public goods game show great variability among the eight statements about different modes of the experiment. Overall, the statements emphasizing aspects of equality and transparency received high rates of acceptance, while statements that contain aspects related to the provision of false information and stochastic payments or behavior-contingent payments received small acceptance. Notably, if an experiment is used in the policymaking process, farmers also have a more negative view of it. For an overview of the statements and the ratings per response category see Table 4.

The stated likeliness to participate in the public goods game is highest for the statement “The study was designed in collaboration with farmers”. Regarding the payment modalities, we see that random payments (“Only some randomly selected participants receive a payment for participation, but this payment is larger”) led to the lowest stated likeliness to participate. Also behavior-contingent payments received much lower acceptance, as compared to equal payment for all participants. This indicates that fixed payments are more acceptable than paying based on behavior, despite this being an established practice in experimental economics. The strong skepticism toward providing payment to only a randomly selected group of participants is in line with the skepticism toward RCTs in the survey experiment, where also the exclusion scenario received very low acceptance.

As a robustness test, we also ran random effects panel ordered logit regressions which account for the correlated errors within respondents. The dependent variable in these models is the ordinal ranking of the statement (from fully disagree to fully agree). Note that for each statement, a constant/cut is estimated. The models include statement dummies, individual, as well as farm characteristics (Table A6 in the Appendix). These models confirm the descriptive results: the different features of the design and payment modalities of the experiment as described in the statements, matter for farmers’ willingness to participate. It also shows that women and older people overall show a somewhat more negative attitude regarding acceptability of the given experimental designs.

4.4 Results of RCT on the effect of payment modalities on the recruitment of farmers

We did not find a difference in recruitment success between the two payment modes (a low payment for everyone in one treatment arm vs. a high but stochastic payment with the same expected value in another treatment arm). In the low payment group, 202 participants finished the questionnaire compared to 205 in the high payment group (Table A5 in Appendix). Note that the number of invitations was balanced. The two types of incentives can be related to two of the statements we tested for the lab-in-the-field experiment “Every participant receives a small payment for their participation” and “Only some randomly selected participants receive a payment for participation, but this payment is larger”. We tested if there is any difference in average responses to these statements between the two groups. The average response for the former statement was 4.23 among the group who received the low payment and 4.63 among the group who received the high but random payment (on a seven-point Likert scale). This difference is small but statistically significant (N = 400, t = −2.15, P = 0.03). For the latter statement, these mean values were 1.98 in the low payment group and 2.40 in the high but random payment group (N = 398, t = −2.72, P = 0.01). This effect is statistically significant but equally small. Generally, the group who received the high but random payment seems to state a slightly higher willingness to participate in an economic experiment than the low payment group, given both of these modes of monetary incentivization. However, the tendency is the same among both groups which clearly state to prefer a low but equal payment over a high but random payment. This is somewhat in conflict with the fact that we did not observe a real difference in recruitment success between the two modes of incentivization for participation in our study.

5. Discussion, recommendations, and concluding remarks

This study has three main findings. First, RCTs are generally not popular among our sample of farmers, and designs that aim to alleviate concerns over exclusion, do not necessarily receive broader acceptance. Second, farmers in our study find some experimental practices of lab-in-the-field experiments much more acceptable than others. Considering farmers’ positive or negative attitudes toward certain features of experimentation may therefore help to facilitate recruitment and trust between researchers and the farming community. Especially concerning, however, is the large skepticism regarding payments that are contingent on behavior, which is a very established and fundamental element of economic experimentation. Third, we did not find evidence of a fixed small incentive vs. a larger stochastic incentive to play a major role for selecting into our study. We discuss these main findings in more detail, and formulate practical recommendations for researchers.

Our findings reinforce that RCTs should be applied with care in the context of agricultural policy since acceptance by farmers seems to be very low. Low RCT acceptance by farmers is generally in line with the only other study on this matter to date by Morawetz and Tribl (2020) who find an acceptance rate for their standard RCT of 22 per cent (compared to approximately 24 per cent in our study). While Morawetz and Tribl (2020) find twice as high acceptance for their alternative unconditional-payment RCT (“upRCT”), our alternative RCT (RCT PAY) is perceived as more problematic than the standard RCT (RCT EXCLU). In part, this may be explained by the low overall acceptance. We also find small and statistically insignificant differences between the LUCK and UNLUCK groups showing that support is higher among farmers who are not part of the experiment. Surprisingly, being excluded from a payment is viewed as more acceptable than having high and low payment levels.

The two studies are not fully comparable. Morawetz and Tribl (2020) use a thought experiment based on a real agri-environmental scheme and survey only farmers that are participating in this specific scheme. This also means that these farmers may have had a good idea of the attractiveness of participating in that specific scheme. In contrast, we use a thought experiment with a more generic description of agri-environmental schemes. As a result, the costs and benefits of eventual enrollment in the program are uncertain to our respondents. Whether the exclusion scenario (RCT EXCLU) effectively prevents some farmers from gaining potential benefits—possibly leading to fairness concerns—is therefore uncertain as well. To the contrary, our low- vs. high-payment scheme (RCT PAY) allows all farmers to reap potential benefits from engaging in the scheme. Given that some farmers would receive a low and others a high payment, to compensate for the same (or similar) costs of engaging in the schemes can be interpreted as introducing a certain unfairness, possibly explaining the lower acceptance.

The study of Morawetz and Tribl (2020) relies on a sample of Austrian farmers; our sample consists of Swedish farmers. While both countries operate under the same overall CAP framework, cultural differences influencing farmers’ attitudes likely exist. A possible follow-up to this study would be a cross-cultural survey experiment with farmers investigating country differences in acceptance and (economic and non-economic) motivators to participate in research tasks (cf. Medvedev et al. 2024).

Our findings call for caution regarding typical experimental practices used in lab-in-the field experiments. Performance-based payments are an established practice in experimental economics to motivate careful decision-making, and the interdependence of payments of participants is a core principle of game theory experiments. However, we find a low acceptance of these features, raising concerns about the usefulness of such payments as a motivator for participation with farmers. In contrast, the stated willingness to participate was higher if all participants received the same small payment. In other words, there is a trade-off between doing what farmers find more acceptable and what the scientific community in experimental economics deems essential for testing certain theories and obtaining valid results. This does not imply that one should give up on these practices of course, but being aware of these tradeoffs may be helpful. Possibly, more careful explanations of the reasons for the use of performance-based payment schemes may increase acceptance. At the same time, researchers may want to keep on looking out for alternative payment schemes. For example, a recent study by Ahles et al. (2024) shows that even very low payment probabilities (1 per cent, 10 per cent) in online experiments are effective in eliciting valuations that are statistically indistinguishable from fully incentivized schemes. Along these lines, combinations of a small equal payment for all participants with a very low probability performance-based bonus could ensure incentive-compatible participant behavior with possibly higher acceptance among farmers.

Our results point to the important role of the general process of research design, implementation, and dissemination. Our sample of farmers reports a higher willingness to participate in economic experiments if the study is co-designed with farmers, and also sharing results with participants afterward increases acceptance. Somewhat surprisingly, using the results for policy-making reduces willingness to participate. While this is something researchers should definitely not give up on, researchers may want to emphasize how research findings may benefit farmers, in the form of better-adapted and more effective (or avoiding less effective) programs, rather than framing them as tests of another set of new policies. Designing and communicating carefully throughout the different stages of the research process, may counter-balance some of these concerns (Höhler et al. 2023).

We found that farmers prefer equal payments over random but higher payments for participation in an experiment both for RCTs and lab-in-the-field experiments. Paying a selected few or paying all participants in a study may often not make a difference in terms of results (e.g. Charness et al. 2016; Ahles et al. 2024) and paying only a subset of participants is therefore an attractive way to reduce transaction costs or to increase sample size within a limited budget. Our findings point out, however, that this may result in acceptance issues, potentially leading to biases or recruitment challenges. Paying more equally—should the research design allow so—may thus be an important option to consider, keeping in mind that high payments may also be perceived as fraudulent (Weigel et al. 2021). Nevertheless, in our own study, different payment modalities did not lead to differences in recruitment success. While our survey suggests that farmers prefer equal payments over random but higher payments for participation in an experiment, recruitment was very similar for both parts of our sample, half of them having been offered a low but equal payment and the other half having been offered a one in 10 per cent chance of receiving a higher payment. Both groups also showed relatively similar preferences regarding these statements. This finding also points toward a limitation. Stated attitudes do not necessarily translate directly into behaviors.

Summing up, our findings suggest actionable insights for the design and implementation of RCTs and lab-in-the-field experiments. First, the notably low overall acceptance rate of RCTs highlights the importance of addressing participant concerns when introducing such methods. To improve acceptance, policymakers and researchers may prioritize transparency and collaboration with stakeholders. For example, involving farmers in the design phase of experiments, as evidenced by the high stated likeliness to participate in the survey experiment when the study was “designed in collaboration with farmers,” can foster trust and a sense of ownership also for RCTs. Second, we find a strong preference for equal and fixed payments over random or behavior-contingent payment structures in lab-in-the-field experiments. While randomized or performance-based incentives are common in experimental economics, their acceptability remains a barrier to recruitment and participation. Researchers could consider adopting hybrid incentive models, where a baseline fixed payment ensures equity, supplemented by smaller additional performance-based rewards. Future research and pilot programs could explore different strategies to identify the most effective combinations of incentives for enhancing participation and maintaining the integrity of experimental designs.

A key limitation is that farmers self-selected into our study. This is a concern in any survey, but even more so in our survey, as we ask about the acceptance of research methods. Ultimately, this limitation can only be overcome by surveying the whole population, for instance as part of a farm census. People who are generally skeptical of research or generally unwilling to participate in research activities, are likely to be under-represented in our sample. In addition, respondents with an interest into voluntary CAP measures may have selected into our survey. Hence, the low acceptance of RCTs and experiments may plausibly represent an upper bound of the acceptance in the larger farming population. Cognitive factors or social norms may also be important drivers of selecting into our study. Unfortunately, there is no population level information on these factors, limiting the possibilities to assess of how biased our sample may be. Our study should be complemented with additional qualitative insights on why farmers show reactance to experimental methods.

Finally, while the low acceptance of RCTs and the high reluctance toward standard experimental practices may seem disappointing to researchers, our results should not be interpreted as a discouragement of the use of experimental methods in Agricultural Economics. Instead, we are convinced that careful consideration in the design of experiments and an awareness of the tradeoffs between different methods, scientific rigor, practical implementation, and the attitude of those participating can result in better and more representative experiments. This requires involving farmers to understand their perceptions and concerns—which itself will increase acceptance and recruitment success—and the development of new, innovative RCT designs and incentive structures that reconcile the concerns of both experimental subjects and researchers. Ultimately, we should rely on a complementary combination of experiments, quasi-experimental methods (Wuepper and Finger 2023; Henningsen et al. 2024), simulation models, and qualitative methods to evaluate policies.

Acknowledgments

The study did not require ethical approval under Swedish regulation. Informed consent was obtained from all participants. No deception was used.

Conflict of interest

None declared.

Data availability

Data and code are available with the submission.

Endnotes

Appendix

References