Heuristic and Systematic Use of Search Engines

Abstract

It is becoming increasingly important to understand in greater detail how people process information and make decisions while searching on the World Wide Web. With the distinction between systematic and heuristic processing, dual-process theories and decision theory provide a useful framework for analyzing this decision-making. A laboratory quasi-experiment was conducted, combining a client-oriented Web content analysis, think aloud technique, and an online questionnaire. From the data obtained, two different search sequence levels were created and analyzed. The results show that within these sequences, different degrees of heuristic and systematic processing occurred, depending on the situational demands as well as the Web experience and the domain specific involvement of the user.

Introduction

Search engines have become a major tool for reaching a growing number of social, political, economic, and cultural domains represented on the Web. The question thus arises how users best apply these tools. Although a large proportion of the work of search engines may be determined by search algorithms, the decision to phrase a certain query and activate a specific link depends on the user. Considering the overwhelming amount of information available through most search queries, the question of how users manage to orient themselves and retrieve relevant information with these tools is becoming increasingly important (e.g., Hargittai, 2000; Pollock & Hockley, 1997).

Information processing and decision theory suggest that in complex decision situations such as information retrieval on the Web, people tend to favor simplified decision strategies. That means that they do not process all available information systematically, but rather heuristically, in order to cope with the amount of information. However, this can lead to inefficient or even useless results. The focus of the present study is directed to this problem by looking at people’s decision-making with respect to search engine use. More specifically, this study explores whether people tend to process information heuristically or systematically while using a search engine and what role user and situational characteristics play in the decision-making process.¹

Theoretical Background

From the early history of decision-making research, it has been argued that humans do not always use their cognitive abilities extensively before they make a decision or execute an action. According to the idea of “bounded rationality” (Simon, 1955), humans do not consider all possible outcomes before deciding, due to lack of time as well as cognitive constraints. Rather, they operate rationally within given boundaries. Decision research also deals with the processing of information (Payne & Bettman, 2004). For a fully reasoned decision, the application of cognitive effort is necessary. When considering a given problem with several multi-attribute alternatives, a highly elaborated strategy or the complete consideration of all alternatives and their attributes is needed. This should provide an optimal outcome while requiring the most cognitive effort. Perhaps surprisingly, this strategy may not be superior to other strategies (Gigerenzer, 2004). Thus, it is reasonable to use simple and faster strategies for decision-making. These take less cognitive effort and, in most cases, are sufficiently efficient. Simon (1955) coined the expression a “satisficing” decision, meaning that humans are often satisfied with a less-than-optimal outcome as long as it suffices for their purposes (Newell & Simon, 1972, p. 681).

Moreover, limited cognitive capacity during information processing (Lang, 2000) restricts the strategic choice to more simplified strategies. This happens especially when there is high uncertainty due to incomplete information. These simplifications may be based on biased perceptions and could lead to suboptimal outcomes. In trying to find an optimal trade-off between cognitive effort and efficient results, individuals act as adaptive decision makers (Payne, Bettman, & Johnson, 1993), who select the most suitable strategy or heuristic from their available repertoire.

Striking parallels can be found in information processing theories in social cognition research, especially in dual-process theories (Chaiken & Trope, 1999), which assume that the human brain generally provides two opposed ideal modes of information processing. The use of these modes is dependent on several external and internal factors. The most prominent representatives of these theories, the Elaboration Likelihood Model (ELM) and the Heuristic-Systematic Model (HSM) (Chen & Chaiken, 1999; Petty & Cacioppo, 1990), describe these processing modes as a central vs. peripheral route and systematic vs. heuristic processing, respectively. Both descriptions refer to the crucial point of cognitive effort that underlies the modes by differentiating cognitively elaborated, conscious, and reflected “systematic” information processing on the one hand, and non-elaborated, intuitive, and spontaneous “heuristic” processing on the other.

Processing in a systematic way would imply a thorough, in-depth, complete, and well-advised processing of all given information. Conversely, processing information in a heuristic way involves relying on cues that signal truth, quality, or validity of that information. For example, it would be a heuristic strategy to choose the top link of a search engine result page (SERP), because many people believe that the top links are often the best links. These two processing modes are seen as ideal types; in reality, the manner of information processing can range between these two poles on a continuum of more or less systematic processing.

The underlying assumptions of these theories relate to the principles of maximizing the outcome while minimizing cognitive effort. As systematic processing requires more cognitive effort, it is more likely that the individual will choose a trade-off between cognitive effort and optimal outcome, resulting in a more heuristic, but (in regards to the outcome) efficient manner of information processing (Chaiken, Liberman, & Eagly, 1989; Gigerenzer, 2004).

The mode of processing selected depends on internal (individual) and external factors. Internal factors consist of the individual’s cognitive abilities (e.g., prior knowledge) and of his or her motivation to invest time and cognitive effort. Among the external factors, the complexity of information, as well as other information-inherent and situational characteristics, can be mentioned (Chaiken et al., 1989; Petty & Cacioppo, 1990).

Although rooted originally in the context of persuasion research in social psychology, the dual-process theories have been referred to by several authors as relevant to information processing and decision-making (Chaiken et al., 1989, p. 235). Based on the principles of minimizing cognitive effort and maximizing the outcome, influencing factors are situational characteristics of the decision situation itself (the number of decision alternatives), characteristics of the individual (cognitive abilities, prior knowledge, experience, motivation), and characteristics of the broader social context (accountability, risk of false decisions) (cp. Payne et al., 1993).² While amount of prior knowledge determines generally available strategies, experience determines how often a specific strategy is used. Motivation influences the use of more reflected strategies (Omodei & Wearing, 1995).

Taken together, both information processing theories and decision research suggest that the application of heuristic, simple strategies in decision-making are widespread and in many cases, sufficiently efficient.

Information-Seeking Behavior on the World Wide Web

For over a decade, numerous researchers have been interested in Web information seeking and Web search behavior (for reviews, see Cothey, 2002; Eveland & Dunwoody, 2001; Jansen & Spink, 2006; Pharo & Järvelin, 2004). This previous research has produced valuable insights into the way users make their navigational decisions. Web information searching often seems rather unsystematic; users seem to behave irrationally and appear to neglect many relevant factors when performing their actions (e.g., Agosto, 2002; Bilal, 2000; Choo, Detlor, & Turnbull, 2000; Cothey, 2002; Pharo & Järvelin, 2004). Employing a qualitative methodology, Agosto (2002) found that users are subject to time constraints and cognitive restrictions. In a longitudinal study, Cothey (2002) demonstrated that users adopt a more passive approach to Web information searching and become more eclectic in their selection actions over time.

The question hinges on how this behavior can be theoretically substantiated. Following the logic of decision psychology, the selection situation can be described as follows (Wirth & Schweiger, 1999): The activity required to sustain the reception process causes high pressure to select between several alternatives, mostly links. However, information about the destination of a link is usually quite low, as there are often no or minimal content descriptions. For example, link descriptions on a SERP often contain little indication of what kind of information is being offered (Borges, Morales & Rodriguez, 1998), resulting in uncertainty and high subjective task complexity (Vakkari, 1999). On the one hand, the combination of the highly decentralized structure of the Web, the often long decision paths, and the large number of available alternatives decreases transparency and increases uncertainty. On the other hand, the individual can determine the way and pace of information processing, giving him or her greater control over the situation. It does not require much effort to revise a wrong decision; the user only needs to click the “back” button.

This underscores the negligible risk and apparent low cost of selection actions in general, as using the Web usually requires little physical effort. Nevertheless, according to decision and information processing theories, there should be variance in cognitive effort, depending on situational characteristics on a macro level (the task type and complexity) and a micro level (number of alternative links and their characteristics, complexity and structure of a website, etc.).

Empirical findings seem to suggest that more systematic processing occurs if the complexity of the decision situation increases. However, according to the economy-minded individual and his or her limited cognitive capacity, it seems very likely that the user will process the information in a rather heuristic manner (Wirth & Schweiger, 1999). The same ambiguous picture emerges with regard to the influence of user characteristics. On the one hand, it seems likely that individuals with high domain specific knowledge, Web experience, and motivation will make more systematic and elaborate decisions (Eveland & Dunwoody, 1998; Hölscher, 2002). On the other hand, the findings also suggest that users with a great deal of Web experience mix systematic and heuristic decision-making routines flexibly. Further, the better the situation is known to them, the more they use heuristic processing, and the more they act in an adaptive fashion. Here, further research would be helpful in providing insights into the circumstances of the different modes of search behavior.

To that end, an empirical approach is appropriate that first considers the situational and user characteristics as precisely and differentiatedly as possible and then as extensively and holistically as possible. Holistic approaches (1) apply process-tracing methods, (2) integrate objective (behavioral), subjective (e.g., cognitive), and situational (e.g., link descriptions) characteristics, and (3) analyze these data on a micro level (single selection actions) and on a macro level (e.g., aggregate actions on a result page). (For other holistic approaches, see Bilal, 2000; Bilal & Kriby, 2002; Hargittai, 2004a; Wirth & Brecht, 1999.)

Research Questions

The theoretical considerations discussed above lead to the following research questions:

RQ1: Can indicators be found that point out different degrees of cognitive elaboration of decision-making during a Web search?

RQ1.1: What is the role of situational characteristics, i.e., task characteristics and characteristics of search engine pages, in Web search decision-making?

RQ1.2: What is the role of user characteristics, i.e., psychological and sociodemographic characteristics, in Web search decision-making?

RQ2: Which selection action types are related to more or less cognitive elaboration in Web search decision-making?

RQ3: Do differences in the cognitive elaboration of decision-making during the Web search result in a different search performance?

Method

Using a holistic quantitative approach, this study analyzes the Web search processes of participants by conducting a quasi-experiment that combines a client-oriented Web content analysis, a think-aloud protocol, and an online questionnaire.³ The participants had to conduct three search tasks with different levels of complexity (low, intermediate, and high complexity). At the beginning of each session, and directly after the completion of each search task, the participants answered questions about their Web use, their surfing habits, and their assessment of the search task they had just completed. The experiments were held in single sessions.

We classified task complexity based on a-priori evaluations. Using four default search engines, two Web-experienced coders systematically explored all search tasks. The search tasks’ complexity was ranked according to the availability of task-relevant information on the Web, as well as the dispersion of this information on different websites (Bilal, 2002; Byström & Järvelin, 1995; Vakkari, 1999).

Alternately, the participants received an open-ended task and a closed task (Bilal, 2002; Marchionini, 1989, 1995). Presumably, more complex tasks should raise the cognitive effort (Chen & Rada, 1996). Furthermore, the amount of time at the participants’ disposal varied. For easy tasks they were given three minutes, for intermediate tasks five to ten minutes, and for highly complex tasks, five to eight minutes (see Table 1).

The three search tasks could be processed using Google, Yahoo, Web.de, or Lycos.⁴ Individuals were randomly assigned to one of the search engines.

Following the holistic approach in search behavior research, all of the participants’ selections and navigations were captured by screen recording. Insights into the decision-making processes during the search were collected by applying a think-aloud protocol (Eveland & Dunwoody, 2000). Thinking aloud prevents rationalizations and reveals momentary thoughts and situational influences (Ericsson & Simon, 1993). In order to shed light on the mental decision-making processes during navigation, verbal and paraverbal behavior and facial expressions were recorded using a microphone and video camera. Both direct (i.e., verbal) and more indirect indicators of the decision-making processes, i.e., paraverbal behavior and facial expressions, allowed us to examine the decision-making and cognitive processing of the participants thoroughly.

The verbal and video recordings and the screen recordings were digitally synchronized and coded using a quantitative coding scheme (cp. Hargittai, 2004a; Wirth & Brecht, 1999). The basic coding unit was a single selection action, defined as any action resulting in at least partially new information, which in turn was the informational basis for the next selection decision. Therefore, horizontal and vertical scrolling was also included in the coding system. Every selection action was coded with its direct attributes (type of action), including micro-situational characteristics (e.g., Web page complexity, link characteristics, mouse moves), verbal and paraverbal responses and facial expressions accompanying the selection action. This coding allowed the authors to analyze single steps of the search process, as well as several sequences of action, by aggregating the data on higher levels. Thus, the demand of a holistic approach was accommodated by incorporating all selection decisions and merging them into coherent sequences. All in all, a total of 15,476 selection actions were coded.⁵

Measures

In order to answer research question 1, the mental decision-making processes during navigation were examined in further detail. For that purpose, indicators were collected from the coded verbal and paraverbal data, as well as from the facial expressions. Four non-overlapping indicators for the degree of cognitive elaboration were computed:

1. The degree of evaluation(including the participants’ verbal and paraverbal evaluations, e.g., comments on information quality or facial expressions of disappointment).

2. The degree of reflection(consisting of verbal and paraverbal comments about the selection destination and orientation, justifications, verifications, astonishment, and comments about one’s own deliberation).

3. Expectancy violation(including verbal and paraverbal comments on met versus unmet expectations).

4. Indecisiveness(consisting of hesitating mouse actions, corresponding paraverbal comments, and actions that were started but not finished).

Additionally, the time spent on a selection unit was assessed as a more indirect indicator for focusing on the decision task.

In order to answer research question 1.1, situational characteristics (absolute number of SERP, absolute number of results, information quality of the result link description) and task characteristics (task type, subjective task complexity—6 items, Cronbach’s α =.791) were taken into account. In order to answer research question 1.2, the influence of domain specific involvement, the processing involvement(5 items, Cronbach’s α =.721), the Web experience(8 items, Cronbach’s α =.852), and sociodemographic data were analyzed. Research question 2 deals with the different types of selection actions(e.g., scrolling), which are related to different styles of cognitive processing. Finally, as an indicator of search performance (RQ3), a relative information gain measurement (measuring the increase of task relevant information during the search process) was developed and analyzed.

Data Analysis

To fulfill the holistic approach, we incorporated not only all selection decisions in our analysis, but also merged them into coherent sequences. First, the search sequence was aggregated. This contained all actions that took place in the context of a specific search query. It began with the introduction of a search query and ended when a new search query was entered. All actions between these two search queries were included.

Within the search sequence we aggregated two further sequences on which our analyses are based: the result page sequence(RPSEQ) and the result evaluation sequence(RESEQ).

The RPSEQ contains all actions that take place on a specific SERP. It begins with the first action on a new SERP and ends when the SERP is abandoned, for example, via a result link or because a new SERP is opened. All actions on the SERP itself (scrolling or jumping on the page) are part of the actual RPSEQ. All interruptions—for example, because of an interim result evaluation—are excluded.

The RESEQ consists of all actions that serve to explore and evaluate a specific result link on a SERP. The sequence begins with the activation of a result link and contains all actions on the subsequently entered Web pages. All interruptions, including those due to an interim visit of a SERP, are excluded. The RESEQ has the sole potential for information gain and for knowledge acquisition because only these pages contain content.

While conducting a search, users usually have to reformulate the first search query again and again to be (more or less) successful, ultimately. This means that they execute several search sequences in a row, each consisting of result page sequences(RPSEQ) and (in the majority of cases) result evaluation sequences(RESEQ). To get a condensed picture of these navigation activities, including the cognitive processes that accompany them and the resulting search performances, a cluster analysis of both types of sequences was conducted. Cluster analysis, a statistical analysis technique that generates a descriptive overview of different search patterns, has been widely applied in studies in the field of Web navigation research (e.g., Chen & Cooper, 2001; Lawless & Kulikowich, 1998).

As we are interested in patterns of heuristic (less effortful) versus systematic (more effortful) search engine use, we clustered only variables indicating cognitive effort: the degree of evaluation, the degree of reflection, expectancy violation, indecisiveness, and the time spent on a selection unit.⁶ To provide further information about the processing types and to answer the research questions, correlations and contingencies between the resulting clusters, the situational characteristics involved (absolute position of SERP, absolute number of hits and the task type and complexity), and individual users’ characteristics (processing involvement, domain specific involvement, Web experience, familiarity to search engine, sex, age, education) are analyzed. Finally, the selection activity is taken into account by considering different types of selection actions (scrolling, “back” button, etc.).

Participants

Participants for this study (N = 128) were recruited through announcements in two local papers and by flyers distributed in the residential quarter neighboring our institute. All participants received an incentive of 15 €. Ages ranged from 13 to 69 years (Ø 28.6 years), with 50% male and 50% female. One-third of the participants were adolescents (ages 13-18). Their highest educational level was distributed equally into low (middle school), middle (high school diploma), and high (university degree) degrees. Thus, the diversity of participants exceeded that found in many similar studies.

Limitations

To avoid artifacts due to unforeseen task characteristics, each participant was assigned three out of eight tasks (Table 1)—thus limiting the possibility of comparing participants’ performance. However, a certain comparability between tasks was guaranteed by classifying them a priori according to task type (closed or open-ended) and complexity (easy, intermediate, high) as described in the Method section.

One limitation of the study is the over-representation of young users with higher education and greater search engine experience.

Results

Overall Task Performance and Search Behavior

Overall, users attained an information gain of 27% in the course of one task. The average search sequence took 99 seconds, in which users consulted 1.4 result pages and visited 2.2 result links, realizing 11 selection actions per search sequence. Search sequences took less time, fewer selection actions were realized and fewer result links were visited for closed tasks than for open-ended tasks; the number of SERPs was about equal for both task types (Table 2). As expected, task complexity is reflected partially in overall search success; however, it does not seem to influence the length of search sequences or the number of result pages and result links visited (Table 3).

Cognitive Processing Within the Result Page Sequences (RPSEQ)

The cognitive processing within the result page sequences (N = 1594) can be differentiated into four clusters (Table 4). Concerning situational factors (RQ1.1), significant differences between the clusters were found regarding the absolute position of the search engine result pages on which the RPSEQ was realized, the absolute number of result links found on that page, and the task complexity, whereas the task type did not vary significantly. User characteristics (RQ1.2) such as Web experience, familiarity with the used search engine, and age differed significantly, leaving domain-specific and processing involvement, sex, and education as insignificant factors (Table 5). The following description of the clusters focuses on the significant factors. Moreover, to provide a condensed picture of the specific characteristics of each cluster, within the significant factors only those with csr-values higher than +2.0 and lower than -2.0, respectively, are included in the cluster descriptions.⁷

The first and largest cluster is characterized by the lowest number of reflective comments and the lowest evaluation rate of all clusters (Table 6). Participants acted very decisively and showed little surprise over the search engine results. This is reflected by the short time they spent on a SERP (23 sec.) and the low number of selection actions they completed during one RPSEQ (Table 6). At the same time, participants entered search queries here more often than in the other clusters (27.0%; Table 7). Apparently, they scanned the search engine results only briefly and in a rather superficial way and did not use much cognitive effort in processing them. In many cases, RPSEQs of this cluster concern not the first but subsequent SERPs (csr = + 4.2; Table 5).

Generally speaking, the cognitive effort invested seems to diminish when participants check subsequent SERPs. This mode of processing indicates a heuristic processing that relies on the experience of resulting links on subsequent SERPs often not matching the search query. Therefore, they may be skimmed over quickly in a cursory manner, permitting participants to act more decisively. This behavior does not stand for an absolutely non-elaborated decision-making, but rather, points to a cognitive “shortcut” requiring less effort, which, however, runs the risk of overlooking useful result links. To sum up, SERPs of this cluster can be denoted as “subsequent result pages with quick and heuristic processing.” Younger participants between 19 and 35 years (csr = +4.9) and experienced Web users especially showed this kind of heuristic processing (csr = +4.4) (Table 5).

The second cluster is the smallest. It is characterized by high cognitive reflection and rather high evaluative, mostly negative, comments (Table 6). The cluster has the shortest RPSEQs, with an average of only two selection actions (Table 7) and a length of 25 seconds. Most strikingly, the SERPs fall very short of the participants’ expectations, while indecisiveness is at an average level (Table 6). The expectation discrepancies seem to be so obvious that users do not feel insecure about the next step (in many cases, using the “back” button (12.3%) or entering a new search query (21.5%); Table 7). Within these RPSEQs, there was not a single result link evaluation that led to an information gain (Table 6). This large incidence of reflection is understandable when considering that this cluster mainly concerns the first SERP after a search query (csr = +2.2; Table 5). In this case, a learned heuristic seems to trigger further cognitive processing. The first SERP presumably provides useful result links and is therefore explored very carefully.

However, the search engine results in this cluster were appraised as unexpectedly inappropriate, and frequently, the search engine failed to present any result links at all (no hit: csr = +5.1). Participants reacted strongly to this unexpected outcome of their search and quickly decided to give up their search. Therefore, we call the RPSEQs in this cluster “unexpected dead end result pages.” This interpretation conforms to the fact that RPSEQs of this cluster are likely to be found in highly complex tasks (csr = +2.2).

Within the third cluster, the high rate of indecisiveness stands out, while evaluations and reflective comments occur at a fairly average rate (Table 6). Participants spent 29 seconds navigating on a single SERP, which is an intermediate length of time compared to the other clusters. The action types also occur on an average level (e.g., scrolling, activating the “back” button). RPSEQs of this type often take place on the second SERP of the search query (p <.01). The participants presumably wonder whether they should explore additional SERPs or enter a new search query.

Since no other situational variable is able to clarify the high rate of indecisiveness, it can be assumed that individual characteristics contribute to that type of decision-making. Indeed, the youngest participants (< 18 years; csr = +2.0) and those unfamiliar with the assigned search engine (csr = +3.4) are over-represented in this cluster (Table 5). These groups presumably did not possess strategies for what to do in situations where the most obvious action (exploring the first SERP) is already taken and has not resulted in a satisfying information gain. In short, the RPSEQs in this cluster can be denoted as “result pages with hampered decision-making.”

The main characteristics of the fourth cluster are numerous evaluative comments and an average degree of reflection (Table 6). The participants were only slightly indecisive and expressed few expectancy violations. With a mean duration of 59 seconds, these RPSEQs lasted twice as long as the other RPSEQs and consisted of six selection actions (Table 7). Thus, an average selection action lasted 10 seconds, indicating rather slow decision-making. The information that could be additionally gained within these RPSEQs was remarkably high (Table 7). However, this additional information is found only by scrolling and not through the activation of result links (Table 7). Additional analysis revealed that this type of RPSEQ often corresponds to the first SERP after entering a search query (p <.01). This cluster is typical, not only for youngsters, but for adults older than 35 years (csr = +5.3), for less experienced Web users (csr = +5.0), and for people with low domain-specific involvement (csr = +2.1) (Table 5). Moreover, tasks of intermediate complexity are slightly overrepresented (csr = +2.9).

Overall, the following picture emerges: It is mostly (but not exclusively) inexperienced or less-involved users who thoroughly evaluate the first SERP and thereby attain remarkable information gain (6.5%; Table 7)—but only if the tasks are not too complex. RPSEQs in this cluster can be characterized as “result pages with high and successful cognitive effort.” Consequently, this cluster corresponds to the second cluster, in which the evaluation of the first SERP resulted in an unexpectedly poor search result. In contrast to the third RPSEQ cluster, a learned strategy seems to be available to trigger the evaluative processing mode. Participants over 35 years old and those with less Web experience frequently used the heuristic cue and relied on the relative position of the SERP; the heuristic suggests evaluating the first result page more elaborately.

Cognitive Processing Within the Result Evaluation Sequences (RESEQ)

The analysis of aggregated RESEQ data (N = 1705) resulted in three significantly different clusters of cognitive processing (Table 4). Among situational characteristics (RQ1.1), the only one differing significantly across clusters is task complexity, whereas the absolute position of the referring SERP, the absolute number of result links, and the task type did not show significant differences. User characteristics (RQ1.2), such as domain specific involvement, Web experience, education, and age, differed significantly, while processing involvement, familiarity with the search engine used, and sex did not differ (Table 10). Within the significant factors only those with csr-values higher than +2.0 and lower than -2.0, respectively, are included in the cluster descriptions.

The first cluster is the smallest, but contains the longest RESEQ. The participants spent on average more than two minutes (M = 139 sec.) and completed 14 single selection actions (Table 9) to explore and evaluate a result link. These RESEQs are characterized by a high rate of indecisiveness correlating with a high level of astonishment over the content of the Web pages and a moderately reflected, less evaluative process (Table 8). RESEQs in this cluster appear to be especially successful. The RESEQs result in high information gain (Table 9) with very few discontinuing actions observed, like activating the “back” button (9.1%) or closing the browser’s window (3.1%). Scrolling is by far the most frequent action and occurs more often in the first cluster than in any of the other clusters (57.0%; Table 9).

Overall, the result link activation seems to be part of a heuristically motivated trial-and-error strategy used on the first few SERPs (p<.01). The high astonishment within the RESEQs found among this cluster, as well as a largely positive evaluation, can be interpreted as a pleasant surprise, as the trial-and-error strategy was successful. This fits with the fact that this RESEQ type generally appears more often among novices (csr = +3.2) and older participants (=35 years; csr = +2.6) (Table 10). Hence, these RESEQs can be described as a typical trial-and-error navigation, consisting of less-reflected “trying,” or exploring actions mixed with several reflected and evaluated selection decisions. Therefore, it will be denoted as “successful trial-and-error result link exploration.”

The second cluster is characterized by a high degree of expectancy violation and reflection. The relative degree of evaluation is the highest, indicating that participants more critically evaluated these content pages versus other Web pages, like the SERPs. The participants showed a moderate extent of indecisiveness (Table 8). On average, these RESEQs lasted only 34 seconds and consisted of about four selection actions (Table 9). Contrary to the first cluster, the RESEQs here resulted in a very low information gain (Table 9). Because of the unexpectedly low success of the result evaluations, the backward navigation action was encountered rather frequently (Table 9). The probability of such RESEQs was higher with complex tasks than with easier ones (csr = +2.7; Table 10).

These RESEQs seem to reflect a very systematic processing that is presumably assisted by the situational context. The high expectancy violations index suggests that the explored contents are easily classified as improper. In the literature, this is assumed to be the case in closed tasks rather than in open-ended tasks (Marchionini, 1989; Navarro-Prieto, Scaife & Rogers, 1999). These assumptions are confirmed by the present study (closed task: csr = +2.1; Table 10). The participants reflected on the contents, saw that they did not satisfy their expectations, and turned back to explore other result links. It is remarkable that experienced users (csr = +2.6), younger ones (19-35 years; csr = +2.6), and participants with high domain-specific involvement (csr = +2.4) are overrepresented within this cluster, while users with higher education are strongly underrepresented (csr =-5.3). These participants are possibly used to moving rapidly through the Web, deciding very quickly whether information is adequate. Thus, this cluster could be called “unexpectedly failed result link exploration.”

The third cluster contains by far the most RESEQs and is mainly characterized by complete conformity to expectations (Table 8). Compared to the other two clusters, the actions are accompanied by very few reflective comments. Evaluations are expressed only from a moderate to low extent. The RESEQs last only about half a minute (33 sec.) and consist of four selections (Table 9). The information gained is on a medium level (Table 9).

As additional analysis revealed, the exploration of the RESEQs in this cluster happens rather late. Often, it is the ninth, tenth, or even a later RESEQ (p <.001). Taking into account that easy tasks are over-represented (csr = +3.3) and that the participants typically have high Web experience (csr = +3.4), are involved in the domain (csr = +2.4), and have higher education (csr = +5.5), the following interpretation seems plausible: If motivated and higher skilled, users explore an above-average number of result links. Therefore, according to their Web experience, they navigate rather speedily, applying little reflection and evaluation. They quickly scan the result link page to see if it holds information for which they seek and then move on. This processing mode seems to follow especially the “satisficing” principle; the gain of information is not optimal but is nonetheless sufficiently efficient to carry on the exploration. This type of RESEQ can be denoted as “cursory result link exploration.”

Conclusion

The study confirms that the combination of dual-process theories and decision theory provides a useful framework for describing search behavior on the Web. In response to the first research question, clusters of different cognitive elaboration efforts could be identified through the indicators applied on the levels of both analyzed sequence types (RPSEQ and RESEQ). Among the RPSEQs, there is a tendency toward more heuristic processing. The first and, by far, biggest cluster shows a clear picture of heuristic processing, namely the scanning of subsequent result link pages. The second and third clusters both show some kind of systematic processing, but also indicate heuristic or, worse, no processing at all. The second cluster can be explained by the unexpectedly poor results found on the SERP; concerning the third cluster, the participants did not seem to possess appropriate strategies for the subsequent result lists. In contrast, the fourth cluster, which mainly contains the exploration of the first result pages and is characterized by a high cognitive effort, clearly represents systematic processing. The tendency towards more systematic processing can be explained by the fact that the success of the entire search depends on users’ decisions on the result pages, and thus simple heuristic processing, or no processing at all, means taking a higher risk.

The RESEQs are also dominated by heuristic processing. The third and the first clusters display heuristic processing; the third cluster shows a low reflection and a cursory scanning of the content pages behind the result links, while the first cluster indicates a heuristic trial-and-error-behavior, mixed with reflective comments. In contrast, the second cluster is determined by high cognitive reflection, which occurs if the result link exploration unexpectedly fails. Thus, users apply systematic processing at points crucial to a further search, as already recognized for RPSEQs.

Differences and similarities between the RPSEQs and the RESEQs can also be identified in regards to situational and user characteristics (research questions 1.1 and 1.2). While some of these are only important for one of the sequence types, others are important for both. Thus, it is only among RPSEQs that the situational characteristics absolute position of a search engine result page and absolute number of result links are important factors for the applied level of cognitive elaboration. Generally, users dedicate more cognitive effort to the first SERP, whereas the following SERPs draw less attention because they appear less promising according to the heuristic that subsequent SERPs will contain less relevant information. If later SERPs attract attention, this is often due to insecurity about whether it is better to continue the search or to start a new one. Likewise, higher cognitive attention is only evoked if the search query resulted in zero result links.

Factors influencing cognitive processing within RESEQs are the users’domain specific involvement and their level of education. Here again the difference between the two types of sequences becomes evident. In RPSEQs, background knowledge is not important because result link pages, with their short link descriptions, do not offer enough information. In contrast, result pages contain the information sought and demand a user’s capacity to analyze the content provided.

Factors of high importance for both sequence types are task complexity as a situational characteristic, and Web experience, familiarity with the search engine, and age as user characteristics. Complex tasks seem to demand a higher level of elaboration up to the point where a task is considered unsolvable and the discouraged user prefers not to invest any more attention in the problem. Web experience influences the degree of cognitive processing in two ways. First, experienced Web users generally process information in a more heuristic manner; for example, they scan SERPs and Web contents rather quickly. This can presumably be traced back to a larger strategy set with very efficient strategies (Gigerenzer, 2004; Navarro-Prieto et al., 1999). Among experienced users, cognitively more elaborate processing occurs only if they encounter unexpected content. However, in these situations cognitive processing is more related to the content than to the question of how to proceed with the search. Inexperienced users, on the other hand, tend to reflect more on their decisions and spend more time and cognitive effort exploring result links. The users’ age seems to be directly connected to the Web experience as users older than 35 years always appear in the same clusters as inexperienced ones. A low familiarity with the search engine seems to influence the cognitive processing indirectly by hampering the search process itself.

In response to research question 2, different types of selection action corresponded to different types of cognitive elaboration. These differences depend on the situational characteristics of the search sequence in which these selection actions are situated, as well as personal characteristics of the user. Thus, activating the “back” button is a typical action in a trial-and-error heuristic, but it may also be the result of a thorough evaluation of the information found on one link or SERP.

Concerning research question 3, the interrelation between search performance and the type of cognitive elaboration applied is rather complex. Neither heuristic nor systematic search can be identified as a generally better performing approach. However, a combination of heuristic and systematic processing in the course of one search sequence seems to be the ideal solution, as, for example, in clicking directly onto the first result link on the first result page and then analyzing that page very carefully. While experienced users achieve suboptimal results by processing in a more heuristic way, inexperienced users may achieve similar information gain by processing in a more systematic way. Thus, to a certain extent, inexperienced users seem to compensate for their lack of Web experience with more elaborate processing. However, if something unexpected occurs, they seem unable to cope with the situation.

Overall, the findings support the theoretical assumption of the adaptive, cost-effective user applying a cognitive processing mode using less effort—as long as this is sufficiently efficient and does not produce too high of a risk (Payne et al., 1993). However, it seems that users process the information in a more effortful manner if the risk of a wrong decision is high (for example, abandoning a search section which might still bring the searched information) or something unexpected occurs, such as finding no results after entering a promising search query. While users with Web experience can switch between heuristic and systematic processing, inexperienced users can only act adaptively to the extent that they can fall back on appropriate coping strategies, such as clicking on the first link offered on a SERP. If they do not apply such strategies, they need to mobilize a higher cognitive effort, but as our results show, this works only as long as the situational demands—notably the search task—are not too complex. Again, this supports the assumption of decision theory (Payne & Bettman, 2004). The findings also reveal the limits of direct adaptive behavior. Sooner or later, even experienced users are confronted with dead ends. In this case, only a meta-strategy can help: to quit the present search and start again.

Overall, this study provides insight into the cognitive processes that may be the basis for many different search engines’ uses. Future research may build on this foundation to proceed in a more prognostic direction by asking, for example, whether users handle decisions differently according to their search motives.

Notes

References

About the Authors

Werner Wirth (Ph.D., University of Munich) is a full professor of communication and empirical media research at the Institute of Mass Communication and Media Research, University of Zurich. His research focuses on diffusion, adoption, and use of new media, as well as cognitive, emotional, and persuasive aspects of media use, media reception, and media effects

Address: Institut für Publizistikwissenschaft und Medienforschung (IPMZ), Universität Zürich, Andreasstr. 15, CH–8050 Zürich, Switzerland

Tabea Böcking is a Ph.D. candidate in mass communication at the Institute of Mass Communication and Media Research (IfKW), University of Munich. Her research interests include user behavior in new media environments, media framing and issue developing, and empirical methods.

Address: Institut für Kommunikationswissenschaft und Medienforschung (IfKW), Ludwig-Maximilians-Universität, Oettingenstr. 67, D-80538 München, Germany

Veronika Karnowski is a Ph.D. candidate in mass communication at the Institute of Mass Communication and Media Research (IfKW), University of Munich. Her research interests include online and mobile communication, diffusion and appropriation research, and empirical methods.

Address: Institut für Kommunikationswissenschaft und Medienforschung (IfKW), Ludwig-Maximilians-Universität, Oettingenstr. 67, D-80538 München, Germany

Thilo von Pape is a Ph.D. candidate in mass communication at the Institute of Mass Communication and Media Research (IfKW), University of Munich. His main research interests lie in the fields of diffusion and appropriation of new information and communication technologies (motives, uses, and consequences).

Address: Institut für Kommunikationswissenschaft und Medienforschung (IfKW), Ludwig-Maximilians-Universität, Oettingenstr. 67, D-80538 München, Germany