The Efficacy Potential of Cyber Security Advice as Presented in News Articles

Abstract

Cyber security advice is a broad church: it is thematically expansive, comprising expert texts, user-generated data consumed by individual users via informal learning and much in-between. While there is evidence that cyber security news articles play a role in disseminating cyber security advice, the nature and extent of that role are not clear. We present a corpus of cyber security advice generated from mainstream news articles. The work was driven by two research objectives. The first objective was to ascertain what kind of actionable advice is being disseminated; the second was to explore ways of determining the efficacy potential of news-mediated security advice. The results show an increase in the generation of cyber security news articles, together with increases in vocabulary complexity and reading difficulty. We argue that these could present challenges for vulnerable users. We believe that this corpus and the accompanying analysis have the potential to inform future efforts to quantify and improve the efficacy potential of security advice dissemination.

RESEARCH HIGHLIGHTS

• News-mediated security advice has been sharply increasing since 2018.

• Many cyber security news articles have a high level of specificity.

• Subject-specific terminology within our security news articles is continuously evolving.

• Cyber security news is often of short length and low readability, with a negative impact on efficacy potential.

• The research indicates increasingly diversified interest in goal-specific cyber security advice.

1 Introduction

The usable security field is now nearly 30 years old Theofanos (2020), although the underlying concepts are much older. For example, Auguste Kerckhoffs acknowledged the role of the user in successful security implementation as early as 1883 when he published his seminal piece on military cryptography Kerckhoffs (1883). Since then, users have become active participants in ensuring the security of systems and important mediators of experts’ security advice—which, in the context of this paper, we define as explicit instructions intended to make recipients more secure, once implemented Herley (2009), Pfleeger et al. (2014), Redmiles et al. (2016a).

The Internet is awash with implicit and explicit security advice, disseminated both by experts and by other users. Much of this advice has its limitations. For example, while the underlying threats may be common, the advice provided to deal with these threats can differ significantly, both in terms of wording and implied level of urgency. Explicit security advice that is either too abstract or too vague, or that assumes a high level of security knowledge or expertise, can fail to serve as usable advice, as characterized by Kerckhoffs: ‘the system must be easy to use and must neither require stress of mind nor the knowledge of a long set of rules’ Kerckhoffs (1883).

Unfortunately, because individuals face a wide range of cyber security threats—including viruses Al-Mhiqani et al. (2018), bot-nets Bertino & Islam (2017), port-scanners Viet et al. (2018), spyware Herley et al. (2015), Sowndarajan & Binu (2017), malware Sowndarajan & Binu (2017), stalkerware Khoo et al. (2019) and rootkits Sowndarajan & Binu (2017), Wang et al. (2016)—it is difficult for them to develop sufficient ‘knowledge’ of the ‘rules’ even when advice is presented clearly and thoroughly. Furthermore, there is the question of what individual users do with the advice they receive. They may, for example, reject advice they deem too difficult to implement Herley (2009), Rader et al. (2012) or delay implementing advice because they underestimate the consequences of losing control of their data Brandimarte et al. (2013). These complications are exacerbated by the ongoing centralization of services Barnes (2006) and heavy use of social media platforms Cyber-security in social media (2019), which may inspire complacency through habituation while introducing novel risks. There is evidence that some users register threats on a subconscious level, raising the possibility that they will ignore novel threats. Such so-called security fatigue has been discussed by authors such as Furnell & Thomson (2009).

Despite the significance of these problems, there have been relatively few quantitative studies analysing the security advice literature available to what we might term ‘everyday’ users. Notable examples of such studies include that of Redmiles et al. (2016a), which explores how individual users actualize and perceive their own cyber security capabilities (which often derive from informally learnt advice), and that of Renaud & Dupuis (2019), which identifies large sub-concepts and classifications in usable-security. We endeavor to bridge these contributions by providing additional context about the current security advice environment. Our research is guided by the following research objectives (to which we return in Section 5):

• RO1: What kind of informally learnt and actionable security advice most often appears in news articles?

• RO2: What is the efficacy potential of this security advice as consumed by an individual user?

We describe the process we undertook to assemble a corpus of security advice that reflects the advice presented to individual users daily within their typical informal learning environment. The corpus spans a 24-month period and was assembled from a data-set containing 15,422 (English language) news and online magazine articles from North American (US and Canada) and UK-based sources, as well as some historical data stretching back to 2000. We ascribed broad classifications to specific advice, ascertained the dominant methods of advice construction and dissemination and analysed their potential efficacy potential over time. Our hope is that this corpus will help to lay the foundations for further work on quantifying and improving the efficacy potential of security advice dissemination.

The remainder of this paper is organized as follows. In Section 2 we provide the motivation for, and the background to, our contribution, and define the terms of interest. In Section 3 we describe the process used to create the data-set and our data cleansing process that develops the data-set into the corpus. In Section 4 we present our results and perform an initial analysis, before, in Section 5, discussing the results in relation to our research objectives. Section 6 considers limitations to the study and considers possible future directions. We conclude the paper in Section 8.

2 Background and motivation

In this section we discuss the background to, and the motivation for, the work described in this paper. We start (in Section 2.1) by contextualizing the term security advice. We define what security advice is and how security advice is learnt. We then consider the individual user and attempt to define a persona to represent this user (in the broadest possible sense) with a view to capturing their media and information landscapes.

2.1 What is security advice?

The concept of advice is difficult to define, partly because it pertains to almost every discipline involving human behaviour Bonaccio & Dalal (2006), Ottaviani & Sørensen (2006). Security advice is particularly complicated, as it derives from diverse disciplines, including psychology Dreibelbis et al. (2018), Wiederhold (2014), medicine Yuan et al. (2018) and computer science.¹ Each of these disciplines has its own approaches to security advice, and, as such, it would be foolish to attempt a comprehensive review of this topic covering all of these fields. Rather, we make reference to those contributions that helped to frame, motivate and scope the present study.

In this paper, rather than taking a broad view of security advice, we limit our concerns to what we term professional security advice. Çelen et al. (2010) describe the term professional advice as ‘advice rendered by experts’ (see Section 2.2 for a consideration of experts), which is then disseminated to individual users. Professional security advice is designed to alleviate security challenges, but, because this must often occur through the mediation of individual users, these instructions are characteristically persuasive and typically explicit, rather than implicit.

Explicit advice often takes the form of a verbal or written appeal, such as Keep your browser up to date at all times. There are even different varieties of appeals, as with Keep your browser up to date at all times … or else your browser will get infected, which constitutes a fear appeal  Bada et al. (2019), Lawson et al. (2016). Implicit advice often takes the form of a threat message Renaud & Dupuis (2019), such as Downloading a file from an untrusted source is risky  Fan & Yeung (2011).

Based on this distinction and the definitions prevalent in the literature Das et al. (2018), Herley (2009), Ion et al. (2015), Pfleeger et al. (2014), Rader et al. (2012), Redmiles et al. (2016a, 2016b), we define security advice thus:

A written instruction, provided by a trusted and professional source, with the explicit goal of enabling the recipient to be more secure once they execute the instruction.

2.2 The expert

Note that this definition depends on the intent of the advice, rather than the outcome. This is because professional security advice cannot ensure the success of its security recommendations as these are often developed based on limited observational data Abomhara & Køien (2015). For example, it is difficult to quantify threat-related data, such as the chance or probable extent of a malicious actor attack Cashell et al. (2004), Jang-Jaccard & Nepal (2014), Tregear (2001), Wagner et al. (2019). Similarly, it is difficult to calculate the loss of customer confidence following such threats Smith (2004).

Although the projected costs of hardware and/or software development to alleviate security threats might be well established in particular cases, their associated indirect costs and downstream issues are not Wagner et al. (2019). Despite these uncertainties, professional security advice must appear authoritative, and this gap means that some professional advice may prove ineffective. Furthermore, it is sometimes the case that some security artefacts are outdated or redundant almost immediately due to the ever-improving armoury tools and techniques available to malicious actors Casas et al. (2017), Chen et al. (2012), Wang et al. (2014). It is unfortunately the case that this arms race or game of ‘whack-a-mole’ is the environment within which most security advice is created and disseminated Solms & Niekerk (2013).

Within the context of this paper, we believe it to be important to provide an explicit definition of an expert. First, expertise as a concept can be divided into two distinct categories: expertise as a function of What someone knows and expertise as a function of What someone does. In the former, we are interested primarily in the expert’s epistemic knowledge of a particular domain—in this case, their capacity to provide justifications for any given range of ideas and knowledge Weinstein (1993).

Many organizations and public-facing institutions employ security experts that provide insight and compose security advice based on considerable expertise gained from education and/or industry experience Caldwell (2013). Shortages of such security experts are regularly reported Li et al. (2019), Miller et al. (2016), Park (2012), Šorgo et al. (2017), and this has led to an requirement for additional tools and techniques that can be used to aid current security experts in their work Li et al. (2019), Mindermann (2016). Shires Shires (2020) assessed the difficulty in establishing a firm definition of cyber security experts; an analysis of self-described practices within media highlighted a varied perception of how they operate in terms of acquiring and disseminating information. Frey et al. (2017) reported upon a test in which participants possessing several levels of cyber security knowledge were tested. The self-identified security experts tended to achieve poor scores in the test: ‘they tended to display a strong interest in looking up advanced technological solutions rather than intelligence gathering’ Frey et al. (2017).

Within almost any field in which professional advice is rendered, two forces of market are always at play. The first concerns reputation, in which professionals are interested in how their advice is seen to be well informed by colleagues and recipients Ajzen (1985), Lahlou et al. (2005), Pfleeger & Caputo (2012); the second concerns competition, which, within our context (as our advice is provided by experts working within media settings), relates to how competition between the advice providers can distort information before it reaches the intended recipient. For example, professional advice can often take the shape of a contest, in which the experts are evaluated on the basis of their opinions. (An example of this phenomenon can be found within the financial markets, where the Wall Street Journal Forecasting survey pits analysts and experts against each other and provides rankings; Guan et al. (2018), Malmendier & Shanthikumar (2007).) Both of these kinds of influences have the potential to alter the contents of any advice rendered. Given the indeterminacy, incompleteness or sometimes faultiness of the data used to generate expert advice, alongside an unknown mixture of experts with a particular education and/or occupational background, we consider the security expert who renders our advice to be an individual who creates cyber security knowledge out of a mixture of epistemological and performative expertise. Their backgrounds and motivations are otherwise opaque to us—as they may well be to the reader of the news articles—and this forms a limitation on how we may perceive expertise in this field.

Of course, individual users do not always get their advice from experts Çelen et al. (2010), Rader & Wash (2015), Reeder et al. (2017). Indeed, some may not receive information from professional sources at all Rader et al. (2012), Redmiles et al. (2016a, 2016b, 2019). Such users may rely on information sourced from their local environment, which we might characterize as naïve advice  Schotter (2003). Contrary to this label, naïve advice has certain efficiency-enhancing properties when used in negotiations Steinel et al. (2007), public-good experiments Chaudhuri (2011) and certain types of games Kuang et al. (2007).

Although not the focus of this paper, naïve advice must be taken into account when we examine the recipients of security advice and how they respond to such advice.

2.3 Who are the consumers of security advice?

Based on existing literature Howe et al. (2012), Milne et al. (2009), Nthala & Flechais (2017), we define consumers of security advice as ‘individual users’ who own systems, devices and/or services that maintain internet connectivity. An individual user can be of any gender, age group or professional background. We include corporate users in this definition because their learning habits extend beyond their formal corporate learning environments—that is, they may engage in informal learning outside of work environments Garrick (1998).

A user may design their home environment to facilitate actions like activity planning, online shopping, interpersonal communication or transmission of sensitive information (such as medical data) Nthala & Flechais (2017). Each technology set-up can be extremely unique, akin to a fingerprint, making it difficult to assess the risks and vulnerabilities relevant to a particular space Byrne et al. (2012), Nthala & Flechais (2017).

Given this variation in system complexity and user activity, each individual user must assume a degree of responsibility for the continued maintenance and integrity of their network—and, by extension, for the network overall Pfleeger & Caputo (2012), Shillair et al. (2015), West (2008). Because networks are permeable, typical users may compromise their own security and the security of others by unwittingly granting system access to malicious actors (for example, by downloading files without scanning them Fan & Yeung (2011)) or by failing to detect the presence of bot-nets in a slowly running system (which can destabilize large swathes of the overall network) Burghouwt et al. (2011).

At the same time, the complexity of the technology environment and the diversity of online tasks makes it difficult for individual users to protect their online assets. This, paired with the (perceived) complexity of security precautions and the sheer variety of security advice and related decisions, leads individual users to report low confidence in their own decisions and in their capacity to secure their own domains Al Hasib (2009), Shillair et al. (2015).

2.4 How do individuals consume security advice?

There are many ways in which individual users can encounter new security advice, but most involve some degree of formal and informal learning  Haney & Lutters (2018), Rader et al. (2012), Renaud & Dupuis (2019), Stanton et al. (2016).

Formal learning occurs through structured courses in an online or in-person classroom environment, usually followed by an assessment Caballero (2017), Hight (2005). For example, the delivery of security awareness programmes such as SETA (Security Education, Training and Awareness)² occurs within organizations and includes classes that train employees to recognize threats. These training programmes tend to focus on compliance with corporate policy Lee et al. (2016), and they evoke generic situational awareness Lee et al. (2016), rather than providing specific contexts and situations from which an individual user can learn.

Informal learning is unstructured, occurring outside of formal education contexts and without direct targeted interaction with security experts Stanton et al. (2016). Nevertheless, it is the primary way in which adults learn about the world around them Malcolm et al. (2003), Ollis (2011). As such, it is the main mode of learning considered in this paper. Informal learning is usually triggered by some internal or external impetus Malcolm et al. (2003), and it occurs primarily when individuals choose to actively seek out new ideas and advice.

Thus, for the purposes of this paper, we distinguish informal learning from incidental learning  Bull et al. (2008), Malcolm et al. (2003) on the basis of their differing intentionality: informal learning requires some kind of prior impetus and concerted effort, whereas incidental learning is often a by-product of carrying out another task Malcolm et al. (2003). Despite being a conscious decision, informal learning is often conducted haphazardly and influenced either by randomized chance Malcolm et al. (2003) or by the learning behaviours of others Bull et al. (2008).

Many of the studies concerned with individual users’ security intentions frame users’ situational awareness and knowledge as necessary conditions for appropriate security decisions Forget et al. (2016), Renaud & Dupuis (2019). Essentially, researchers assume that individuals must know about the issue at hand before they can make a reasonable decision. Thus, when an individual is faced with a security message about a potential threat, their decision process could proceed in one of two ways.

1. First, if they already possess prior knowledge about the threat (and, more importantly, about how to prevent it), they will take appropriate action. This is a threat control process  Renaud & Dupuis (2019).

2. Second, if they do not possess prior awareness or knowledge, and therefore do not know how to neutralize the stated threat, the security message may be rejected. The individual user may instead act to control the psychological fear generated by the message (rather than the practical threat implied by its contents) Forget et al. (2016), Renaud & Dupuis (2019).

Individual users may initially accept security advice, but subsequently reject it if they lack relevant coping strategies and actionable means to counteract the threat, choosing to deal with the issue in some other way Howe et al. (2012), Ion et al. (2015), Redmiles et al. (2016a). Arguably, then, the efficacy potential of security advice depends on how well suited it is to a given individual’s existing frame of reference. This poses interesting problems for security advice that is disseminated to a broad audience, as is the case with media-acquired advice. As such, we give particular consideration to the role of the media.

2.5 The efficacy potential of advice

According to self-efficacy theory, individual users pass judgement on their own ability to cope with a given situation, thus developing self-efficacy beliefs for a specific domain. Based on these beliefs, individual users are able to initiate and persevere with behavioural strategies that lead to successful outcomes Bandura et al. (1999), Maddux (1995). Self-efficacy in these cases tends to be a generative capability that allows individual users to organize their skill-sets and beliefs, which allows for an efficacy potential for these users Bandura et al. (1999).

What this means for cyber security advice is that, to enhance the efficacy potential, researchers must enact strategies which help structure and direct the behaviour of individual users towards goal setting, and measure the progress towards this goal Saks & Ashforth (1996) and many usable-security studies have investigated this, e.g. Pfleeger et al. (2014), Redmiles et al. (2016a), Warner (2012). Recent work such as that of Furnell has additionally highlighted through current usable-security concepts how the the field may require a return to a first-principles approach Furnell (2024). Furthermore, self-efficacy is closely linked to motivation, with the level of self-efficacy needing to be higher in order to correspond to the difficulty of the faced problem Bandura et al. (1999), Stumpf et al. (1987). As already noted, cyber security is seen as both important and complex, yet the motivation to enhance self-efficacy is limited (as explained in Herley (2009) and explored through psychological and cultural means in Halevi et al. (2016)).

2.6 The role of the media in security advice consumption

There are many possible sources of security advice available to individual users engaged in informal learning Rader & Wash (2015), including retailers and vendors of security software and services Rader & Wash (2015), Stanton et al. (2016), online sources with varying levels of expertise and credibility Stanton et al. (2016), governmental organizations such as NIST (in the United States)³ and the National Cyber Security Centre (in the United Kingdom)⁴ , professional media services such as the BBC and the Associated Press and online media organizations such as Ars Technica⁵ , which often create and distribute security content.

The media and communications field has the greatest reach of all of these sources. In 2017, Ruoti et al. (2017) reported that individual users primarily learnt about threats through four primary sources: advertisements, news reports, television dramas and movies. Subsequently, in 2018, Das et al. (2018) documented that news reports about threats (including cyber threats) were among the most-shared stories between individuals. Resources such as news outlets are particularly important for older users, especially when assessing the severity of threats and the pertinence of advice Nicholson et al. (2019). Even fictional news can influence individual decisions about security Fulton et al. (2019).

Additionally, the media and communications field is uniquely capable of influencing public opinion about security advice Ruoti et al. (2017). News sources facilitate group-based consensus Lasswell (1948) and set the agenda for what is regarded as an important topic, be it a presidential election McCombs & Shaw (1972) or an event such as the 2017 Wannacry threat Schirrmacher et al. (2018), which may be accompanied by security advice. Given this unique influence, we accord special importance to media-acquired advice in our research. Indeed, we would argue that researchers in usable-security have a duty to understand current practices in national and international media communications.

Media sources are adept at controlling both of these factors, using various strategies to prime the individual user and make them feel invested in the given topic—regardless of whether it truly pertains to them. This ‘taste-making’ function complements their primary advice-creation function. As such, media sources act as ‘knowledge brokers’ within informal learning contexts, facilitating the one-way delivery of information, concepts and ideas from professional sources to individual users Meyer (2010), Wenger (1998) and ultimately influencing the opinions, actions and personal development of the recipients of security advice Contandriopoulos et al. (2010). Consequently, the work described in this paper addresses both of these elements, as we endeavour to analyse both how media sources magnify the risk of certain security threats and their potentially associated mitigating strategies (through the use of an ontology to compare our results to), and how this security programming might gradually orient individual users’ perception of security advice in general over time through a sentiment analysis.

Thus, at this point, it is worth reiterating the research objectives of Section 1:

• RO1: What kind of informally learnt and actionable security advice most often appears in news articles?

• RO2: What is the efficacy potential of this security advice as consumed by an individual user?

3 Methodology

In this section we discuss the two elements that we utilized to obtain the necessary data. The first element was a news-scraper, which was developed in Python and was designed to extract complete articles from structured data sources. The second element was a viable search methodology, which was assembled from multiple components. We first give consideration to the news-scraper.

3.1 The news-scraper

Web scraping is a technique that allows researchers to automate the capture of online information. Scrapers are popular tools for digital research, and they are often characterized as ‘outsider’ tools that can be used with freely available online data—that is, data that does not require privileged access Marres & Weltevrede (2013). To ensure that we had enough information to answer our research objectives, we designed our news-scraper to collect as much data as possible from our news sources. The tool’s basic functional requirements are shown in Table 1; this gave rise to the abstract architecture depicted in Figure 1.

We utilized a news-aggregation API to filter content from a variety of unstructured and structured news sources were consistent with our definition, and we added functions to enable the complete capture of content, in accordance with Requirements 2 and 3.

The captured data was then fed into a data-storage pipeline before being converted into a flat-file database storage solution. Incoming data was merged with existing records when required to avoid duplicate data.

3.2 The search terms

To fulfill Requirement 1, we followed the precedent of Schatz et al. (2017), who sought to derive a more precise definition of security by utilizing Google Trends to automatically collect the phrases that individuals were using to search for security content. As this had to be accomplished from the perspective of our individual user, this excluded the possibility of replicating the work of Humayun et al. (2020) who looked at primary studies undertaken within academia. Instead, we followed the Systematic Mapping Study protocol of Kosar et al. (2016).

We defined a set of search and inclusion/exclusion criteria (for example, Cybersecurity OR Cyber AND Security) and additional queries containing both base search terms and queries derived from Google Trends (online OR advice OR protection OR protect OR prevent OR preventative OR tips OR email OR social network OR password OR hack OR hacked OR hacking).

We augmented the Google Trends queries with phrases pertaining to 20 cyber security events that (1) had occurred in the previous 24 months and (2) had been covered by at least 10 major English-language news outlets (for the queries, please refer to Table A8). Except for where it was appropriate within the event searches, all search terms were technology-agnostic—they did not include explicit references to products or services. The news-scraper then carried out searches over a 24-month time span and returned all results that included these terms within the title or body of the content. Therefore, while not exhaustive, our corpus represents security advice as accurately as possible within the confines of our scope.

3.3 Cleaning the data

First, we screened our results according to the inclusion/exclusion criteria. These were defined as follows:

• Must be a news or blog article that directly addresses at least one aspect of cyber security/contain our search terminology directly. Blog articles were limited to tutorials, editorials, tool demonstrations and discussion of technical reports.

• Must be written in English (due to the nature of our analysis methodology).

• Must be accessible and not hidden behind a paywall or other kind of lockout mechanism (as in these cases only a few lines of text may have been retrieved).

Any article found to be in breach of these criteria was excluded.

In this way, we reduced the initial pool of 16,876 usable articles from our first cleaning process to 15,422 individual articles. For the remaining articles our focus and technique were informed by recent work, such as that of Satyapanich et al. (December, 2019), which describes the process for extracting semantic information (such as people, places and events) from security articles and that of Al Moubayed et al. (2017), who used Bayesian topic modelling to ascribe classifications to, and uncover trends in, security and criminal documents. We prepared the corpus for analysis using common data pre-processing techniques. We utilized tokenization to break down the text, first into sentence units and then into individual words. We then replaced uppercase text with lowercase equivalents and removed punctuation. We lemmatized the corpus to standardize the tense and to replace any third-person words with first-person variants. Finally, we used a stemming technique to reduce words to their root form, where appropriate Porter (2006).

3.4 Classifying the data

As we saw in Section 2.4, individuals require actionable elements within their security advice. We therefore utilized an ontological framework to help us classify and integrate the data collected from the sources queried by our news-scraper. The application of ontologies to model and reason about cyber security requirements has gained significant attention in recent years, particularly for complex systems such as critical infrastructure and smart cities. These ontologies provide a formal, machine-readable representation of key concepts and relationships, enabling precise capture and communication of security needs, automated reasoning and analysis, knowledge sharing and reuse across domains and integration of security with other system aspects early in the development lifecycle De Nicola & Villani (2021). While the critical role of end-users in the overall cyber security posture of organizations and systems is increasingly recognized, current research focuses more on incorporating end-user perspectives into broader cyber security frameworks and models. For instance, some frameworks aim to identify users’ security behaviors in real-time and provide targeted interventions Ruighaver et al. (2007), while others leverage serious games to train users in detecting and responding to social engineering threats Hendrix et al. (2016). However, no clear examples of ontologies solely focused on targeting end-users and their immediate requirements were found, indicating a potential gap in current research that warrants further investigation Groš (2021), Oltramari et al. (2015).

In response to this we decided to perform a non-exhaustive search for an ontology which could be applied to end-user behaviour, even if the intended target audience is not explicitly defined as such. We began by searching for ontologies using keywords such as ‘cyber security’, ‘end-user’ and ‘actionable advice’, and then reviewing them against a set of selection criteria shown in Table 2.

In attempting to follow this criteria we found that many ontologies were indeed aimed at a technical or policy audience and often included several layers of abstraction within the work, used vaguely defined terminology or simply did not include our original requirement of actionable security advice⁶.

An ontology by the Center for Internet Security (CIS)⁷ was chosen, which meets all of the inclusion criteria outlined in Table 2. We discuss how it does so below.

• Criteria 1: Although it was not intended specifically for individual users, this ontology prioritizes risk-based security and focuses on the practical mitigation of these risks by identifying and utilizing 20 domain-specific CIS-vectors that represent practical and actionable remedies for security threats. It does so through providing an evaluative framework that allows users to assess their security posture against specific control objectives. The controls are prioritized and provide clear guidance on essential cyber hygiene measures, making them accessible even to those with limited cybersecurity expertise. A high-level version of the framework can be seen in Figure 2. The individual CIS-vectors are discussed in detail in Section 4.

• Criteria 2: CIS Controls provide specific, actionable guidance on the most critical steps organizations should take to tangibly improve their security, covering Criteria 2, whereas other ontologies may be more descriptive, or somewhere in between Adach et al. (2022). As an illustrative example of Criteria 2, we refer to Woods et al. (2017), where the CIS ontology was shown in use with insurance underwriting professionals when selecting policy controls.

• Criteria 3: According to CIS, the CIS Controls are also frequently updated by a global community of experts to address the evolving threat landscape and incorporate recent security concepts (further confirmed by the fact that version 7.1 was utilized at the time of writing, which was then superseded by version 8) ⁸. In addition to meeting criteria 3, the CIS ontology has been aligned with other ontological frameworks such as that of NIST to allow for easier adoption by organizations and projects⁹.

This pragmatic approach, combined with the clear tie-in to demonstrating security achievement, allows it to provide us with the requisite entity types and properties which are ascribed to individual news articles within the corpus as additional metadata. Thus, we are able to use the ontology to define the entities, relations and other factors that can be extracted from the corpus. The ontology also allowed us to focus the corpus and to restrict our vision to the research objectives, as the language utilized within security can range from extremely specific to extremely ambiguous Ruohonen & Kimppa (2019). In many cases, this range can make it difficult to apply an ontology to specific news articles within the corpus.

3.5 Additional work to encompass null values from CIS-vectors

Given our search terminology, we observed that 6,134 of the 15,422 articles (representing 36.3% of the total corpus) contained references to any of our CIS-vectors. We performed a second pass on the corpus, introducing additional syntactic variants of the terminology utilized within the CIS-vectors. For example, we separated ‘malware defenses’ into ‘malware AND defences’, ‘malware defence’ and ‘malware defense’ to correct for localization issues.

The results of the second pass are illustrated in Figure 4.1. As can be seen, the occurrence rate was subsequently 7,988 articles, or 51.7% of the corpus. Each of these articles contained references to one or more CIS-vectors. For the remaining 48.3% of the corpus, we performed a Latent Dirichlet Allocation (LDA) analysis of these articles in order to generate further details, the results of which are outlined in Section 4.2. LDA is a statistical modeling tool that allows for the discovery of otherwise abstract topics within text files. It provides us with both a topic-per-word and topic-per-document model. To ensure the accurate selection of topic numbers and models, we followed the methodologies proposed by Cao et al. (2009) and Deveaud et al. (2014).

4 Results

In this section we give consideration to our results. First, we corroborate the findings of Alagheband et al. (2020), which indicated that coverage of security topics in the New York Times has steadily increased over the last decade. Figure 3 highlights this increase over time: the ‘vast terra incognita of print’ Taylor & Wolff (2004). The data also exposed the sheer diversity of publishers, ranging from traditional outlets such as the BBC news and CNN through to specialty security blogs. Even so, we must acknowledge that this list is inevitably incomplete, as our search methodology, while extensive, was non-exhaustive and it was limited to English-language media.

Next, we identify the prevalence and features of ‘ideal’ news articles in our corpus and use this information to help answer our research objectives. An ideal news article must contain a summary of the information that an individual user requires (in this case, regarding security advice), eliminating irrelevant and redundant information wherever possible Goldstein et al. (1999). To determine the prevalence of such articles in our corpus, we first utilized our CIS-vectors to ascertain how many of the articles contain content-specific vocabulary that users may expect to find within these articles, and we performed additional analysis on those articles that contained no such terms. We then derived statistics pertaining to sentence length and vocabulary size, which we then compared to third-party corpora (where available). Finally, we utilized sentiment analysis as an efficacy potential measurement tool, building on work by Kalra & Prasad (2019), who used it for stock market assessments. This was done to decipher any trends that could inform our efficacy potential research question.

4.1 CIS-vector occurrences

Figure 4 highlights the occurrences of our CIS-vectors in the corpus. The most-used CIS-vectors were CIS-13 (Data protection), CIS-11 (Limitation and control of network ports, protocols and services) and CIS-2 (Inventory and control of software assets). CIS-13 highlights the growing trend towards data protection awareness and its relevance for individual users; it occurred 0.4 times per article, on average.

Delving deeper into the reasons for this expanding data protection coverage, we find that, between 2018 and 2019, the most significant topics were related to data breaches, data protection guidelines for individuals and organizations (such as the EU’s General Data Protection Regulation (GDPR)¹⁰ ) and data privacy-related security advice for social media users. In 2020 there was a shift towards protecting health-related data in medical contexts, with advice and threat messaging geared towards disease contact and exposure tracing applications, such as those mentioned by Yasaka et al. (2020).

CIS-11 indicates network security-related information and advice, and its occurrence rate increased significantly between 2019 and the end of 2020. At least one publication (Lindner et al., 2020) notes a similar increase in interest. Again, we found that most of this network security advice was related to privacy, and it appeared in texts ranging from technical articles to installation guides for the Tor Project. In many cases, these articles contained more difficult vocabulary and technical terminology than the average publication.

CIS-2 pertains to software assets and their associated CIS-vectors, and it proved to be one of the most diffuse topic. In our corpus, we found articles linked to Internet of Things home security, smart grid and connected vehicle software and security issues that arise in connection with these devices and services.

Correlations between the CIS-vectors are depicted in Figure 5. The correlations were weak across the corpus, with one notable exception: the correlation between CIS-16 (Account monitoring and control) and CIS-4 (Controlled use of administrative privileges). Though CIS-16 appeared more frequently overall, tokens associated with both vectors appeared consistently between articles.

4.2 Articles containing no CIS-vectors

Table 3 lists the most common topics that occurred in those articles that featured no CIS-vectors from our classification (representing 48.2% of the corpus). The topics were derived through LDA topic modelling, as described in Section 3.5.

We can see that, despite the absence of CIS-vectors, security is still a focal point in these articles. In these cases, though, the focus is on national (cyber) security (Topic 11), cyber crime (Topic 14), business threats (Topic 9) and health and safety issues related to cyber crime and security (Topic 6). Topic 3 embodies similar concepts as CIS-vectors CIS-13 (Data protection), CIS-11 (Limitation and control of network ports, protocols and services) and CIS-2 (Inventory and control of software assets).

4.3 Sentence length and vocabulary size

We use sentence length, vocabulary size and a selection of readability scores as proxies for difficulty.

4.3.1 Sentence length

Sentence length is an often-utilized tool in the discovery of readability within corpora Goldstein et al. (1999), Lim et al. (2018). Figure 6 displays the average article length. The mean article length was 9.92 sentences, and the median length was 10 sentences. We can compare to the work of Goldstein et al. (1999) on the automated summarization of news articles, which led to a corpus of 1,000 Reuters articles with a (post-summarization) average length of 23 sentences. We can also compare this to the work of Lim et al. (2018), whose smaller corpus yielded an average of 14 sentences per article.

The gap between the publication of these comparators (1999 and 2018, respectively) may suggest an overall decline in the length of news articles. It also suggests that our corpus of security-specific news is on the shorter side of the spectrum. This last point is, however, caveated by the fact that a comparison with a more historical data and a wider potential variety of possible sources would be needed to further confirm this finding.

4.3.2 Vocabulary

We estimated the vocabulary growth of the corpus using Heaps’ law Heaps (1978), which describes the relationship between tokens and types. This law states that a vocabulary, expressed as $v$ unique word types, is proportional to the power law of $n$⁠, the number of tokens in an arbitrary text. The relation is expressed as

Here, $K$ is a positive constant and $\beta$ lies between 0 and 1. In effect, as a body of text increases, the potential to discover new distinct word types decreases. In our corpus, we can see from Figure 7 that the vocabulary range largely adheres to the predicted value (black line). This means that new vocabulary terms are continually arising in the data, which could complicate users’ informal learning.

4.4 Readability scores

A readability index, such as the ones shown in Table 4, is an estimation of how difficult a text is to read. In online environments, it is often measured to assess click-through rates and user satisfaction Kanungo & Orr (2009). Grinberg (2018) utilized it, alongside sentence length, to model user engagement with news articles. As such, it is an interesting variable to consider when assessing the efficacy potential of the texts in our corpus.

Readability is determined by measuring a text’s complexity, which is approximated via quantifiable attributes such as word length, sentence length, syllable count and so on. The Flesch–Kincaid test Flesch (2007) is one of the most utilized readability tests, and it calculates readability by (1) dividing the number of utilized words by the number of sentences and (2) dividing the average number of syllables per word by the number of utilized words. The scoring range starts at 100 for the easiest to read and descends to 0 for unreadable texts. As an example, the combined Harry Potter novels have a score of 72.83. Other frequently used systems include the Gunning–Fog index Roberts et al. (1994), which looks at sentence length and number of polysyllabic words; the Coleman–Liau index Coleman & Liau (1975), which does not assess syllables; the Automated Readability index Senter & Smith (1967); and the Simple Measure of Gobbledygook (or SMOG) Laughlin (1969), which utilizes a similar methodology as the Flesch–Kincaid, but from sections within the text. All of these metrics utilize a 100–0 scoring system and are broadly comparable with each another. As such, we employ all of them in this study.

Table 4 highlights a selection of readability scores, all utilizing the same 100–0 scoring scale. To ensure the accuracy and reliability of our analysis, outliers in the readability scores data-set were identified and removed. The outlier detection was performed using the Interquartile Range (IQR) method. We calculated the first quartile (Q1) and the third quartile (Q3) for each readability measure. Outliers were defined as scores falling outside the range given by the following:

where $\text{IQR}=Q3-Q1$⁠. Outliers were removed to improve the accuracy and interpretability of the results. The presence of extreme values in the data can distort statistical measures such as means and variances, and can affect the distribution and visualization of the readability scores. After removing outliers, we re-evaluated the distribution and summary statistics of the readability scores.

Table 5 above summarizes the number of outliers, total data points and outlier percentages for each readability measure. The percentages of outliers were relatively low, ranging from 2.73% to 4.05%, which is generally considered manageable. Figure 8 illustrates the distribution of readability scores after removing outliers.

The readability analysis of the corpus reveals that the text exhibits substantial complexity, as indicated by various readability metrics. The Flesch–Kincaid test’s Grade Level, with a score of 12.52, suggests that the text is suitable for readers who have completed secondary education. The Gunning Fog Index, at 16.03, implies that the text is intended for individuals with a college-level education, reflecting its complexity through longer sentences and a higher proportion of complex vocabulary. The Coleman–Liau Index, scoring 13.23, aligns with a reading level approximately one year beyond secondary school, focusing on average word length and sentence length. The SMOG Index, which stands at 14.55, indicates that a more advanced educational background is necessary for full comprehension, generally suggesting some college education or higher. The Automated Readability Index (ARI) of 13.00 further supports this, suggesting that the text is best understood by high school graduates or college students. Finally, the Average Grade Level (AGL) of 13.87 corroborates the high complexity of the text, pointing to a university-level readership.

These indices collectively illustrate that the corpus is tailored for an educated audience, characterized by complex sentence structures and sophisticated vocabulary. If the aim of the articles is to make the information more accessible to a wider audience, it may be beneficial to simplify the language and structure. However, due to the inherently complex and multifaceted nature of cybersecurity issues, such discussions will inevitably involve challenging concepts. As a result, the average casual reader may find limited value in engaging with these articles.

4.5 Sentiment analysis

Sentiment analysis is a group of text analysis techniques that allow for the automatic derivation of sentiment (positive or negative) from large data-sets Hussein (2018). Sentiment analysis is widely used across domains, from marketing Hussein (2018) to stock market analysis Kalra & Prasad (2019). Previous sentiment analysis work within the security field has focused on predicting cyber attacks or identifying potential perpetrators, for example, by assessing sentiment in online hacker forums Macdonald et al. (2015). The lexicons generated from these studies (for example, those pertaining to sentiment within political analysis of sovereign cyber capabilities) are of limited use within our work, as their terminology often differs substantially from what could be construed as ‘security advice’ based on our definition. As such, we utilized Latent Semantic Scaling (LSS), which is a semi-supervised technique for scaling documents based on work by Deerwester et al. (1990). It allows for a limited set of pre-generated seed words, which are words embedded with a specific positive or negative value. To produce our small library of seed words, we utilized the SENTPROP Hamilton et al. (2016) framework. We chose this framework as it combines word-vector embeddings with a label propagation approach, which are well-known techniques to generate seed-word libraries. Additionally, SENTPROP can generate accurate results with smaller corpora. Hamilton et al. (2016). In our system, the overall sentiment of a news article is correlated with the sentiments of individual words within that article, thereby allowing for a sentiment polarity check.

The results of this sentiment analysis process can be seen in Figure 9. The scores suggest an overall decrease in positive sentiment over the time period; however, these results are not statistically significant, likely because (1) the increase in published articles over the time period distorted the results and (2) a high p-value deviated significantly across the standard alpha value (set at 0.05).

5 Discussion

We now consider our results in the context of the research objectives of Section 1. We focus on the results derived from our CIS framework and associated vectors in Section 5.1, and ascertain how the readability, vocabulary and sentiment of the corpus affects its efficacy potential in Section 5.2.

5.1 What kind of informally learnt and actionable security advice most often appears in news articles?

Three overarching themes prevail in our security corpus. The first is data protection (Theme 1), which is reflected in the strong focus on CIS-13 (Data protection) and Topic 3 of our LDA analysis. The second is physical and digital security (Theme 2), which is supported by CIS-11 (Limitation and control of network ports, protocols and services), CIS-2 (Inventory and control of software assets) and Topic 3 of the LDA analysis. The third is personal and collective safety (Theme 3) in the face of personal, business or sovereign threats to one’s security, which is supported by Topics 3, 11 and 14.

All of these themes represent a unique set of constructs and associated user behaviours. For Themes 1 and 3, a significant driver for personal safety is privacy: ‘the right of a party to maintain control over, and confidentiality of, information about itself’ Oldehoeft (1992). Although privacy is a significant token by itself (appearing 4,887 times in the corpus), further indirect references to it suggest that it is the underlying motivation for a significant number of data protection-related articles, be they in the realm of health data, shopping data or, more broadly, associated with the GDPR.

In Theme 2, personal safety entails the need for user intervention in faulty systems, either because the system cannot determine the cause of a certain threat or the appropriate corrective action to take, or, in some cases, because the system itself is acting maliciously towards the user. These articles were the most likely to contain directly actionable security advice, and thus were the most efficacious for individual users.

Theme 3 also encapsulates threats to business and sovereignty. These articles are unlikely to contain actionable security advice, but they can aid in the creation of policy Cook (1998), which may then lead to actionable advice. These articles may even influence public opinion regarding the (cyber) security of national sovereignty, much like how terrorism news shaped national opinion and policy, as seen in work by Gadarian (2010). This cycle of influence leads to the creation of policies and legislation, such as the aforementioned GDPR, which in turn influences public awareness of potential data security threats, ultimately stimulating new forms of cyber offense and defensive capabilities. These capabilities are then disseminated to individual users, potentially as a form of security advice. Assessing future developments within these themes and re-assessing their relevance periodically could provide a lens for evaluating the past, current and future impact of news media on security advice efficacy potential.

5.2 What is the efficacy potential of this security advice as consumed by an individual user?

Many of the articles an individual user may access for cyber security advice may contain subject-specific vocabulary (such as that found within our ontological framework). Given that (1) there is limited overlap between advice sets within our ontological framework and (2) the average length of the articles in our corpus (expressed as sentence length) is shorter than the average length of comparator articles (see Section 4.3), there appears to be a certain level of focus within the articles that could indicate efficacy potential. However, we have also seen from ontological frameworks such as the CIS-Control schema that these tools may not encompass all of the possible security vectors within the current media environment. Furthermore, these results must be qualified given our topic modelling methodology. Our application of Heaps’ law highlights the growing vocabulary within our corpus, demonstrating that the subject-specific terminology in news articles on security advice is continuously evolving. This may point to an increasingly diversified interest in security advice that is tailored to a specific, predetermined goal. This encourages us to question the efficacy potential of all-encompassing frameworks such as the CIS-Control schema.

The results of our readability tests and sentiment analysis may further challenge the efficacy potential of current media-mediated security dissemination. We find within our corpus a trend towards high reading difficulty levels: ease of reading correlated with publication type, and news articles ranked higher on all readability indices. As all five of our assessment metrics reported statistically significant results with similar distribution scores (see Table 4), we can confidently assert that just 3% of our corpus was written at a U.S. school system 6th-grade level, which is typically the recommended reading level for standard distributed materials Kher et al. (2017). Most of the articles in this corpus require a reading level of a typical college undergraduate.

Recalling that an individual user must have (1) a sense of certainty about the content, (2) a personal interest in the content and (3) sufficient ability to deploy the content in order to feel sufficiently compelled to act on the information, this threat control process could easily be derailed by the continued divergence and growth of subject-specific vocabulary and dense prose. Haney & Lutters (2018) argue that there is a rejection threshold that informs the maintenance of security in a rapidly evolving landscape, and they maintain that individual users are approaching this threshold.

Security is not the only specialized field that deals with these dissemination issues, and it may be helpful to observe the solutions pursued in other contexts. For example, medical advice dissemination to the general public (taken here as the equivalent of our ‘individual user’) also involves communicating complicated concepts and extensive vocabulary to individuals who have no relevant formal training on the subject. Britt et al. (2017) found that many readers stop reading medical texts if they gauge significant difficulty within the first few sentences. Consequently, the American Medical Association (AMA) and the U.S. Department of Health and Human Services (USDHHS) have set explicit guidelines that require public-facing information to achieve a U.S.-standardized readability level of 6th grade or below Kher et al. (2017). Extrapolating these considerations to our own corpus, it would stand to reason that increasing readability to a more generally accessible level could constitute a cost-effective remedy.

Although the overall sentiment of the corpus would not suggest that users may be being treated as an enemy (as, for example, was documented in Adams and Sasse’s seminal 1999 paper; Adams & Sasse (1999)), it does appear that what we encountered would not fulfil Kerckhoffs’ criterion for ease of use. Neither would we agree that cyber security advice as portrayed in our corpus allows for self-efficacy upon reading. Instead, an individual user must face security topics using a multi-pronged approach, whereby self-efficacy is derived from multiple sources of increasing complexity. If the cyber security field is to continue down the path of increased specialization, perhaps the time has come to recognize this emerging reality and clarify—in a transparent fashion—the expectations that are being placed on users.

6 Limitations and future work

The scope of this study was limited by the type and amount of information we were able to acquire to build the corpus. In our case, this meant focusing on English-language material, even though a preliminary search conducted before implementation unearthed a rich catalogue of data in other languages. This also means that our security topics, analysis and findings likely exhibit Anglo-Saxon bias. The technical tools utilized for the readability scores were also designed for English-language articles. There is significant scope for the enhancement of our search methodology, where for example users may only utilize the first page of any search enquiry Höchstötter & Lewandowski (2009). It is our hope that this methodology be utilized to answer the same research objectives in other languages and cultural contexts.

Whilst we underscored the suitability of the CIS ontology, we also must recognize the drawbacks of this approach. The CIS ontology, although prescriptive in the manner in which it prioritizes controls, lacks risk assessment specifics and may lead to misaligned priorities and gaps as the end-user may have differing priorities. Furthermore, its suitability can also be attributed that it is due to ambiguity around its own intended target audience, and finally the CIS Controls have not undergone rigorous scientific analysis of their efficacy despite their popularity Groš (2021). However, as we are utilizing this ontology in an effort to answer our research questions rather than appealing directly to users, and as the other ontologies we surveyed suffer from broadly similar drawbacks, we do not consider these drawbacks to be sufficient to remove it as our choice. Instead we believe that more scientific analysis and sharing of case studies on CIS Control implementations by the community would also help solidify their value proposition, and its use underscores the need for further development in user-focused cyber security ontologies which may serve as a better basis from which to base a study such as ours.

We utilized automated methodologies in order to classify topics and measure sentiment and reading difficulty, and the results are tempered by the respective limitations of these methodologies, in particular the use of a bag-of-words model, which does not capture semantic meaning or context. This method treats words as independent features, potentially leading to overestimations in mapping articles to CIS controls. For example, the mere presence of keywords might incorrectly suggest relevance to a control, disregarding nuanced meanings conveyed through context. This limitation can skew our analysis, highlighting the need for advanced techniques like word embeddings or transformer models to improve semantic understanding and mapping accuracy. Moreover, our results represent a specific snapshot in the security timeline; access to a larger historical data-set would inevitably change the overall results, potentially yielding a more statistically significant sentiment analysis.

Our approach to tackling the second research objective may limit the usefulness of our conclusions. We approximated article efficacy potential by using text analysis to predict user engagement, and we did not consider other metrics that could have enhanced the findings. Traditionally speaking, reading-difficulty assessments in laboratory settings involve comprehension tests, eye tracking and brain-imaging. Knowledge of how users interact with our corpus in these terms would allow for a significantly richer analysis of security advice efficacy potential.

The aforementioned limitations can, of course, be addressed in future research that builds upon what is presented here—not least because our research method (described in Section 3) allows for continuous data capture. Furthermore, the data within this corpus could serve as the foundation for further analysis of security advice dissemination. Because this corpus contains a significant variety of sources, structural analysis of sentence construction for threat messaging could reveal the rhetorical structure of fear appeals, as per previous work in the field such as that of Renaud & Dupuis (2019). A fear appeal is designed to motivate the reader to execute security advice, and an in-depth analysis of its features could yield results that would improve the efficacy potential of security advice dissemination.

The corpus itself could be augmented with social media data, which would add the significant vector of digital naïve advice Schotter (2003). Bias within the articles could be used as another indicator of efficacy potential via methods like that presented by Lim et al. (2018). We believe that the results of this study can provide a basis for further reflection on security advice dissemination, and that it can stimulate a conversation about individual users’ learning environment. Importantly, we hope that it serves as a point of departure for future studies.

7 Conclusion

We have presented work on a corpus of security advice generated from mainstream news articles as might be faced by individual users on a regular basis. The work was oriented by two questions: (1) What kind of informally learnt and actionable security advice most often appears in news articles? (2) What is the efficacy potential of this security advice as consumed by an individual user?

We found that news-mediated security advice has been increasing since 2018, and that many such news articles focus on specific security topics. This level of focus may indicate efficacy potential. Additionally, we found that news-mediated security advice is characterized by short article length and low readability, making it difficult for many individual users to comprehend its content. We found that the subject-specific terminology within our security news articles is continuously evolving, potentially indicating increasingly diversified interest in goal-specific security advice. Again, this may increase the relative difficulty of acquiring and comprehending news-mediated security advice, with an associated impact on efficacy potential. Our approach involved using quantitative methods to yield qualitative findings. Our hope is that this research can help lay the foundations for various means of quantifying and improving the efficacy potential of security advice dissemination.

7 Data availability statement

The data that support the findings of this study are available in a repository and can be accessed here: https://huggingface.co/datasets/Quinm101/cybernewsarticles.

Footnotes

References

A Appendix A Search terms