The ethical problems of ‘intelligence–AI’

Abstract

Interest in artificial intelligence (AI) has grown rapidly alongside the ability to examine information in far vaster quantities and from more diverse sources, and to provide previously unattainable forms of evaluation. For the intelligence community, AI offers an important solution to their data collection bottleneck, allowing the data to be processed and analysed at speeds and in ways not previously possible. While AI has received some general criticism, when it is combined with the reach, secrecy, and coercive power of the intelligence community it creates unique ethical problems. Intelligence–AI exacerbates the biases AI creates, undermines proposed transparency solutions, and creates new ethical dilemmas and harms. This article examines intelligence–AI across its collection, processing, and analysis phases. It argues that open-source does not necessarily mean ethical, as the AI collection en masse of social media data violates citizens' privacy, consent and autonomy. The article also argues that AI-aided categorization is overly reductive and perpetuates harmful social binaries, while also revealing new private information beyond what was initially shared. Finally, it argues that the secretive intelligence environment prevents critical interrogation while promoting practices that, through the coercive power of the state, cause unequal harms across society.

Interest in artificial intelligence (AI) has grown rapidly in recent years, along with the ability to process and analyse information in far vaster quantities from a more diverse range of sources, and to propose new forms of evaluation, far beyond what humans can achieve. Given this processing power, the potential that it represents to state security and intelligence actors should come as no surprise. In order to protect the political community across its social, economic, political and military interests, the intelligence community relies on being able to collect as much information from as many sources as possible—both open and secret—so as to detect threats and prevent them from materializing. However, this drive for ever greater quantities of information has generated a significant bottleneck at the processing and analysis stage, where there is too much information to review. AI therefore offers the potential to process this information in quantities and at speeds previously not seen, while also offering new analytical insights. Indeed, the United States' National Security Commission on Artificial Intelligence stated in 2023 that ‘AI will help intelligence professionals find needles in haystacks, connect the dots, and disrupt dangerous plots by discerning trends and discovering previously hidden or masked indications and warnings’.¹

This growing interest in AI is part of a much broader drive to advance AI capabilities as quickly and widely as possible, reflecting Russian president Vladimir Putin's statement that whoever becomes the leader in this field will rule the world.² Indeed, research into the field of AI has grown rapidly, outlining the good AI can play in citizens' lives, especially in areas of government and bureaucratic efficiency, the provision of health care, well-being initiatives and education, tackling climate challenges or providing greater national security.³ The US' 2018 National Defense Strategy identified AI as one of the key technologies that will ‘ensure [the US] will be able to fight and win the wars of the future’.⁴ In a dedicated AI strategy document, the Department of Defense directed itself ‘to accelerate the adoption of AI and the creation of a force fit for our time’,⁵ while the administration of President Joe Biden has warned against the growing assertiveness of China and Russia in the AI space.⁶ In response, the US intelligence sector, spearheaded by the Office of the Director of National Intelligence through its Intelligence Advanced Research Projects Activity (IARPA), has prioritized AI by harnessing its ability to process massive and noisy data sources, where there is a large amount of meaningless information, analyse language and forecast major geopolitical trends and societal crises.⁷ This commitment is evident in the CIA's ‘over 100 AI initiatives’ and the Department of Defense's substantial investments. These reached US$2.5 billion in 2021, with the military already integrating AI into live combat via drone programmes.⁸ Similarly, China and Russia are making significant investments in AI, with China setting itself the goal of becoming a world leader in AI by 2023, and Russia placing significant focus on military AI, with the emphasis on robotics.⁹ In the United Kingdom, the Government Communications Headquarters has set out its aims for using AI to improve its intelligence provision.¹⁰

However, this boom in AI technology has also sparked some concern over its tendency to inherit biases from its training as well as over its inaccurate technology, which leads to misrepresentation and unfair results, with a lack of sufficient transparency, oversight and reliability.¹¹ Work on the governance of AI has resulted in the development of a series of ad hoc and patchwork international agreements, domestic policies and guidelines. Internationally, this has included the United Nations' regulation of autonomous weapons in 2014, as well as the adoption by the OECD in 2019 of AI ethical principles, UNESCO's 2021 Recommendation on the Ethics of Artificial Intelligence, the G7's Hiroshima AI Process and the Council of Europe's work towards a legally binding international convention on AI and human rights.¹² At the domestic level, a variety of guidelines have been developed that argue for the need to embed key ethical principles such as transparency, accountability, respect for human dignity, freedom of the individual, equality and unbiased analysis and application, and privacy and data governance into AI design and implementation.¹³

The difference with what I shall term ‘intelligence–AI’, however, is that in combination, intelligence and AI exacerbate each of the initial problems while simultaneously undermining the currently proposed solutions, creating new ethical dilemmas and potential harms to citizens. For instance, state intelligence actors have the legal authority, physical infrastructures and institutional drive to collect data, to a greater extent than any private AI company, and when these collection mechanisms are combined with AI it enables new datasets that would otherwise be unmanageable or too disconnected to be of use. This widens the reach of data collection beyond that of any private actor and creates an exponential threat to privacy. The inherently secretive nature of intelligence further exacerbates this impact by preventing many of the proposed oversight and transparency mechanisms, meaning that populations are unaware—and unable to challenge—how their information is collected or used. It also prevents both critical internal reflection of the AI's calculations and external review. Unlike private actors, intelligence organizations using AI possess the coercive power of the state, representing an important symbolic and actual threat to people's human rights. In combination, therefore, intelligence–AI raises new and unique harms to both individuals and society that are not yet fully detailed in other works.

In order to fully explore the ethical implications of intelligence–AI, this article will examine how emerging technology has been used across the collection, processing and analysis phases of intelligence.¹⁴ Despite the inherently secretive nature of the world's leading intelligence organizations, they have publicly reported data on intelligence–AI, most notably in the US and the UK, as a result of their reliance on academic and industrial leaders for co-developing AI technology. Mapping the use of intelligence–AI is possible through the examination of public documentation by private companies, research institutions and state security actors, with examples often shared via media outlets as well as across academic writings.¹⁵ In this article, I will first argue that open-source does not necessarily mean the same as ethical, as the AI collection en masse of people's social media information violates their privacy and undermines their consent and their autonomy. Second, in terms of the processing phase, I will argue that AI-aided categorization is overly reductive and perpetuates harmful social binaries, while also revealing new private information beyond what people have intentionally shared. Finally, I will argue that the inherently secretive environment of intelligence prevents critical insight or reflection, further embedding the social divides AI necessarily creates, while also promoting intelligence practices that, through the coercive power of the state, cause unequal harms across society.

Intelligence–AI

AI is an umbrella term that refers to a variety of computer-aided processes and data analytics that mimic human thought processes or are tasked with activities that usually require human intelligence and insight. This includes machine learning, a subset of AI that involves training algorithms to learn patterns to make predictions from data without explicit programming, ‘to accomplish a task without being explicitly programmed to do so (i.e. learn on its own)’.¹⁶ ‘Deep learning’ is a specialized subset of this process, ‘inspired by … the human brain’, that involves layers of analysis conceived as neural networks that are capable of automatically learning representations from data via trial and error at multiple levels of abstraction.¹⁷ This can involve ‘supervised learning’ using known and labelled data to train the algorithms; ‘unsupervised learning’ which identifies patterns in unlabelled data to uncover insights or group similar data points; or a hybrid approach to combine different AI techniques, including supervised and unsupervised learning, along with rule-based systems, to solve complex problems by integrating diverse learning and decision-making strategies.¹⁸

For intelligence actors, this promises that they will be able to take truly massive quantities and varieties of data and through descriptive, diagnostic, predictive and prescriptive analysis determine what has happened, why it happened, what will happen next and what the response should be, detecting and forecasting new threats of societal crises, disease outbreaks, cyber attacks or major geopolitical trends.¹⁹ For example, in the US the National Security Agency uses machine learning on open and closed signals intelligence, analysing language and speech to provide insights into the decision-making of individuals, groups and organizations.²⁰ Meanwhile, the National Geospatial-Intelligence Agency and the US military use AI on satellite images to aid in object classification and to identify potential threats and appropriate targets,²¹ and in the social sphere, ‘imagery analysis, biometric technologies (such as face, voice, and gait recognition), natural language processing [hereafter NLP], and algorithmic search and query functions’ mean that intelligence actors can analyse the interconnections of social life of both the individual and society in ways that were not previously possible.²² This includes gathering and analysing social media data to identify, locate and track individuals or groups of concern and predict their behaviour. For example, X (formerly Twitter), Instagram, YouTube, Facebook and various other online platforms are now realistic options for data collection, as AI can analyse the billions of data points generated daily around the world. This can include using NLP, a developing area of AI that enables machines to understand, interpret and generate meaning from human language sources.²³ What is key is that the AI develops its own route to determining the correct answer. It can be given correct inputs and marked on successful outputs, but the route through the layers and algorithms used is decided by the AI, the complexity of which often means that this process is beyond the developers' or users' ability to track or necessarily understand.

Collection: open-source does not mean ethical

One practised assumption of intelligence–AI is that the acquisition of open-source data is ethically unproblematic. Many intelligence–AI programmes use the en masse collection of open-source data across a range of different sources without any indication of a limit, whether at the collection, processing or analysis stages. This involves the continuous, automated analysis of publicly available data in order to anticipate and/or detect significant societal events such as political unrest, humanitarian disasters and disease outbreaks, and includes gathering information from social media platforms, official government data and online data from websites, blogs and vlogs.²⁴ It does not involve the collection of information from targeted individuals where there is some indication that they represent a threat. Rather, this is the practice of routinely, pervasively and expansively collecting information from sources often considered as ‘open’, as there is no direct restriction on who can access the information.

One example that uses this type of data is the Mercury programme, which collects both open and secret information ‘to anticipate and/or detect significant events, including military and terrorist activities, political crises, and disease outbreaks’.²⁵ Another is the Early Model Based Event Recognition using Surrogates (EMBERS) programme, which gathers information from a variety of sources, including ‘Twitter's public API …, RSS news and blog feeds, Talkwalker alerts, NASA satellite meteorological data, Google Flu trends, Bloomberg financial news, TOR usage data, OpenTable's restaurants’ cancellation data, the PAHO health survey, and webpages [referencing] tweets' in order to ‘forecast population-level changes’.²⁶ EMBERS can mine ‘up to 2,000 messages a second and purchases open-source data such as Twitter's “firehose”, which streams hundreds of millions of real-time tweets a day’.²⁷

Other programmes use ‘image scraping’, scanning social media platforms such as Instagram and Facebook to collect people's physical images, alongside other identifying data and posted activity, to aid in facial-recognition operations and surveillance. This can include ‘face-matching’, which matches two or more faceprints to determine if they are the same person, using photographs of unknown people and linking them to their real identities gathered from a database. ‘Face-tracking’ involves following and identifying individuals via CCTV as they move through various spaces, for example, security services ‘might use [the] technology to follow an unidentified protester from a rally to their home or car, and then identify them with an address or license plate database’.²⁸ ‘Face analysis’ uses facial recognition ‘to try to guess a person's demographic traits, emotional state, and more, based on their facial features’, with some providers claiming to be able to ‘assign demographic attributes to their targets, including gender, race, ethnicity, sexual orientation, and age’. ‘Emotion analysis’ is similar and is claimed by some providers to be able to ‘determine how a person is feeling based on their face’.²⁹ One prominent provider in this space is Clearview AI, which offers a system that allows law-enforcement agencies to upload a photograph of a face and match it in a database of billions of images it has collected. It then provides links to where matching images appear online. Clearview AI has allegedly ‘harvested more than 20 billion images from the internet and social media to create its global facial recognition database’,³⁰ including from platforms such as Facebook, taken without user permission. Its customers include, in the UK, the Metropolitan Police, the Ministry of Defence and the National Crime Agency, and in the US, the Immigration and Customs Enforcement, the Department of Justice, the FBI and the Miami Police Department, which has confirmed that it uses Clearview AI across all crime types for face-matching purposes.³¹ Clearview AI is considered one of the most powerful and accurate facial-recognition tools in the world.³²

However, the fact that a platform is open-source does not automatically mean it is ethical to collect, process and analyse the information, as doing so can still violate people's privacy by using their information beyond the limits of how they have intended it to be used and without obtaining informed consent. According to one view, privacy can be conceptualized as existing as a collection of different property rights that people have in relation to themselves, information about themselves and information that is generated by their authorship or activity.³³ This cluster of rights includes positive rights, such as the right to sell or give the information away, as well as negative rights that prevent others from using, viewing, selling or damaging the information.³⁴ Even if looking at or using a person's information does not materially damage it or the individual, it still violates their right of control and privacy.³⁵ Moreover, these are strong rights of control since while ‘we have fairly stringent rights over our property, we have very much more stringent rights over our own persons’, which places a greater set of protections over information either about an individual or created by that individual.³⁶

For individuals who put their information online, this is still their information to control, including who has access to it and how it is used. It is an individual's right to allow some people to see or even use it. They do not necessarily waive all their control over that information by putting it on social media, especially in terms of intelligence collection and use. For an individual to waive such protective rights, they must consent to that exchange: they must willingly, knowingly and, without coercion, agree. Consent is ‘morally important because it expresses an agent's autonomous will’.³⁷ For an individual to truly consent, they must therefore have their autonomy intact, meaning that consent is both a manifestation and a reflection of that autonomy. The individual should therefore be able to direct their own will by basing their decisions on all the relevant information and by possessing the capacity and opportunity for self-reflection, with an understanding of the implications of their decisions.³⁸ The individual should have all the relevant knowledge available to them and should reflect on that knowledge, with the necessary capacity and the opportunity to (withhold) consent and be free from coercion. This means that people must have a realistic understanding of how their information is going to be used and of the implications (and harm) that such use can represent for themselves and others. If all these criteria are not satisfied, the individual has not waived their normal protective rights and collecting, processing or analysing the information is a violation of both their privacy and autonomy.³⁹

Importantly, it can be argued that the consent required for intelligence agencies to use such information should be informed rather than implied consent, given the potential for harm that intelligence can cause, whether to the targeted individual, to other individuals or to their social groups. It can be argued that, given the coercive power of the state and the threat to people's privacy, autonomy, liberty, and even physical and mental integrity, security and intelligence is an area of activity that carries with it a higher degree of threat and risk to people's well-being, not only to those targeted directly but to wider social groups as well.⁴⁰ Moreover, intelligence–AI represents an even greater threat to social groups, as it has been demonstrated across several studies and investigations that in practice these AI algorithms have resulted in more people of colour being arrested, jailed or physically harmed and that the same algorithms encourage intelligence–AI to become biased, discriminatory and promote unequal treatment.⁴¹

Harm is therefore caused not only by violating the privacy and autonomy of those whose data is collected, but also by the repercussions of the problematic analysis and intelligence practices promoted in terms of how they are used against others. The individual should be explicitly aware of how their information is used rather than assuming they are aware, making their consent akin to the types of consent seen in the fields of medicine and research.⁴² The expectation of informed consent means that individuals are not only notified, but that there is an indication that they have understood and reflected on the request made, and on the implications or consequences of their consent.⁴³ Moreover, there should be an alignment between the underlying intent of the consent and the resulting action: ‘if A consents to B's using his car and (without A's knowing it) B uses the car to carry out a bank robbery, it would ordinarily be misleading to say that A consented to B's use of the getaway vehicle’.⁴⁴ While A could have asked whether B was intending to use the vehicle for an illegal activity, the question is: how reasonable is it for A to expect to enquire as to the use of their item, especially as B is the one seeking the consent?

Across public social media pages, people see access to their data as being closer to a walled garden rather than an open field, visible to one's peers rather than society as a whole.⁴⁵ There is no expectation that the wider world should be able to access their online data, with research showing a particularly strong aversion to authority figures having access. Indeed, a 2018 Pew Research Center review of people's attitudes towards online privacy in the US revealed that ‘81% say they feel very or somewhat concerned with how companies use the data they collect about them. Fully 71% say the same regarding the government's use of data’. With 79 per cent of people not feeling in control of their data collected by the government. 77 per cent report that they do not understand what data is collected by the government. And ‘72% of Americans say they have little to no understanding about the laws and regulations that are currently in place to protect their data privacy’. In terms of social media specifically, 76 per cent have little or no trust in social media companies to use their data ethically and not sell it; and 70 per cent say they have little to no trust in companies to make responsible decisions about how they use AI in their products.⁴⁶ Indeed, in terms of online social media, even though there should be a greater awareness that others could access such information given its outward-looking nature, there is a discrepancy between the level of privacy people expect and the number of people who have access to their information.⁴⁷

Processing and analysis

In addition to ethical concerns raised by the collection of the data, intelligence–AI creates further issues through its processing and analysis of that data. As two distinct but intertwined parts of the intelligence cycle, the processing is the means through which the information is categorized, labelled, aggregated and presented, while the analysis phase is how that information is understood and given meaning. Machine learning enables ‘the interpretation and analysis of otherwise inaccessible patterns in large amounts of data’ and ‘may involve filtering, analysis of relationships between entities, or … sophisticated image- or voice-recognition’.⁴⁸ Machine learning can also build its own models based on the data provided, and then use these models to make inferences, allowing machines to determine for themselves what to look for in the data and ‘to learn and improve from their experience automatically without further programming’.⁴⁹ AI can carry out various types of analysis, including descriptive analytics—determining what happened in the past; diagnostic analytics—why it happened; predictive analytics—what will happen in the future; and prescriptive analytics to outline the optimal course of action.⁵⁰ Social network analysis examines how actors, entities or phenomena interrelate with each other and draws out meaning from those linkages; it can be used to draw maps of urban gangs, city-wide alert systems and crime-spot predictions.⁵¹ Text analysis is used to identify patterns in texts and recompose them into a useful summary, while NLP can draw out the underlying sentiment of the text and its author.⁵² For intelligence actors, given the exponential rate at which data is generated across the globe and cyberspace, AI offers a means of processing and analysing those large databases, with intelligence organizations citing the ability to leverage ‘artificial intelligence, automation and augmentation technologies to amplify the effectiveness of our workforce’ and as a result ‘advance mission capability’.⁵³

Cross-referencing and secret analysis

The access that intelligence actors have to citizens' data is particularly unique and powerful. State security actors are able to gain access to information stores across an individual's most personal fields, including their home life, physical location, finances, medical and health, online activity, purchase history and services such as water, internet and electricity providers. AI has the potential to condense diverse and large datasets automatically into digestible intelligence reports, or provide the intelligence analysts themselves with new searching tools that ‘extract knowledge from massive corpora of information [that] relates not just to words but acts and entities, the state of the world, and how they relate to each other’.⁵⁴ This processing and analytical power of AI, in conjunction with the reach of the intelligence community, means that it can analyse multiple databases in a quantity and at a processing speed previously unseen because it would have been impossible for either human operators or software.

This ability to meaningfully cross-reference different databases, however, can create additional privacy violations, as the AI can make statements about an individual that would not be knowable from any of the singular datasets. For example, studies have demonstrated that people can be re-identified from anonymous data using postcodes, date of birth and gender with 87 per cent accuracy.⁵⁵ This type of cross-referencing processing and analysis can then be used to determine individuals' personal and political features—which would normally be considered to be of a very intimate nature and which were not directly revealed in the singular databases. Such features might include sexual orientation, age, gender and religious or political views.⁵⁶ This represents a greater ultimate violation of people's privacy, and so results in harms that are greater than the individual parts. Moreover, this type of analysis has been shown to undermine efforts to protect or anonymize the data collected. As researchers at the Alan Turing Institute have written:

Simply removing primary keys (e.g. name, birthday, postcode, etc) from a database and replacing them with some pseudo-random numbers … doesn't work in general—due to the many diverse holders of records it's often possible to link data from different sources and infer who the subject is. … For example, [when] Netflix released a large collection of [viewer information], removing people's names and other identifying details … researchers were able [to] cross-reference the Netflix data with public review data on IMDB … and add names back into Netflix's supposedly anonymous database.⁵⁷

Furthermore, not only does this re-identification represent an increased privacy violation, but due to the inherently secretive nature of intelligence citizens are not aware that their information has been accessed—nor are they aware of the type of assumptions that are being made about them. People are not aware of how that information is being used (potentially against them), or if it is being used to cause harm to other people: importantly, they are not able to challenge such usage. This is a particular issue for intelligence–AI, as the inherent ‘black box’ problem of AI is exacerbated by the secret environment of intelligence, which in combination prevent internal and external interrogation, as well as preventing any real opportunity to convey to those affected how their information was used. The black box problem is a term used to describe the inability of humans to understand, in any relatable fashion, how an AI system came to its conclusions. This is because, while humans might set the parameters or input training materials, and can initially verify results, the AI is left to determine its own internal routes for reaching that result, which quickly becomes too complex for humans to understand.

In one study, ‘researchers analyzed a program designed to identify curtains in images and discovered that the AI algorithm first looked for a bed rather than a window, at which point it stopped searching the image’, and where it was ‘later learned that this was because most of the images in the training data set that featured curtains were bedrooms’.⁵⁸ The greater the volume of data and the more complex the processing or analysis required, the less knowable the means become. For example, Google's BERT NLP model consists of 110 million parameters, and even if these were to be examined, they would ‘not yield an understanding of the logic of the model’.⁵⁹ So, while analysts can ‘mathematically explain how algorithms optimize their objective functions, the complexity of the algorithms make it nearly impossible to describe the optimization in understandable and intuitive terms’.⁶⁰ The use of machine learning and its specialized subset of deep learning, with the latter's multiple levels of abstraction, increasingly obfuscate the means by which the decision is made, making the understanding impenetrable and necessarily reducing the accountability, understandability and traceability of the process as well as—in the long term—the predictability of the result.

More problematically, the black box problem will further exaggerate and be exaggerated by the distortive effect of the secretive nature of the intelligence field, as the use of AI will embed already existing biases through feedback loops, tech-washing (‘the process by which proponents of the outcomes can defend those outcomes as unbiased because they were derived from “math”’)⁶¹ will ratify the results as scientifically derived by a machine, and groupthink will limit interrogation. This is because secretive environments within security are problematic, creating an internal culture that isolates the intelligence community on the inside from the rest of society on the outside, reinforcing the need for greater isolation and hyper-secrecy. When overly secretive environments or cultures are placed above or outside the normal political sphere, they isolate their members and their structures, separating off those on the outside who are unaware and unable to engage, and preventing them from acting as a counterbalance and reference point to the internal cultures and escalation. Those on the inside are subjected to a process of in-group/out-group differentiation that dehumanizes ‘others’. When this is coupled with a lack of outside input, there is no means of measuring one's moral compass.⁶² As a result, officers learn to exclude those considered as outsiders from their universe of obligation.⁶³ Cognitive restructuring means violence or harm is redefined as honourable, for a greater abstract good, and becomes increasingly socially and morally acceptable to those inside.⁶⁴ Secretive environments normalize this process while also reinforcing both the need for greater secrecy and a lack of regard for the negative consequences for those on the outside. In such an environment internal criticism is limited, as it is seen as a betrayal to the group, and so restricts alternative analysis as the group mentality smothers dissenting points of view.⁶⁵

Secrecy does play an ethically important role in enabling intelligence organizations to perform their role, and this is not a call for complete transparency. Nor is the resulting behaviour the product of some nefarious intent, but a natural result of the divisions and isolation that the secrecy itself creates. The secretive and isolated culture does not empower the required critical inquiry of a system such as intelligence–AI, which is itself inherently closed off and difficult to unravel. The secretive nature of intelligence means that individuals are unaware of what types of conclusion are being made, and there is no opportunity for individuals to appeal or challenge these assumptions, or the methodology on which the assumptions are based. Those who have suffered harm might not even be aware of how they are being categorized or the conclusions being drawn about either them or their social group. This unequal treatment is, for intelligence–AI, beyond using poor sets of training data, but becomes more deeply rooted as well as more difficult to avoid, detect and remedy because of how AI and secretive intelligence interact.

Social sorting

A key underlying problem with AI is that it relies on digitizing, quantifying and aggregating data in order to turn it into an analysable and usable product. One of the selling points of intelligence–AI is that it can take data from unmanageably large and diverse datasets and create user-friendly means of searching and presenting the information. This process relies, however, on the AI's ability to format the data—whether it consists in biometric details, online activity, video or photographs of people's faces, gait, behaviour or transcripts of text—into a numerical value and to attach this digital output to some socially derived label. Complex social and physical activities are therefore necessarily reduced in order that they can be quantified. This sorting and processing rely on the allocation of labels and then the aggregating, grouping, separating and further labelling of social attributes through repeated runs to take the vast quantity of data collected and turn it into something usable.

This is problematic, in the first instance because such digital systems rely on being able to allocate a fixed numerical data point to create something which is analysable. While this process might not necessarily be a problem with discrete or factual data such as addresses, age or some types of biometric information, it becomes more problematic for socially constructed or socially dependent information, most notably in cases involving race, gender, identity, behaviour, culture, attitudes and beliefs. As a process, this necessarily creates categories and divisions within social phenomena that are complex, fluid, context-relevant and contested, representing a modern panoptic sorting process that describes a system of categorizing the population to a hitherto unimagined degree.⁶⁶ Simone Browne has argued that through this type of processing the visual human body is classified, and by doing so ‘race, as a constructed category, is defined and made visible’,⁶⁷ reducing and classifying people ‘into sets of logical manipulatable signs’, which has been ‘a hallmark of racializing scientific and administrative techniques going back several hundred years’.⁶⁸ Marta Maria Maldonado and Adela Licona describe this process of racialization (by categorization/facial recognition) as ‘the production, reproduction of and contest over racial meanings and the social structures in which such meanings become embedded’,⁶⁹ articulating a connection between systems of racial oppression and quantification and driven by a logic of conceptualization that is concerned with arbitrarily dividing human populations.⁷⁰ ‘The mechanistic gaze of facial recognition’, for example, ‘consists solely of the extraction and abstraction of … personal features from what are essentially statistical images’.⁷¹

Second, this reduced data is then allocated to a label, which is an inherently political, predefined understanding of the complex social world carrying with it inherent assumptions, biases and context. These simplified labels are overly reductive—man/woman, white/non-white, threat/non-threat—and serve to perpetuate the divisions or binaries that the labels themselves inhabit, attributing a particular meaning and applying the characteristics to these arbitrary and simplistic divisions.⁷² Facial recognition, for example, can involve identifying an individual's social attribute from their physical appearance, including their race, ethnicity, gender, class, social behaviour, attitude or sexual orientation. This processing is then made more problematic as the intelligence–AI then attributes a particular meaning to those classifications. It is not only that the target is quantified under some predefined label, but that this is given status or meaning through its connection to other data points and how they are themselves categorized by the AI, which is then used and exported onto another individual in other scenarios. For example, ‘facial recognition involves identifying, extracting, and selecting contrasting patterns in an image, and then classifying and comparing them to a previously compiled database of other patterns’.⁷³ Those who fall under this label are then treated as the label dictates, rather than as the individuals they are. As Spiros Simitis argues, a profiled individual is ‘necessarily labeled and henceforth seen as a member of a group, the peculiar features of which are assumed to constitute [their] personal characteristics’.⁷⁴ This practice of intelligence–AI ends up ‘conflating biological characteristics with social attributes’ as it formalizes ‘phenotypic differences’ with the tendency ‘seemingly irresistibly, to spill over into claims about genetic capabilities and aptitudes’.⁷⁵ Furthermore, labelling people in this way can cause a tendency to create self-fulfilling prophecies, when individuals feel they must act to meet certain expectations of them because of how they are treated.⁷⁶

Application: the coercive state

This intelligence–AI processing, analysis and attaching of meaning to the data leads to a set of practices, beliefs and positions that encourage, rationalize and exacerbate existing social biases by embedding feedback loops, promoting self-fulfilling prophecies and falling foul of tech-washing. Intelligence–AI shapes security practice, and in doing so manifests the biases seen at the processing and analysis phases, taking the characteristics of one target and overlaying them over their group to identify and classify suspect populations.⁷⁷ Singular attributes can be used to create a profiling that locates pre-threats even though other individuals do not have any of the other ‘threatening’ attributes seen in the original offender.⁷⁸ Simply having similar patterns to previous threats is not sufficient to count as a legitimate reason for targeting someone, and is more about guilt by proximity rather than representing some form of threat. This is indicative of a larger move in security towards pre-emptive risk assessment, as security is ‘not based on individualised suspicion, but on the probability that an individual might be an offender’.⁷⁹

The trend described above is problematic, as it has been reported that the AI algorithms overly predict individuals from certain racial groups, or from particular neighbourhoods, as a ‘proxy variable’ for race. One investigation demonstrated that the predictive technology used by the policing software PredPol disproportionately predicted threats would be committed in neighbourhoods by working-class people, people of colour and Black people in particular.⁸⁰ Moreover, even if the data on which the AI was trained was made less biased, the system would still face a feedback issue where the focus of resources in any given area created a self-fulfilling prophecy, while tech-washing means that the outcomes are assumed correct.⁸¹ Intelligence–AI will therefore not only miss problems from unmonitored or emerging arenas, but will reinforce the idea that a particular group or area is the right target for intelligence surveillance and interventions.

These intelligence–AI decisions and conclusions are then given coercive weight and force, receiving official recognition from the state and its security apparatus. This results in security activity that can represent both a material and a symbolic threat to people's physical well-being, privacy, autonomy and liberty. This can include incorrect, disproportionate, biased or arbitrary application of the direct force of the state, such as the power to arrest and detain, physical attacks, punishment, control, intimidation, subjugation, coercion, agenda-setting, domination, marginalization or the maintenance of existing power dynamics. It can also include symbolic threats, often referred to as the ‘chilling’ effect of state surveillance, which can alter how people decide to act or limit their options, thus violating their autonomy. If, at the core of autonomy, there is the capacity to make decisions freely, without undue influence or control, then distorting influences both overtly and covertly violate an individual's autonomy. This includes cases where an individual's decision-making process is altered through a self-imposed pressure caused by how people think others are viewing, evaluating and possibly intervening in their life, a force which is increased when it comes from coercive actors such as the state and its security apparatus.⁸²

Indeed, given intelligence–AI's reach and ability to transcend space and time limitations and bring the asymmetric gaze of the panopticon across social spaces, both online and in the physical world, it can detrimentally affect an individual's autonomy as they start to ‘self-discipline’ their actions and surrender to the wishes of the observers as the individual ‘becomes the principle of his own subjection’.⁸³ Even when the impact of intelligence–AI is not felt materially, its increasing use can establish social and political environments that then threaten people's (political) autonomy as a result of the new types of surveillance AI enables, creating direct implications for the individual's (and that individual's social group's) liberty, choices, opportunities and social acceptance. The power of the panoptic gaze means that people do not have to experience direct intervention such as physical detainment; however, the sensation of being monitored is sufficient to influence their decision-making.

Moreover, this can threaten people as politically autonomous agents, as maintaining one's autonomy forms the foundation of a bundle of civil rights including freedoms of expression, association and political engagement. Political autonomy means being able to create one's own ideas and to share, shape and critically reflect on them through access to other individuals and other information collections; to engage with the wider political community; and to use this autonomy to review and check on political authority.⁸⁴ Intelligence–AI's panoptic gaze focuses on political spaces, both in cyberspace and the real world, and so its (perceived) surveillance and intervention affect how people decide to act as political agents. This can include monitoring what information people share and consume, how they congregate and travel, and how they carry out political activity such as protesting. For example, Israeli security forces' use of intelligence–AI to scrape data from social media in 2017 resulted in the arrest of 300 Palestinians in the West Bank and East Jerusalem on charges related to posts on Facebook.⁸⁵ The intelligence–AI analysis is using social profiling to classify and securitize people's political expression.

Conclusion

Intelligence–AI has the potential to increase the speed, reach and analytical power of intelligence actors in ways that are still to be fully realized. However, intelligence–AI also needs to be placed under a more critical lens than other forms of AI development. Intelligence has a reach unlike any other organization, and AI now offers a realistic and timely way to process and analyse that information, solving the bottleneck problem and driving ever further a mandate to collect more information. The concern over the processing carried out by AI is not just limited to how accurate the system can be made, as being more accurate does not undo the problem of making arbitrary, politically and socially contingent conclusions about people, which become unchallengeable due to the secretive nature of the intelligence community and the impenetrable nature of AI. Conclusions are then given force, as they shape and inform security responses which—without the ability to interrogate its use, can unknowingly distort intelligence policy application, promoting distrust between individuals and the state as well as between different social groups, having real repercussions for individuals in terms of social mobility and treatment. In terms of intelligence, this can result in harsher tactics, including escalating interrogation techniques, increasingly intrusive collection methods or unequal treatment based on race or ethnicity.

This analysis therefore has important implications for how intelligence–AI should be understood and practised. However, the existing guidelines, principles, regulations and agreements are not sufficient to tackle the unique ethical concerns that intelligence–AI creates. Not only are they often vague, abstract and lacking in a usable process to direct ethical behaviour, but in the case of intelligence they are insufficient to tackle the challenges raised by its coercive power, secretive environment and the lack of understanding or consent from the general population. Looking forward, therefore, there is a need to develop a specialized ethical framework that provides practical guidance on how AI technology should be used and how the population should be informed. This includes creating a more explicit set of targeting guidelines on what data should be collected and from whom; the types of analysis allowed, moving away from making determinations based on characteristics such as race, religion, location, ethnicity, gender and physical appearance; and introducing post-bellum mechanisms to explicitly inform people on how their data has been used.

Footnotes