COMPUTING HUMAN OVERS[A]IGHT: LAW/APPARATUS\VISION/AGENCY

The point of departure of the Computing Human Overs(a)ight critique is the notion of human oversight enacted by the EU law regulating AI – The European Artificial Intelligence Act, Article 14. We argue that computer vision-based systems, understood by automated processing in real-time, render human oversight in high-risk contexts as declared by the EU AI Act useless. This legal provision refers to “human-machine interfaces”, “real-time control”, “natural person” and even a “stop button” to bring the system to a halt. Despite invoking the human, these mechanisms displace the people's agency and political action by making oversight subject to the AI systems infrastructures, protocols, and automation. Human oversight is the currency that the legislator asks society to pay in exchange for the possibility of having high-risk AI systems. As such, it lies at the heart of the question of control, subjugation of machines to humans, or put differently: what is left for us, humans, now that AI is there and everywhere?

In the promise of a world computed by machinic perception, Computing Human Overs(a)ight investigates the tensions and paradoxes of human oversight in applications of high-risk AI-based systems as declared in the EU AI Act, Article 14, that has already been set into action. In the expansion of the global infrastructure —as Tech is accelerating from Big to Bigger— the decade behind us has been marked by the growing pressure that looms over our societal capacities to adopt and adapt to all emergent technological hopes and hypes. Guided by the law as a framework for our critique we shape an argument that law reveals the skeleton of a society, its commitments, preferences and predilections and, also, its vices. The law is formulated and placed into action once a certain set of properties, behaviours and regulations are enacted in the function of society, therefore it can help us navigate current understanding of human-machine operations, mainly in the scope of computer vision, not only from a technical, but from a social, ethical and legal level, questioning human agency in the automated apparatus.

From a legal perspective, we take inspiration from Robert Cover's opening of Nomos and Narratives, “We inhabit a nomos — a normative universe. We constantly create and maintain a world of right and wrong, of lawful and unlawful, of valid and void” [6] in conjunction with Henri Bergson's “this aggregate [ensemble] of images is what I call the universe” [3]. As a result, our view of legality is summarized in the following sentence: we construct a normative universe, which is an aggregate of images. This is in line with the idea that the legal system is first a system of images and then a system of rules, put forward by Peter Goodrich [12].Thus, the question is how do we construct a legality when there is no image to see, how do we exercise human oversight, and create and maintain a world of lawful and unlawful in AI systems from computational images that do not let them be seen by human eyes?

An image in this critique is placed on a threshold between the observational and operational – what the oversighter sees on the representational layer of an image is a surface beneath which many computational processes emerge. While the legislator's view of the image and vision refers to is a representation of the rich concrete phenomena that makes our social life, the computational images that AI systems operate on are abstractions that do no longer refer to phenomena itself, but an aggregate of data layers. A computational image is not a photographic proof – it is a data image. In processing of such an image, the data is rendered through layers of machinic calculations, becoming an operative substance for automated decision-making processes, rather than serving as a legal testimonial upon which a human operator tends to deliver its judgement. By the time the human gets to decide and understand what is wrong or right in the content projected on a visual interface, there was already an irreversible information encoding protocol [11] that took place in computational processing, creating a web of values and meanings [that are by its nature not complementary with human logic of causality].

Building on a tradition of image studies from media theory and arts; and critical legal theory [12] [32,36], we work with concepts of technical image [10], invisuality [20], operative image [28], through this critique to human oversight we address the imaginaries of the EU legislator exposing the [im]possibility of human oversight. At the bottom of the issue lies the collision between the will of the law and the operations of the AI systems, the constitutive tension between their ideologies and materialities. More broadly we address the social impact of computation under the notion of AI; marking the current state of human agency in the automation of decision-making, labour, law, and governance, and by providing insight into the computational, legal and artistic examination of human oversight in high-risk AI systems. After introducing the frameworks of the law-in-action, we discuss the ontology and teleology of an apparatus for human oversight. A formation of a fixed place is an illusion of control, with its function and aesthetic mainly symbolic, rather than operative.

This introduction is followed by four sections that critique human oversight of AI systems and the possibility to construct legality from the perspective of the law, apparatus, vision, and agency. The critical analysis of the legal provisions reveals the tension between the de jure subject of oversight, the natural person, and the de facto subject of oversight—the platform, as provider and deployer of AI systems. The apparatus section argues how the mechanism of oversight came to matter, by removing the human excess from the machine-based systems, while furnishing this role with concepts from cybernetics, as shown in the Cybersin project and the concept of an Operations room. The vision section highlights that the computational images to be seen are not produced for human sight but for the operations of the AI systems, the machinic vision. As such, oversight when the image is invisual to the human eye, is a form of machinic/platform seeing over itself. The closing section, questions the agency humans have on constructing a meaningful present and future when oversight is limited by predictive and preemptive politics of AI systems, forming a techno-solutionist self-fulfilling prophecy through re-engineering of society at a global scale.

Through this framework, we decode and deconstruct human oversight under notions of contemporary algorithmic culture [29,30] and relate it to psychological, perceptual and cognitive shifts in the visual culture [2] and artistic practices of representation [8,35] and socio-demographic concerns and consequences of automation [23] that have accelerated over the last decade, due to the global distribution and adoption of the tools, models and products under the roof-term of Artificial Intelligence. Against a background of the public branding, as mystification or commodification of these tools, and the actual technological developments and systemic implementation on a global scale, —confusion and dissonance— the understanding of how, why, and for whom these systems work comes into urgent question.

Political determination and social challenges are conjoined with the technological environment, and in such a constellation, it is necessary to disambiguate the distribution of responsibilities, and the notion of agency between the human [cognition] and the automated [systems]. In the public discourse, there are frequent instances of praising automated systems and algorithmic data processing as a form of intelligence, which obfuscates the purpose and the limits of the implementation of these algorithms and tools for different systems and industries. This perspective is one of the key motifs in this critique - to differentiate the operations of AI as a technological cluster of tools and models, hardware and software components, and AI as a discourse, concept and, nevertheless, ideology. Such understanding of AI is central for us, so to be able to [critically] compute human oversight in all its [statistical] probabilities.

“Taking contemporary technical images as a starting point, we find two divergent trends. One moves toward a centrally programmed, totalitarian society of image receivers and image administrators, the other toward a dialogic, telematic society of image producers and image collectors.” [10]

A byproduct of the apparatus is the creation of a subject —human oversight creates a subject called the natural person who is awarded the capacity of overseeing, but cannot see. This is explained in the tension between the de jure subject and the de facto subject [36]. The trick here is that while the natural person as de jure subject oversees an object, an AI system, the natural person is, in turn, being seen by the AI system, provider, and deployer as a de-facto subject of oversight – who really sees. This also raises the question of boundary making of the subject first, but then within the de jure subject, the natural person. Who is granted with the capacity to oversee? As it is now, loyal to its commitments and predilections to corporations, while the provider gets to design and develop the “technical” part of the apparatus, the deployer gets to decide who can see. This is a duty given, by the law to deployer, that “shall assign human oversight to natural persons who have the necessary competence, training and authority” (Article 26). This is just another form of corporate bureaucracy [13], a transferring oversight in the ways the platform sees [20], its infrastructures and procedures for image production and collection.

The European legislator gives the power to decide who can see to the deployer, be it a self-interested profit-driven corporation or a public organization at risk of clientelism and authoritarianism. The EU AI Act leaves us with the question: can the individual citizen see? And if yes, from where? Does the individual citizen still see/over, or sees from below, or from within? The way the apparatus is constructed by the law, it is a fictional game between fictional players called provider and deployer, leaving the human out—the undesirable influence that exceeds the secured space for AI to function. For an apparatus to effectively manage and order “those qualities resistant to systematization must be stricken from the record. This subtraction process, this shaving off the “excess,” is necessary for the apparatus to function.” [16]

The first excess that the EU AI Act removes is the humane excess, by defining AI systems as machine-based, making them discrete, manageable, and governable [39]. By removing this excess, the apparatus not only displaces the human by the provider-deployer corporations, but it also turns oversight from a matter of law, ethics, and politics into a matter of infrastructures, procedures, and automation.

There have been extensive efforts to make the human entanglement with AI systems visible—through physical and intellectual labour [7] consumption, and most importantly though being the [crowd]source of collective intelligence that AI attempts to mimic [30]. However, for the apparatus to work that excess needs to be left out, excluded, and indiscernible. This is not trivial, at the centre of the question of human oversight is the answer to what is left to us now that AI is there. The role assigned to people, now that AI is there, is overseeing – while we know AI is not there, the ideological blindness of the legislator assures us that it is there.

At the centre of this apparatus, there is the constitutive tension between what is revealed and what is concealed and this same tension comes for the law. Human oversight can be understood as “an act of concealment in which, once again, a performance mode is called upon. [...] a ‘counter-performance within performance’.” [16] While the governance of AI systems, designed and developed by the provider and the deployer is in the hands of these corporation-platforms, the law produces a symbolic organ for human oversight that, in practice, is corporate oversight on themselves. As such, the apparatus is born out of the tension between ideology and materiality. Now the trick of the apparatus is to renegade from its ideological kinship, the contestability of its values, and relate itself to the technique, the factual and machinic operations of science. Far from God, close to the server.

“Yet what if it is not a human eye, but the inhuman, digital and rhizomatic eye of the web that contemplates images?” [31]

“Algorithmic productive force avoids causality, evades accountability, and restricts agency to participation and adaptation.” [22]

In the Eye of the Master, Matteo Pasquinelli [30] reminds us that —similar to the theory of division of labour in the process of industrialization— AI is a hypermimicry of collective intelligence, such as labour–robots in factories that do not reinvent the arts of the chain, they just become metallic versions of the arms of the labourers. The idea of human oversight is the last death rattle of reason on its way out, a last attempt to keep the show on, the reasonable man at the cusp of an order falling apart into automated thoughtlessness (as per McQuillan).

It is not only labour, but any formation of algorithmic protocol that involves all sorts of practices and refined ways of doing things, achieving something, transforming the environment, relating to each other, which have been under a process of sophistication for thousands of years [29]. Not only that, but it also removes that collective knowledge from the public domain, privatizing it by turning it into algorithms, scripts, and codes that can be controlled, while they control the human co-labourer, those who remain part of the contemporary division of labour.

While human/over/sight apparatus is pointing at AI systems, specifically high-risk AI systems, following Pasquinelli, what we are seeing at the end of the chain is an entangled human-machine labour. In a way, it works as a remediation of the control of labour but from a distance and at scale. In this term seeing from afar, can be understood as the fantasy of remote sensing labour, remote sensing the other. This brings the owner of capital the possibility to continue to extract the surplus, without having to be close to those producing it—perhaps they can do it from Mars (but that's another fetishism). In understanding such attempts, Yarden Katz states that the misunderstanding of what AI does, reshuffles power and accountability from democratic institutions to AI systems and models, from politicians to engineers and entrepreneurs, presenting the latter as the big architects of society [18].

Furthermore, the algorithmic protocol as a seemingly decentralised, zero-agent, depersonalized power structure, imposes an extractivist method that renders global-scale data for profit —whereas intelligence comes as a collective effort reclaimed and appropriated by the AI entrepreneurs— that is not erasing, but displacing human labour and agency. Meanwhile, the agent and the power are allocated to the algorithm and its proprietors.

In the context of AI ideology and politics, we create the future by statistical determinism. McQuillan argues that ‘no alternative futures’ discourse is embedded in such a system, where “AI's solutionism selects some futures while making others impossible to even imagine.” [24]. As such, we have left the realm of history driven by political will and trapped ourselves in a statistical evolution—attuning to the techno-material conditions of the digital [2]. This also reinforces the statistical evaluation of possible risks, that an AI system not only predicts, it, in fact, overdetermines its foreseeable future [24]. It is the perfect calculative oracle for preemption. Having its deterministic functions in hyper-accelerated data processing, it leaves little space for negotiation of its social consequences, if any.

The intertwinement of technology with the political agenda becomes a totality of technocratic rule under a neoliberal disguise. It is reshaping social relations, governance, surveillance, dismantling institutional power, and obfuscating human agency. In the preemptive politics of AI, a predictive system maintains its political legitimacy by taking the promise of predicting the future while at the same time creating it, whereas data science serves us to find the patterns where we want to see them. In reference to McQuillan's understanding of the technology of anticipation and preemption, we also note Massumi's remarks on the logical disjunction and regress of preemptive politics where “a logical gap opens in the present through which the reality of threat slips to rejoin its deferral to the future.” [21]. Every AI system, as an apparatus allows its proprietors to reshape the meaning of our reality, to prepare the message for the future and to put a “threat [that] makes a runaround through the present back toward its self-causing futurity.” [21]

In the techno-solutionist worldview, in the promise of the future, we are recycling the past. The promise of understanding the future, and therefore preempting it, holds the political power that allows it to affect and conceive the future that it is trying to preempt [21]. The human factor is there to pick up the mess left behind (accident of human consequence), while stumbling after the progressive accelerated calculus. The human individual as the worker is entitled to hold the responsibility for systemic overlooking. One of the most notable spins on accountability and agency in the displacement of power is the obfuscation of responsibility in predictive, preemptive engineering of social dynamics and automated platform-based geopolitical control.

In this last section on agency, we argue that while the ideal provision of the law is that the human remains in control of AI systems, its capacity to affect the course of action of these systems is limited. The preemptive quality of automated systems excludes the human agent from understanding, participating and therefore shaping their social or political future. As such, the possibility of constructing a meaningful legality has to do with the formation of platform operations that can involve social negotiations, and reduce automated frictions by forming space for underrepresented and invisible nuances of social interactions. Behind the thirst for accuracy in their predictions, AI systems override any possibility of external disturbances, the human excess in the path set ahead, in fulfilling their own prophecies.

Algorithmic computational systems have no space for social negotiations, they predict, classify, categorise, approve or ban. They also impose their self-optimisation over civil society as an aggregate for automation. Paradoxically, human oversight is a mere addendum to automated infrastructures, protocols, and algorithms. True power is embedded in the operative system. However, this bureaucratic role in the apparatus differs from the rest of the computational chain because it is the only role recognised by the law. The law does not recognise an apparatus, or a system as an agent of decision-making, so there must be a human agent to be held responsible in the eyes of the law, by the EU AI Act. The Act pretends that the human agent ensures control over the system, yet the human fills a symbolic role, in a form and purpose to legitimize the functioning of the apparatus of a high-risk AI system.

An apparatus of oversight, such as a control or operations room, holds access to observation but denies the power to act on actual situations. It is not a view from above; it is a displaced centre of physical surveillance. It is a procedural ritual, routine, a notion of a fixed [human] agent in a specific place and time, being a part of the system, as a ceremonial placeholder with a specific duty in the data processing protocol. Even though systems do not have any cognitive capacity to replace humans in the loop, the transposition of technologies, from their purpose as a tool to their image as a solution, in the public discourse gives liberty to promise a life-improving asset while private capital is generating all substantial power and impact on global societies by endorsing a politics of preemption.

The human-oversighter is neither a negotiator on behalf of humanity. Their role comes with a set of rules and regulations that they must comply with, so their agency is limited to a specific set of instrumentalised decisions, protocolised actions, as well as programmed interactions. They see the display surface of the unseen, invisual computational operations, automated decision-making processes fuelled by statistical determinism, and inaccessible to social negotiation. Their agency within the system is reduced to a set of instructions, but their responsibility is extended into the legal realm. The human worker is also prompted and solicited by the system; their agency is their duty to preserve the image of a safe system rather than providing any notion of safety to the subjects whose data is being processed in that same system. Overall, their actual duty, or the effect of their labour might be to serve the servers.