On 7 October 2023, a video surfaced on Facebook showing a woman taken hostage during the Hamas-led attacks in Israel. She was pleading with her kidnappers not to kill her. The content of this video was undeniably atrocious. To the platform’s automated systems, the post was a clear violation of the ‘Dangerous Organisations and Individuals’ policy. It was flagged and initially removed. But to the human eye, the video’s atrocity was precisely why it was essential: it was a crucial, albeit disturbing, document of a war crime and a visceral plea for survival that demanded to be witnessed.
Witnessing, in this context, is a necessary act of accountability. In an era where narrative warfare and digital disinformation obscure the reality of conflict, such raw footage serves as an irreducible record of reality – essential for future legal investigations, for the collective memory of a nation, and for the world to grasp the human cost of a crisis. By deleting the footage to satisfy a safety protocol, the platform did not just remove content. Rather, it effectively digitally cleansed a crime scene, prioritising algorithmic order over the preservation of historical truth.
This is the central collision of our digital era: a binary governance system – one that operates on a reductive logic where content is either ‘safe’ or ‘unsafe’ – designed to sanitise the internet is fundamentally at odds with a nuanced, human reality that requires us to bear witness to the unthinkable. As Benjamin Labutut describes in his novel The MANIAC (2023), we are increasingly confronted by ‘a profoundly inhuman form of intelligence’ that is indifferent to the needs of humans. The ‘machine’ – the rigid, binary infrastructure of digital governance – that governs today conflicts with the ‘ghost’, the irreducible, messy reality of human context, intent, and suffering.
While the machine seeks to order the world through algorithms and rules, the ghost constantly escapes these boundaries. This is exemplified by the plethora of online content that is legally permissible but widely considered to be harmful or offensive, a category best described by Mark Little and Simon Chesterman as ‘lawful but awful’. From graphic violence to misinformation and borderline hate speech, this content exposes the limitations of online platforms in the new era of digital technology.
To understand why the system failed the woman hostage in the video, we must first recognise the scale of the machine. We often refer to Meta or X as ‘platforms’, holding an implication of neutrality, but this vocabulary is long outdated. As Kate Klonick argues in ‘The People, Rules, and Processes Governing Online Speech’ published in 2018 in the Harvard Law Review, these entities have effectively become the ‘New Governors’ of online speech, functioning less like traditional companies and more like a private administrative state. They are complete with internal constitutions, an enforcement bureaucracy, and appeals mechanisms.
Moreover, as Karen Hao suggests in Empire of AI (2025), these platforms-turned-states are distinctly imperial. Like the imperial powers of the nineteenth century, they govern a vast and heterogenous periphery from a distant and centralised core in Silicon Valley, relying on a rigid legal code to maintain order. At Facebook, for instance, the company’s internal rulebook has grown from a single page to a 15,000–word document. The explicit aim of these codes was to overcome cultural biases in favour of objective rules, ensuring that a moderator in Oxford makes the same decision as one in Mumbai. Yet, this ‘benevolent paternalism’ is built on the backs of a digital proletariat. The AI that governs our speech is sustained by thousands of human moderators in the Global South, from Manila to Nairobi. These workers, often outsourced by third-party vendors and poorly paid, endure severe psychological trauma from constant exposure to horrific content that keeps the digital public sphere sanitised.
Nevertheless, despite the clear ethical crisis occurring in the digital world, the system remains committed to algorithmic absolutism, the belief that complex human realities can be solved by enforcing a binary classification. In its pursuit to strip away subjectivity, the machine strips away the human ghost that makes speech meaningful. This algorithmic absolutism is precisely why the system fails.
The failure of the machine is most acute in the ‘lawful but awful’ grey area, where content technically complies with frameworks for content moderation but nonetheless corrodes the social fabric. The algorithm excels at detecting explicit violations (nudity, specific terrorist symbols) but is blind to the subtleties of human communication, such as ‘dog whistles’ or local political intent.
We see the machine’s lack of subtlety clearly in the American political sphere with figures such as Charlie Kirk, whose assassination last year was much discussed around the world. Kirk sparked significant controversy at a Turning Point USA event in December 2023 by stating that the Civil Rights Act of 1964 was a ‘huge mistake’ that created a ‘beast’ of anti-white bureaucracy. In doing so, Kirk skirted the edges of the rules, offering content that the machine reads as high-performing political discourse but the ghost recognises as a dog whistle for white-nationalist narratives that seek to delegitimise the legal foundations of equality. The machine, lacking the cultural sensitivity to hear the whistle, amplifies the content as ‘engaging’. In doing so, the system effectively perpetuates extremism into mainstream discourse because it cannot distinguish between policy debate and racial provocation.
The consequences of this context blindness are even more lethal in the Global South. Consider the role of Facebook in Myanmar during the Rohingya crisis, the ongoing persecution and displacement of a stateless Muslim minority from Myanmar’s Rakhine state. Here, the platform’s engagement algorithms amplified ‘lawful’ political speech (nationalist posts and local rumours) that the platform translated as high-engagement content. Perpetrators of the incitement easily bypassed explicit bans by using nationalist rhetoric or local slurs that Western-trained algorithms and non-local moderators failed to flag as dangerous. Yet in the specific context of Myanmar, these posts were clear incitements of ethnic cleansing. Because the machine lacked the cultural context and the Burmese-speaking moderators to distinguish between patriotism and propaganda, it ultimately accelerated a genocide.
If our reliance on online platforms is to continue, and it inevitably will, the discourse it provides must not lose touch with our humanity. We cannot simply rely on better code or algorithms. Instead, we require a layered system that counters the invisible manipulations of the machine with visible human rights. The urgency of this shift is underscored by the silent threat of ‘shadow banning’, content moderation sanctions that are undetectable to the affected user. Research by Tauhid Zaman and Yen-Shao Chen at Yale has demonstrated that platforms can effectively shift public opinion and increase polarisation simply by selectively demoting content. Because online platforms wield opinion-shaping power under the guise of neutral relevance-ranking, they can quietly suppress topics by subtly nudging the algorithm or applying discreet, unannounced demotions. This affords platforms the power to shape public discourse without taking any accountability by shielding their decisions behind the opacity of content policies. From a user standpoint, this practice can mute dissenting voices and controversial opinions, fundamentally obscuring the distinction between content moderation and content curation in a potentially dangerous manner for the digital public sphere. All of this is to say, we are not citizens of a public sphere but rather subjects of a black box, manipulated by a system that we cannot see or question.
In order to dismantle this black box, we must move beyond the binary of ‘keep up’ or ‘’take down’ and require fundamental changes to the governance of online platforms. First, we must fundamentally alter the legal relationship between the user and the platform. Currently, we live in an age of what Yanis Varoufakis calls ‘digital feudalism’ where users are serfs subject to the arbitrary terms of service of online platforms. To transition from subjects to citizens, legal scholars Niva Elkin-Koren, Maayan Perel, and Giobanni De Gregorio propose a shift to ‘contractual networks’. Users inherently create value for each other, not just the platform; as such, contracts should recognise users as stakeholders with rights. Under this model, users would transition from data subjects to digital citizens with enforceable rights to due process (the right to know why you were banned) and procedural justice. If the machine silences or shadow bans a user, it would owe them not just a notification but a reason, thereby breaking the ‘black box’ of algorithmic governance.
Secondly, we must abandon the ‘move fast’ ethos that currently defines Silicon Valley in favour of guardrails. In Guardrails: Guiding Human Decisions in the Age of AI (2024), Urs Gasser and Viktor Mayer-Schönberger argue that reintroducing friction in the system should be an intentional design choice. Instead of maximising speed, the system should be designed to slow down when it detects uncertainty. High-stake content (like hostage videos and other lawful but awful content) should trigger a ‘human-in-the-loop’ pause where the guardrail routes the decision to a human context layer rather than instant automated deletion. The solution is ultimately not to exorcise the ghost of human judgement from the machine, but to institutionalise it.
Today, the failure of our digital institutions is driving us backwards. When the public sphere is governed by an opaque and arbitrary algorithm, trust collapses. Users resultantly retreat into digital ‘clans’, echo chambers where dog whistles and a lack of cultural sensitivity dominate public discourse. To reverse this trend, especially in an era where online platforms have assumed vast amounts of power in regulating user expression, we must build digital institutions capable of recognising human sentiment. As Steven D. Smith warns in The Disintegrating Conscience (2023), a society without a shared moral vocabulary cannot govern itself. Ultimately, we cannot code our way out of this. We must build institutions that protect the ghost – the human capacity for judgment, mercy, and context – within the digital machine.
Tiffany You is a third year PPE student and Organ Scholar at the University of Oxford. She has a strong background in student journalism, having written for The Oxford Blue and The Oxford Student. In addition to her editorial work, she currently serves as the Entz President for her college’s JCR and Outreach Director at Oxford Women in Business.