The Seduction of Certainty: AI, Criminal Justice and the Risk of Automated Trust

Artificial Intelligence is rapidly becoming the answer to one of criminal justice's biggest problems.

Investigators are drowning in digital evidence. Mobile phones contain years of communications and location data. CCTV systems record around the clock. Social media, cloud platforms and connected devices including vehicle telemetry, generate information on a scale no human workforce can realistically review.

In many cases, the challenge is no longer collecting evidence but finding what matters.

AI promises to solve that problem.

It can transcribe interviews, translate communications, classify images, identify patterns, prioritise lines of enquiry and help investigators navigate datasets that would be impossible to review manually. Used appropriately, these capabilities have the potential to improve efficiency, reduce delay and focus scarce human expertise where it adds the greatest value.

The opportunity is real.

But criminal justice should approach AI with both ambition and caution. Recent history offers a warning.

The Horizon scandal¹ demonstrated the consequences of institutional overconfidence in complex technology. Computer-generated evidence was repeatedly treated as inherently reliable, while legitimate questions about accuracy, error rates, governance and quality assurance were overlooked. The result was one of the most serious miscarriages of justice in modern British history.

The Danish telecoms scandal² revealed a similar failure. For years, telecommunications data was presented in criminal proceedings as objective and dependable. Only later did fundamental flaws in the collection, processing and interpretation of that data emerge, forcing the review of thousands of criminal cases and undermining confidence in evidence that had previously been treated as authoritative.

The lesson from both scandals is not that technology should be rejected. It is that technology should never be exempt from scrutiny.

The risk posed by AI is not that it will suddenly replace human decision-making. The greater danger is that it will subtly shape it.

When an AI system flags a suspect, identifies a pattern, prioritises evidence, translates communications, generates an investigative summary or produces a risk score, its

conclusions arrive wrapped in the appearance of scientific authority. The output may be presented as a probability, but it can easily be interpreted as a fact.

That distinction matters.

Every AI model is built upon assumptions. It reflects choices made about training data, classification thresholds, confidence scores, acceptable error rates and model design. Those decisions are often made long before an investigator, prosecutor or court ever encounters the result.

The question therefore is not whether AI is accurate.

The question is whether the organisations deploying it understand how that accuracy has been defined, measured, validated and governed.

This is where the lessons of Horizon and the Danish telecoms scandal become particularly relevant.

In both cases, the technology itself was only part of the problem. The greater failure was institutional. Organisations placed confidence in systems they did not fully understand, could not adequately explain and were insufficiently equipped to challenge. Errors persisted because governance failed before technology failed.

Criminal justice cannot afford to repeat that mistake.

AI may help determine which evidence is reviewed first, which suspects attract attention, which risks are prioritised and which investigative opportunities are pursued. These are not merely administrative decisions. They are decisions that shape investigative direction, influence resource allocation and ultimately affect outcomes.

As AI systems become more capable, governance becomes as important as capability

Indeed, AI should be viewed first as a governance challenge and only second as a technology opportunity.

Before any AI-derived output influences a criminal justice process, four questions should be capable of being answered clearly and transparently:

1. What data was used to train the model?

2. How has the model been independently tested and validated?

3. What are its recognised limitations, what is its accuracy, and what conditions are known to lead to mistakes or misleading results?

4. Who remains accountable when it is wrong?

If those questions cannot be answered, the system is not ready for evidential use.

This matters because AI does not eliminate poor quality data. It scales it.

An investigator making an incorrect assumption may affect a single case.

An AI model trained on incomplete, biased or inaccurate data can replicate and amplify those flaws across thousands.

The history of criminal justice provides countless examples of evidence that appeared robust until someone asked difficult questions. The purpose of governance is to ensure those questions are asked before failure occurs, not afterwards.

This requires more than technical validation.

It requires independent assurance, documented testing, transparent standards, reproducible methodologies, continuous monitoring and meaningful accountability. Most importantly, it requires organisations to foster a culture in which AI outputs can be challenged.

That challenge role cannot be performed by technology alone.

AI, Citizen Evidence and the New Trust Threshold

The growth of artificial intelligence in policing coincides with an extraordinary expansion in the volume and variety of data available to investigators. Dashcams, smart doorbells, mobile phones, wearable devices, social media and connected-home technologies allow citizens to record incidents and share material with police on a scale that would previously have been unimaginable. Recent research undertaken by the Police Foundation³ in their report: Policing and Digital Citizen Evidence: Unleashing the Potential for Public Participation (2026) outlines the opportunities, challenges, and policy needs associated with this shift.

This material creates considerable opportunity. It can reveal incidents the police did not witness, identify evidence that might otherwise be missed, support safeguarding and enable the public to participate more directly in community safety. But every submission is also an act of trust. A citizen who provides footage is not simply transferring a data file. They are placing confidence in the police to receive it securely, understand its context, assess it fairly, protect the rights of those depicted and act proportionately.

The Police Foundation research found that poor handling, unexplained decisions or a failure to respond can damage confidence not only among individual submitters but across their families, communities and wider social networks.

As the volume of citizen-generated material grows, AI will undoubtedly have a role in helping police manage demand. It may assist with classification, transcription, translation, duplicate identification, metadata extraction and the prioritisation of potentially relevant material. However, simply feeding citizen-provided footage, images or messages into an AI model without effective triage, validation and human oversight would be a serious mistake. Citizen-generated information is not automatically reliable evidence. Authenticity, provenance, relevance, context, metadata and continuity must still be established before that material can properly support an investigation or prosecution.

Nor can an AI system fully understand the circumstances in which material was created or submitted. A clip may be incomplete, misleading, altered, stripped of metadata or captured from a perspective that omits essential context. It may contain private information about victims, witnesses or uninvolved third parties. Increasingly, investigators must also contend with synthetic media and the possibility that fabricated material could be presented as genuine, while authentic material is dismissed as AI-generated. These risks make secure collection, verification, audit trails and accountable human judgement more important, not less.

The central question is therefore not whether AI should be used to process digital citizen evidence, but where automation should end and accountable professional judgement should begin. AI can help police find the signal within an expanding volume of data. It should not determine, without meaningful human review, whether a citizen is believed, whether a person is treated as suspicious, whether material is evidentially reliable or whether coercive police action should follow.

The bond of trust between policing and the public depends upon that distinction. If AI is used transparently to support triage, preserve evidential integrity and help officers make better-informed decisions, it can make public participation more meaningful. If citizen-provided material disappears into an opaque automated system, is misunderstood, unfairly discounted or acted upon without context, the technology risks weakening the legitimacy on which policing by consent depends. The governing principle should be clear: automation may manage volume, but responsibility for judgement must remain visible, auditable and human-led.

Human oversight must remain central

Unfortunately, "human in the loop" has become an increasingly common phrase and an increasingly vague one. If human involvement amounts to little more than accepting or endorsing an algorithmic recommendation, it provides little protection. Human oversight only has value when individuals have the competence, authority and confidence to disagree with the machine.

A human reviewer must be able to understand the provenance of the data, recognise the limitations of the model, appreciate the context of the decision and challenge the output where appropriate.

Accountability cannot be delegated to an algorithm simply because its conclusions appear objective.

None of this argues against the use of AI.

Far from it.

Some of the most valuable applications of AI are likely to be those furthest removed from evidential decision-making itself. Disclosure review, redaction, transcription, translation, triage and the identification of potentially relevant material all represent areas where AI can significantly improve efficiency while leaving ultimate judgement with investigators, prosecutors and courts.

AI is at its most effective when it acts as a force multiplier rather than a decision-maker.

The challenge, therefore, is not technological. It is institutional.

Can criminal justice harness the benefits of AI without repeating the mistakes that accompanied earlier generations of digital evidence?

Can organisations remain sceptical when systems become increasingly persuasive?

Can governance keep pace with capability?

Most importantly, can public trust be preserved?

Trust in criminal justice is difficult to build and easy to lose. The lessons of Horizon and the Danish telecoms scandal demonstrate that confidence is undermined not simply by technical failure, but by the failure to question technology when it appears authoritative.

AI undoubtedly has a role to play in the future of policing, investigation and prosecution. Given the scale of modern digital evidence, it may become indispensable.

But criminal justice must resist the temptation to confuse efficiency with certainty.

Technology can assist judgement.

It cannot replace it.

The core duty of criminal justice is unchanged: to scrutinise evidence, challenge assumptions and test conclusions.

In the age of artificial intelligence, that duty becomes more important, not less.

The greatest technology-related miscarriages of justice rarely arise from deliberate wrongdoing. They happen when institutions place too much confidence in systems, they no longer question.

That was a lesson from Horizon.

It was a lesson from the Danish telecoms scandal.

And it is a lesson that should inform every discussion about AI in criminal justice.

The organisations that will succeed in deploying AI responsibly will not be those with the most sophisticated algorithms. They will be those with the strongest governance, the greatest transparency and the deepest commitment to evidential scrutiny.

The future of AI in criminal justice should therefore be governed by a simple principle:

No matter how advanced the technology, every conclusion must remain capable of explanation, challenge and human accountability.

Because the most important safeguard in any justice system is not artificial intelligence.

It is informed human judgement.

Sources:

1. The Criminal Cases Review Commission Horizon case summary. https://ccrc.gov.uk/post-office-horizon-cases/

2. The use of Historical Call Data Records as Evidence in the Criminal Justice System: Lessons Learned from the Danish Telecom Scandal. Authors: Lene Wacher Lentz and Nina Sunde. Published in Digital Evidence and Electronic Signature Law Review (2021)

3. Police Foundation; Policing and Digital Citizen Evidence: Unleashing the Potential for Public Participation (2026)