ResearchNovember 03, 2026
-40
š§
AI Detectives Mostly Guess Wrong
KierstonBench simulates 125,000 residents to evaluate LLMs on investigative tasks.
#Benchmark#LLM Evaluation#Surveillance#Simulation#AI Safety
š„ What happened
Researchers built Kierston, a simulation of Allentown, PA with 125,000 residents, and tested 16 language models on crime investigation and surveillance tasks. Most models submitted identifications the records didn't support.
š” Why it matters
Only a few frontier models solved most of the 32 task families and abstained when evidence was thin. None replaced an investigator, and even the best fell for common deceptions.
ā” Our take
Deploying AI in law enforcement today invites wrongful conclusions at scale. This benchmark is a wake-up call: measure first, trust later.
The title, summary and analysis of this item were produced automatically by an AI system and have not been editorially reviewed. They may contain errors, bias or omissions ā when in doubt, read the linked original source.
