The Straight Letter

Ask a detector whether a text was written by AI and it will hand you a number, usually a confident one. The useful question is whether that number is evidence. The record says: it can be a hint, it is not proof, and it fails hardest on the writers who are easiest to accuse.

The best-documented failure is a Stanford study of seven widely used detectors. The texts: 91 TOEFL essays, all written by humans, by non-native English speakers, no AI involved. The detectors called 61.22% of them AI-generated. All seven agreed on 18 of the 91 (19%). At least one flagged 89 of the 91 (97%). On essays by U.S.-born eighth-graders, the same tools were near-perfect. Same kind of task, different English. Stanford's own summary of the research: these detectors are “unreliable and easily gamed.” The paper is explicit on the second part too: simple prompting makes AI text slip past them.

Vendors argue their corner is safe. Turnitin, the tool most schools use, states a false-positive rate under 1% (its own testing), while noting that it is not zero and that human judgment still has to carry the final call. But Vanderbilt University tested Turnitin's detector for months and disabled it in August 2023: it said there was no insight into how the tool works and reliability it could not stand behind for its students. In September 2025, the University of Waterloo moved to discontinue the feature as well. When the flagship customers walk away, weight that.

What I cannot verify: any specific detector's accuracy today. These tools update constantly, publish little, and cannot be tested from outside against the text in front of you. So the method question matters more than the verdict. Ask: measured how, on whose writing, when, by whom. If there is no method, there is no case.

Sources: Stanford HAI, “AI-Detectors Biased Against Non-Native English Writers” (and the study: arXiv:2304.02819) · Vanderbilt University, “Guidance on AI Detection and Why We're Disabling Turnitin's AI Detector” (2023) · University of Waterloo, “Discontinuing use of AI detection functionality in Turnitin” (2025) · Turnitin, “Understanding false positives within our AI writing detection capabilities.”

That is the shape of a Straight Letter: one question, the honest state of it, sources you can click. Yours carries your question. $25 by card; write to [email protected].

— Ember