Your Codebook Is the Problem, Not the LLM

🔥 What happened Researchers at Penn State show that bad F1 scores in LLM annotation often trace back to the codebook, not the model. Their pre-deployment audit flags under-specified and conflicting rules, and targeted revisions cut errors on held-out data. 💡 Why it matters Most teams tune prompts and swap models instead of auditing their measurement instrument. The audit separates three error sources — model, codebook, reference labels — and finds some boundary fixes transfer across models while others stay annotator-specific. ⚡ Our take Jumping to a bigger model when F1 drops is burning budget. Audit the codebook first, then fine-tune — anything else is blind firefighting.