LLMs cannot reliably self-report when they've been adversarially manipulated, and training methods meant to improve this detection can paradoxically make models more vulnerable to attacks while appearing more confident in false claims.
This paper investigates whether large language models can accurately recognize when their own outputs were manipulated by adversarial prefill attacks. Testing 10 models across 4 safety benchmarks, researchers found that models fail to reliably detect their compromised responses, often falsely claiming they acted intentionally.