AI Models Get the Answer Right but Cite the Wrong Source
A new study from Peking University and the Shanghai Artificial Intelligence Laboratory has revealed a troubling flaw in how leading AI models handle documents: they frequently get the answer right but point to entirely the wrong source to back it up. The researchers call this problem AI attribution hallucination, and it could have serious consequences in fields like law, medicine, and financial auditing.
To measure the problem, the team built a benchmark called CiteVQA, the first of its kind designed to test not just whether an AI gives a correct answer, but whether it can correctly identify where in a document that answer comes from. The dataset spans 1,897 questions across 711 PDFs, covering seven subject areas in both English and Chinese, with documents averaging over 40 pages each.
The results were revealing. Gemini-3.1-Pro-Preview, the top-performing model in the test, scored only 76 out of 100 under the benchmark’s strict scoring system. GPT-5.4 showed an even sharper drop: it scored 87.1 on raw answer quality, but when correct citations were required, that figure fell to just 59. The model knew the answer, but couldn’t show its work.
AI attribution hallucination turned out to be even worse among open-source models. Qwen3-VL-235B-A22B, the strongest freely available system tested, managed only 22.5 points. Smaller open models mostly fell below 10, which the researchers described as “extremely risky” for any regulated industry where claims must have a verifiable paper trail.
The scoring system used is called Strict Attributed Accuracy. A model only earns points when both the answer is correct and the citation lands precisely on the right paragraph, table, or figure. A page number alone is not enough. This standard exposed a gap that standard benchmarks like DocVQA and MMLongBench-Doc have long overlooked, since those tests only grade the final answer and have no way of knowing whether the model actually found the information or simply guessed from prior knowledge.
Document layout matters too. Academic papers with clean, structured formats scored best. Newspapers and magazines with busier designs held even top models to around 63 points. Multi-document tasks also proved harder, with Gemini 3.1 Pro Preview’s recall dropping from around 69 percent to 55 percent when pulling information across multiple files.
When the researchers gave models a narrowed search space, feeding them only the relevant pages or the correct document, scores jumped by more than 13 points in some cases. This confirms that AI attribution hallucination is largely a retrieval and localisation problem. Models that know where to look not only cite better, they also answer better.
This finding connects to a broader pattern in AI research. A 2024 study from the same Shanghai AI Laboratory showed that language models struggle with long documents generally. Google DeepMind has also tackled a related problem with its FACTS Grounding benchmark, which checks whether model answers stay grounded in a provided document rather than drawing on outside knowledge. And OpenAI has acknowledged that training systems tend to reward confident answers over honest uncertainty, a dynamic that likely feeds into AI attribution hallucination in source citations as well.
The CiteVQA code is publicly available on GitHub, and the dataset can be downloaded on Hugging Face.
