Abstract
Large language models used for peer review draw about half or more of their identified weaknesses from paper appendices, whereas human reviewers draw only 8.4% and focus on presentation and core ideas.
Large language models (LLMs) are increasingly used by reviewers to highlight weaknesses during the reviewing process, and their outputs often shape the reviews that humans submit. Prior work studies how LLM reviews align with human ones, yet we know little about which parts of a paper these models actually criticize. In this work, we investigate 2,000 papers from ACL, EMNLP, COLM, and ICLR, and show that the latest GPT and Claude models draw about 50% or more of their review weaknesses from the appendix. This behavior holds across the API, coding-agent, and web access methods for using LLMs to review a paper. In contrast, we find that humans draw only 8.4% of their weaknesses from the appendix, and these weaknesses primarily revolve around a paper's presentation and content related to its core idea. Meanwhile, LLMs draw weaknesses from deep in the appendix that concern fine-grained technical details not central to the paper's contributions. We anticipate that our analysis will contribute to the scholarly discourse on AI reviews. Our code is publicly available at: https://github.com/cincynlp/review_appendix.
Community
Large language models (LLMs) are increasingly used by reviewers to highlight weaknesses during the reviewing process, and their outputs often shape the reviews that humans submit. Prior work studies how LLM reviews align with human ones, yet we know little about which parts of a paper these models actually criticize. In this work, we investigate 2,000 papers from ACL, EMNLP, COLM, and ICLR, and show that the latest GPT and Claude models draw about 50% or more of their review weaknesses from the appendix. This behavior holds across the API, coding-agent, and web access methods for using LLMs to review a paper. In contrast, we find that humans draw only 8.4% of their weaknesses from the appendix, and these weaknesses primarily revolve around a paper’s presentation and content related to its core idea. Meanwhile, LLMs draw weaknesses from deep in the appendix that concern fine-grained technical details not central to the paper’s contributions. We anticipate that our analysis will contribute to the scholarly discourse on AI reviews.
Hi authors, thanks for the work. A question on Section 5: it says human reviews reuse the LLM attribution step, but in the released code the human prompt adds "Assign a weakness to the Appendix ONLY if it is very obvious", which the LLM prompts lack. Does the 8.4% vs 62% contrast hold under the same prompt? Details in this GitHub issue: https://github.com/cincynlp/review_appendix/issues/2
Thank you for this question. The prompts differ because AI-generated and human-written reviews are structured very differently.
AI-generated reviews always say which section a strength or weakness refers to. They do this either directly ("Section 6.2 mentions xxx...") or in parentheses after the comment ("there might be a confound in the analysis (Section 6.2)"). This pattern is consistent across AI reviews, which makes attribution easy. The instruction "Base your answer on where the content actually appears in the paper, not on where the review says it appears" handles a different case that is common in AI reviews: a weakness that cites several sections to support its claim. For example, the main issue may be in Section 3, but the weakness also points to Sections 5 and 6 and Appendix B as support. In such cases, we want the weakness attributed to where the issue actually lies, the main body (Section 3), and this instruction is designed for that.
Human-written reviews, on the other hand, rarely mention which section a weakness refers to. In our initial runs, this led the LLM to guess. For example, a generic comment like "the mathematical notations are hard to follow" was attributed to the appendix, since the appendix contains most of the proofs. But a careful reading of the review made it clear the reviewer was not talking about the appendix. We added the instruction to stop the LLM from assigning sections based on such assumptions.
Most importantly, during our test runs we manually went through the outputs for both human and AI reviews and checked the attributions ourselves. We refined the prompts until they gave reliable attributions for each type of review. The instructions differ slightly only because the reviews themselves differ.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Defensive Writing in GPT Models (2026)
- Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation (2026)
- PremiseCheck: Efficient Fact-Checking of Human and AI Peer Reviews (2026)
- PaperScorer: How Predictable Is Paper Acceptance? (2026)
- I would rather quit NLP than read another paper like this: The rise of antithesis in NLP papers (2026)
- How Much Were You Told? Measuring External Information in Peer Reviews (2026)
- A Missing Piece for Trustworthy AI Reviewers: From Benchmarking Rhetorical Robustness to SciCore Review (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper