The novelty exposed: ablation, reproducibility and transparency in 40 technical reports of open-weight large language models (LLMs)
Abstract
In 2025 alone, 48 new large language model (LLM) architectures were added to the Hugging Face transformers library, and many of them arrive with a technical report that presents the model as new. What is actually new in these models, and is it tested? We put this question to LLM technical reports. We look at three things: what the reference code shows about how new open-weight architectures differ from earlier ones; what 40 randomly sampled technical reports claim as new and whether they support it with ablation studies; and what they disclose for reproducibility and transparency. In the library, 39 of the 48 architectures added in 2025 take at least two of their three core blocks from another model, most often Llama, although part of this reflects how the library writes code. In the reports, 19 of 73 architectural or training novelty claims come with a controlled with/without comparison; 4 of 40 reports state more than one training run; 8 rerun the models they compare against under one setup; and 1 discloses data mixture, token count, and compute together. We also looked for known components given new names: none of five candidate cases hid its origin, though we did not study model names. We propose seven reporting rules for LLM release reports and apply them to two reports. We do not judge model quality, and an untested claim may still be true.
Community
We coded 40 randomly sampled LLM technical reports for what they claim as new and how they test it. Only 19 of 73 architectural or training novelty claims come with a controlled with/without ablation, 4 of 40 reports state more than one training run, and 1 discloses data mixture, token count and compute together. In the transformers library, 39 of the 48 architectures added in 2025 reuse at least two of three core blocks from another model. We propose seven reporting rules for release reports. All coded claims (with page-anchored quotes) are in the dataset below.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- From RL Papers to Framework Code: Inclusion and Review in Seven Open-Source Training Frameworks (2026)
- When Local Definitions Are Missing: Stable Errors in Language Models (2026)
- Review Rarely Beats the Stronger Model: Separating Verification from Capability Transfer in LLM Pipelines (2026)
- Written Once, Read Differently How Language Models Own Answers Steer Their Next Ones (2026)
- ALIAS: Evaluating Performance and Memorization in LLM-Written TLA+ Specifications (2026)
- Judges Follow the Reference: The Derivation Gap in Verifying Tool-Call Arguments (2026)
- Reading Is Not Leaking: Local, Auditable Measurement and Reduction of Inference Exposure from Public Footprints (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2611.20003 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 1
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper