NAMAA-Space/aragenre-generalist-ensemble-zeroshot-vote

AraGenre Generalist Ensemble — Zero-Shot Equal-Weight Vote

An equal-weight majority-vote ensemble of 4 models for hierarchical Arabic genre classification. This is not a single fine-tuned checkpoint — the repo contains a weights JSON and a combination script, not trained weights of its own.

Authors: Hassan Barmandah (NAMAA Community; Umm Al-Qura University), Israa Elhosiny (NAMAA Community), Yousra El-Ghawi (NAMAA Community), Omer Nacar (NAMAA Community)

⚠️ Generalization Note

This ensemble's development-set score (0.9201 hierarchical F1) is not the same as real-world test performance, and dev numbers across this project's fine-tuned-encoder lineage should not be trusted as generalization estimates. Per the project's system-description paper, this broader lineage collapsed to 0.22–0.44 hierarchical F1 on the actual 27,972-item hidden test set (74 specific genres under 6 broad genres). No dev labels were used to select this ensemble's weights (see below), but it was not the system submitted to the official leaderboard.

The system that actually won for this team — 0.7013 hierarchical F1, 3rd of 18 teams on the official CodaBench leaderboard — was a separate, zero-shot DeepSeek-LLM pipeline with no fine-tuning at all (stage2_llm_zeroshot_pipeline/ in the project repo). This artifact is not that system.

Approach

Equal-weight ensemble of mdeberta-nli-zeroshot + llm-zeroshot-reasoning + e5-large-paraphrase-augmented + e5-large-multicorpus-xgenre. Deliberately not dev-weight-tuned, unlike the 8-model/9-model ensembles which grid/random-search weights against dev hierarchical F1 — equal weights use zero bits of dev information beyond the choice of which 4 systems to include, at some cost to raw dev score versus the dev-tuned ensembles.

Combines by majority vote over each system's top-1 predicted specific genre (not softmax-score averaging), since the LLM-reasoning member only produces a discrete label, not a per-genre score matrix. Ties are broken by preferring the multicorpus-xgenre model's prediction — a fixed, dev-independent rule, not a tuned weight.

Components

mdeberta-nli-zeroshot, llm-zeroshot-reasoning, e5-large-paraphrase-augmented, e5-large-multicorpus-xgenre — all four are themselves designed to generalize past dev.

Training data

None directly — this is a weight/vote recipe over pre-scored component models. No dev labels were used in selecting weights.

Usage

Requires cached dev scores/predictions from running all 4 component scripts first, then:

python generalist_ensemble_zeroshot_vote.py

See the project repository for the full script and component-model requirements.

Citation

If you use this work, please cite our system-description paper:

@inproceedings{barmandah-etal-2026-namaa,
  title = {NAMAA at AraGenre 2026: From Encoder Baselines to Self-Consistent LLM Ensembling for Hierarchical Arabic Genre Classification},
  author = {Barmandah, Hassan and Elhosiny, Israa and El-Ghawi, Yousra and Nacar, Omer},
  booktitle = {Proceedings of the 4th Arabic Natural Language Processing Conference (ArabicNLP 2026)},
  address = {Budapest, Hungary},
  publisher = {Association for Computational Linguistics},
  year = {2026},
}

Please also cite the AraGenre 2026 shared task overview paper:

@inproceedings{elhaj-etal-2026-aragenre,
  title = {AraGenre 2026: A Hierarchical Definition-Guided Arabic Genre Classification Shared Task},
  author = {El-Haj, Mo and Ezzini, Saad and Abudalfa, Shadi and Lamsiyah, Salima and Jarrar, Mustafa},
  booktitle = {Proceedings of the 4th Arabic Natural Language Processing Conference (ArabicNLP 2026)},
  address = {Budapest, Hungary},
  publisher = {Association for Computational Linguistics},
  year = {2026},
}

License

Apache 2.0