hannah-khallaf/e2r-strategy-gemma-3-12b-pairwise-qlora

E2R Strategy Gemma 3 12B Pairwise QLoRA

This repository contains the PEFT/QLoRA adapter for a Gemma 3 12B pairwise multilabel classifier for identifying Easy-to-Read (E2R) simplification strategies.

Base model

google/gemma-3-12b-it

Task

Given a standard sentence and its Easy-to-Read rewrite, the model scores each candidate strategy independently.

The six strategy labels, in benchmark order, are:

1. Synonymy
2. Modulation
3. Compression
4. Explanation
5. Syntactic Change
6. Omission

The model therefore performs six binary candidate-label evaluations for each sentence pair.

Model formulation

The checkpoint uses:

• pair input: standard sentence + Easy-to-Read sentence;
• pairwise binary-relevance formulation;
• QLoRA;
• all negative candidate labels during training;
• token-probability decisions based on the next-token probabilities of true and false;
• Gemma 3 12B Instruct as the base model.

Development-selected thresholds

The thresholds were selected using the development set only:

Strategy	Threshold
Synonymy	0.54
Modulation	0.87
Compression	0.11
Explanation	0.16
Syntactic Change	0.32
Omission	0.11

The thresholds are also provided in thresholds.json.

Recovered checkpoint

This repository contains seed 13 from a reproducibility rerun of the six-label benchmark.

This is not claimed to be the byte-identical historical checkpoint from the earlier experiment, because that checkpoint was no longer available. The model was retrained from the preserved experiment configuration.

Development performance

• Macro-F1: 0.713499
• Micro-F1: 0.773989
• Exact-set accuracy: 0.335878
• Hamming loss: 0.181298

Test performance

• Macro-F1: 0.640985
• Micro-F1: 0.703438
• Exact-set accuracy: 0.163701
• Hamming loss: 0.245552

Files

• adapter_model.safetensors — QLoRA adapter weights
• adapter_config.json — PEFT adapter configuration
• tokenizer/ — tokenizer and Gemma chat template saved with the run
• resolved_config.yaml — full experiment configuration
• thresholds.json — development-selected strategy thresholds
• dev_summary.json — development-set results
• test_summary.json — test-set results
• dev_per_label.csv — development per-label results
• test_per_label.csv — test per-label results
• training_history.csv — training history
• runtime.json — recorded runtime information
• hf_reload_full_test_comparison.csv — Hugging Face reload validation
• inference.py — benchmark-compatible model loading/scoring utilities

Important usage note

This repository contains a PEFT adapter rather than a merged copy of the Gemma 3 12B base model. Access to and use of the base model remain subject to the terms applicable to google/gemma-3-12b-it.

The classifier requires the candidate-label prompting formulation used by the E2R benchmark. Loading the adapter alone as a generic text classification pipeline does not reproduce the task.

Quick start

The classifier takes a standard sentence and its Easy-to-Read rewrite and predicts all six transformation strategies independently.

Installation

pip install torch transformers peft bitsandbytes accelerate huggingface_hub

Gemma 3 access must be available for the Hugging Face account used to run the model.

Classify one sentence pair

from inference import E2RStrategyClassifier

classifier = E2RStrategyClassifier.from_pretrained(
    "hannah-khallaf/e2r-strategy-gemma-3-12b-pairwise-qlora"
)

result = classifier.predict(
    standard=(
        "Before moving to Spain, Kate had worked "
        "as a barrister in the UK."
    ),
    easy_to_read=(
        "Kate was a lawyer in the United Kingdom. "
        "Then she moved to Spain."
    ),
)

print(result["labels"])

for label, probability in result["probabilities"].items():
    print(
        f"{label:20s} "
        f"{probability:.4f} "
        f"(threshold={result['thresholds'][label]:.2f})"
    )

The result has the form:

{
    "labels": [
        # strategies predicted for this pair
    ],
    "probabilities": {
        "Synonymy": ...,
        "Modulation": ...,
        "Compression": ...,
        "Explanation": ...,
        "Syntactic Change": ...,
        "Omission": ...
    },
    "thresholds": {
        "Synonymy": 0.54,
        "Modulation": 0.87,
        "Compression": 0.11,
        "Explanation": 0.16,
        "Syntactic Change": 0.32,
        "Omission": 0.11
    }
}

The values above are deliberately left unspecified here. Use the model output itself for the probabilities; the classifier applies the development-selected threshold for each strategy automatically.

Batch prediction

examples = [
    {
        "standard":
            "Before moving to Spain, Kate had worked "
            "as a barrister in the UK.",
        "easy_to_read":
            "Kate was a lawyer in the United Kingdom. "
            "Then she moved to Spain.",
    },
    {
        "standard":
            "Well, uh, I am hungry.",
        "easy_to_read":
            "I am hungry.",
    },
]

results = classifier.predict_batch(examples)

for result in results:
    print(result["labels"])

Output interpretation

A probability should not be interpreted using a universal threshold of 0.5. Each strategy uses the threshold selected on the development set.

For example, a Compression probability of 0.20 is classified as present because the Compression threshold is 0.11, while a Modulation probability of 0.80 is classified as absent because its threshold is 0.87.

The model performs six candidate-label evaluations for every sentence pair, so several strategies may be returned simultaneously.