Upwork-AIML/Conditional_Causal_Difficulty_for_LLMs

Conditional Causal Difficulty for LLMs

Code for "Fragility Is Not Reducible to Hardness: Conditional Causal Difficulty for LLMs" (arXiv:2611.02065).

Abstract

LLMs often fail on a task after it is rewritten in a way that keeps its requirements and answer the same. We treat this as a causal question: how often does a meaning-preserving rewrite break a task the model can already solve? We call this the conditional causal difficulty (CCD): the probability that a randomly drawn twin breaks a solvable task. Because harder tasks tend to be more phrasing-sensitive, complexity is an effect modifier rather than a confounder; residualizing it out gives the complexity-residualized difficulty (CRD).

To measure this we release GlassGSM and GlassMATH: GSM8K and MATH problems with answer-preserving rewrites ("twins"). On GlassGSM, 44.9% of problems are fragile, compared with 3.7% under GSM-IC-style distractors. Our predictor Frail-π learns from the original problem text which tasks are fragile. It reaches 0.613 ± 0.023 of the suite reliability ceiling on GlassGSM and 0.557 ± 0.023 on GlassMATH, against 0.444 and 0.463 for the strongest baseline (VAP3). At a 5% curation budget on GlassGSM it doubles random recall of fragile items, matching an oracle ranking.

The idea

Figure 1 shows the pipeline. A problem the model solves is rewritten by deterministic operators (rename, reorder, inject; no LLM), each twin is re-solved, and CCD (π) is the share of twins that break it. Frail-π then predicts π from the original problem alone, so fragile problems can be found without running any rewrite. Comparing a task with its own twin is not confounded by complexity, so the effect is identified by design.

Fragility is not hardness

Seven hardness measures (answer entropy, pass rate, solution length, gold-solution step count, declared complexity, difficulty across three solvers, base correctness) rank CCD at no more than 0.39 of the reliability ceiling, and 0.36 when combined in one linear model. Frail-π keeps 0.47 after all seven are partialled out. These runs use greedy labels (temperature 0, one sample), so their values are not on the scale of the main results.

Across solvers

Frail-π beats the strongest baseline for every solver (gpt-4o-mini, Qwen3.7-plus, DeepSeek-v4-pro) on GlassGSM. Each solver's group is trained and evaluated on that solver's own CCD over its own solvable stratum, on single-dose (m = 9) twins. The twins are deterministic, so the solver is the only thing that changes the label.

Dataset

The benchmarks live on the Hugging Face Hub (gpt-4o-mini labels) and are not duplicated in this repository:

• GlassGSM (1,319 GSM8K problems): reza2ei/GlassGSM
• GlassMATH (4,999 MATH problems): reza2ei/GlassMATH

hf download reza2ei/GlassGSM  --repo-type dataset --local-dir release/glassgsm
hf download reza2ei/GlassMATH --repo-type dataset --local-dir release/glassmath

import pandas as pd

twins = pd.read_parquet("release/glassgsm/data/glassgsm/twins.parquet")     # GlassGSM
math = pd.read_parquet("release/glassmath/data/glassmath/twins.parquet")    # GlassMATH

Split by base_id, never by row. See the dataset cards for all fields.

Setup

python -m venv .venv && source .venv/bin/activate
pip install -e ".[hf,openai,dev]" pandas pyarrow

Python 3.10 or newer. An LLM API key is only needed to build new twins, not to run the steps below.

Quickstart

# 1. Download GlassGSM from the Hub (see Dataset above)
hf download reza2ei/GlassGSM --repo-type dataset --local-dir release/glassgsm

# 2. Convert it to the format the training scripts read
python scripts/release_to_labeled_pairs.py --suite glassgsm --solver gpt-4o-mini

# 3. Train and evaluate Frail-π (default: fine-tuned DeBERTa encoder, use a GPU)
python scripts/fragility_report.py \
  --dataset "$PWD/data_audited_multiop_mix/gsmic_adv_m9_multiop_mix_4omini_labeled_pairs.jsonl" \
  --seeds 0,1,2,3,4 --out-dir runs/frailpi

# 4. Run the tests
python -m pytest tests -q

Use an absolute path for --dataset (as $PWD/... above).

Reading the results. runs/frailpi/report.md first shows Spearman divided by the reliability ceiling (1.0 is the best any predictor can reach given label noise; this is the scale of Figure 1), then the raw Spearman.

Best configuration. bash scripts/run_frailpi_base.sh runs the base recipe: fine-tuned DeBERTa-v3-base plus 24 ex-ante probes of each base problem (0.61 of the reliability ceiling on GlassGSM at m = 9). It needs a GPU and the probe file described at the top of that script. Other runs change single flags on top of it.

No GPU? Add --featurizer tfidf_svd --seeds 0 for a quick CPU run. Its numbers are lower than the paper's.

What is where

Folder	Contents
src/causal_difficulty/	Frail-π model, encoders, baselines, estimands
scripts/	Experiment entry points
configs/	Run configurations
release/glassgsm/, release/glassmath/	Where the downloaded datasets are expected (not tracked in this repo)
tests/	Unit tests
docs/DETAILS.md	Full reference: estimand definitions, every experiment step, config options

Older file and config names use gsmic_adv for GlassGSM and math_adv for GlassMATH.

Citation

@misc{rezaei2026fragilityreduciblehardnessconditional,
      title={Fragility Is Not Reducible to Hardness: Conditional Causal Difficulty for LLMs},
      author={Mohammad Reza Rezaei and Dingxian Wang and Nikhil Verma and Sajjad Pakdamansavoji and Andrew Rabinovich and Zhao Chen},
      year={2026},
      eprint={2611.02065},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2611.02065},
}

MIT license (see LICENSE). The datasets contain text derived from GSM8K and MATH (both MIT); see release/*/NOTICE.md.