Hugging Face's logo

Upwork-AIML
/
Conditional_Causal_Difficulty_for_LLMs

Model card Files
xet
 Community
Conditional_Causal_Difficulty_for_LLMs / docs /DETAILS.md
reza2ei's picture
reza2ei
Duplicate from reza2ei/Conditional_Causal_Difficulty_for_LLMs
a0d1de2
preview code
|
Raw History Blame Contribute Delete
41.3 kB

Conditional Causal Difficulty for LLMs: full documentation

Short version: see the README. This is the long reference.

Code for "Fragility Is Not Reducible to Hardness: Conditional Causal Difficulty for LLMs" (arXiv:2611.02065).

LLMs show large, seemingly arbitrary performance drops when a task is rewritten in ways that leave its requirements untouched. We treat task difficulty as a causal quantity β€” the treatment effect of meaning-preserving perturbations on a model β€” and ask which tasks are anomalously fragile, beyond what their complexity predicts.

This repository contains everything needed to reproduce the paper:

  • the estimands CCD (conditional causal difficulty) and CRD (complexity-residualized difficulty), with cross-fitted residualization;
  • the predictors Frail-Ο€ (CCD from text) and Frail-Ο„ (the unconditional contrast, used for Synth-CD and as a control);
  • the GlassGSM and GlassMATH benchmarks: construction pipelines, the No-Op contamination audit, and the multi-operator-mix corpora;
  • the Synth-CD mechanism-check suite and every baseline (BERTPE, VAP3, PromptSET measures, surface features, IRT, …);
  • the reliability-ceiling, identity-twin-floor, curation, and solver-transfer analyses.

Naming. The benchmarks are called GlassGSM (twins of GSM8K) and GlassMATH (twins of Hendrycks MATH). Older config and file names still use gsmic_adv (= GlassGSM) and math_adv (= GlassMATH). They are the same thing.

Headline results (from the paper)

Test-fold Spearman, normalized by each suite's reliability ceiling (0.969 GlassGSM, 0.972 GlassMATH), mean Β± sd over five seeds (Table 2 of the paper):

Method (predicting CCD, Ο€) GlassGSM GlassMATH
Frail-Ο€ (fine-tuned encoder) 0.613 Β± 0.023 (raw ρ_S 0.594) 0.557 Β± 0.023 (raw ρ_S 0.541)
VAP3 0.444 Β± 0.049 0.463 Β± 0.010
BERTPE 0.420 Β± 0.051 0.451 Β± 0.015
Surface features (GBT) 0.171 Β± 0.023 0.401 Β± 0.014
Declared complexity only 0.207 Β± 0.060 0.285 Β± 0.026

Other headline findings:

  • Fragility is a different quantity from hardness. GlassGSM raises fragility from 3.7% (GSM-IC-style distractors) to 44.9% on the same problems while keeping the answer key unchanged. The measured base pass rate Θ²(J) is negatively correlated with CCD.
  • CRD is orthogonal to the declared complexity proxy (|Corr(target, C)| ≀ 0.06 on the m=9 run), and the contrast architecture is complexity-orthogonal even without explicit residualization on Synth-CD (Frail-Ο„ ρ_S = 0.617 at |ρ_C| = 0.028).
  • Actionable for curation. On GlassGSM, selecting the top 5% of problems by Frail-Ο€ gives Recall@5% = 0.099 of truly fragile items vs 0.05 for random and 0.100 for an oracle ranking.
  • Partial, solver-dependent transfer. Training Frail-Ο€ on one solver's CCD and testing on another's transfers only partially, as expected because the stratum is defined by each solver's own Y(J).
  • Measurement floors. The identity-twin floor (break rate of re-solved, unperturbed bases) is 1.1% on GlassGSM and 2.5% on GlassMATH.

See Β§11 for what to expect when re-running.

Table of contents

  1. The estimands: CCD and CRD
  2. Models: Frail-Ο€ and Frail-Ο„
  3. Datasets
  4. Repository layout
  5. Installation
  6. Quickstart (no API access needed)
  7. Reproduction guide
  8. Orchestration scripts
  9. Configuration reference
  10. Outputs and tests
  11. Reproducibility notes
  12. Citation, license, third-party material

1. The estimands: CCD and CRD

A task (prompt) is a token sequence J. An LLM maps it to a response O(J), which a quality-scoring function phi collapses to a scalar

Y(J) = phi(O(J), J) in [0,1]

(For the math suites this is exact match on the extracted modal answer over 5 samples at temperature 0.2.) A meaning-preserving perturbation operator (MPPO) T rewrites a task into a twin T(J) with the same requirements and the same gold answer but a different surface form. Let P be a distribution over such operators.

Per-instance contrast. Each task has a control score Y(J) and a treatment score Y(T(J)); the per-instance effect is

Delta(J, T) = Y(J) - Y(T(J)),        tau(J) = E_{T ~ P}[ Delta(J, T) | J ]

T is drawn independently of every attribute of J, and Delta compares the same task under two randomly assigned surface realizations β€” a randomized experiment within the item. Its contrast is unbiased without adjustment; the complexity back-door S <- C -> Y only opens for observational comparisons across prompts, which is what every Y(J)-predictor baseline estimates.

Why the unconditional average (U-CCD) is the wrong target

A meaning-preserving rewrite can only be observed to hurt a task the model already solves. On the stratum Y(J) = 0, every realized contrast satisfies Delta(J, T) <= 0: a task the model cannot solve at all is recorded as maximally robust rather than untested β€” the stratum is censored, not informative. The unconditional average D_avg(J) = E_T[Delta(J, T)] therefore mixes a live effect with a structural zero, and empirically it is unlearnable on the real suites: against U-CCD on GlassGSM every method, ours included, is weak (best ρ_S = 0.162, surface length, vs Frail-Ο€'s 0.594 on CCD). U-CCD results are retained as a control (paper_supports_real.py --estimand d_avg).

CCD β€” conditional causal difficulty

Restrict the estimand to the solvable stratum (Y(J) = 1) using the per-twin break event

B(J, T) = 1{ Y(J) = 1 and Y(T(J)) = 0 }

On this stratum Delta(J, T) = B(J, T) exactly, so CCD is the treatment effect evaluated where it is non-degenerate:

pi(J) = E_{T ~ P}[ B(J, T) | Y(J) = 1 ]

pi(J) β€” the conditional causal difficulty β€” is the fraction of the twin family that breaks a task the solver solved. It is a principal-stratum quantity (stratification on the control potential outcome). The paired design makes it the tractable case: stratum membership is observed (the base and every twin are actually run), and conditioning on Y(J) opens no selection path because T is independent of J.

Dose response. A twin's dose m in {1, 3, 5, 7, 9} indexes the perturbation budget of a corpus file (see Β§3). Fragility is summarized by a logistic dose response

logit Pr[ B(J, T) = 1 | m ] = a_J + b_J (m - m_bar)

where a_J is fragility at typical dose and b_J the degradation rate. Frail-Ο€ learns the per-perturbation break probability and aggregates it analytically to pi and to (a_J, b_J), instead of regressing a job-level scalar.

CRD β€” complexity-residualized difficulty

Harder tasks tend to be more phrasing-sensitive, so intrinsic complexity C is an effect modifier, not a confounder: the raw pi of a task is complexity-inflated. To isolate anomalous fragility β€” the part the declared complexity cannot account for β€” we residualize C out. Fix a base quantity Q in {tau, pi}, a declared complexity observable C, and a task population p (restricted to Y(J) = 1 when Q = pi). With m_{Q,C,p}(c) = E[ Q(J') | C(J') = c ]:

D^perp_{Q,C,p}(J) = Q(J) - m_{Q,C,p}( C(J) )

Properties (Proposition 1 of the paper):

  • D^perp is the L2 projection residual of Q onto measurable functions of the declared C; hence E[D^perp] = 0 and D^perp is uncorrelated with every function of C.
  • It is estimated by cross-fitted residualization: regress the measured Q_hat on the complexity proxy out-of-fold (folds shared with the main evaluation) to get m_hat, and set D^perp_hat = Q_hat - m_hat. This removes own-observation leakage and is consistent whenever Q_hat -> Q and the complexity regression is consistent.
  • D^perp = 0 for a task exactly as fragile as the typical same-C task; D^perp > 0 for anomalously fragile ones. Orthogonality is to the declared proxy only β€” a different C would give a different CRD.

The declared proxies are suite-specific and external to the pipeline: the problem-setter-assigned competition level for GlassMATH, and a complexity score computed from the problem text for GlassGSM (see Β§11). Implementation: src/causal_difficulty/eval/deconfound.py (cross_fitted_residual, orthogonality_diagnostics, stratified_rank_correlation), exercised by scripts/fragility_report.py (orthogonality block), scripts/paper_supports_real.py, and scripts/solver_crd_vs_ccd_transfer.py.

2. Models: Frail-Ο€ and Frail-Ο„

The repository contains two model families. They share losses and utilities but are trained by different entry points, so it matters which one a script uses.

Frail-Ο€ β€” break-probability model (the headline predictor for CCD)

Implemented in models/fragility.py (frozen features + MLP) and models/fragility_encoder.py (joint encoder fine-tuning / LoRA), driven by scripts/fragility_report.py.

It predicts the probability that a single perturbation at a given dose breaks a given task,

p(job, m) = sigmoid( f([ x_job, dose_features(m) ]) )

and is trained with binary cross-entropy on every (job, twin) observation rather than on a job-level scalar. Job-level estimands are then computed from p in closed form: pi is p averaged over the dose composition of the evaluation design, (a, b) come from a least-squares fit of logit p(job, m) over doses, and worst-case / quantile targets at budget K follow from 1 - (1 - p)^K and the binomial tail. One model therefore serves every estimand at every budget.

  • Paper configuration: --featurizer hf_finetune fine-tunes a pretrained encoder (DeBERTa-v3-base by default) end to end, with encoder LR 2e-5 and head LR 1e-3. Frozen encoders and LoRA are the ablations in the paper.
  • Contrast supervision is optional. --lambda-contrast (default 0.0) adds a within-job contrast term between two twins' predicted break probabilities. At the default, training is the plain per-observation BCE objective.
  • Other objectives: --loss-mode {bce, soft_rank, reliability_weighted}.
  • Two protocols, never merged: text (problem statement only) and text_probe (adds statistics of repeated solver samples of the base prompt β€” requires model access; see Step 4).
  • The CRD prediction subtracts the cross-fitted complexity regression from the prediction, D^perp_hat(J) = D_hat(J) - m_hat_pred(C(J)).

Frail-Ο„ / SCE β€” Siamese Counterfactual Encoder (unconditional contrast)

Implemented in models/sce.py, models/train.py, models/losses.py (scripts/train.py, scripts/evaluate.py, scripts/paper_supports_*.py). A shared encoder embeds the base J and the twin T(J) into h and h'; their difference feeds a contrast head, while a single-tower ex-ante head predicts difficulty when no twin exists:

Delta_hat(J, T) = psi_rho( h - h' )        # contrast head (per-observation supervision)
D_hat(J)        = g_omega( h )             # ex-ante head (deployable, base text only)

L = sum_T ell_delta( Delta_hat(J,T), Delta(J,T) )
  + ell_delta( g_omega(h), D^obs(J) )
  + lambda * R_inv(theta; J)

R_inv = E[ || Pi_C h - Pi_C h' ||^2 ] is an invariance regularizer with a learned complexity-subspace projection Pi_C, encouraging h = c(C(J)) + b(R(J), S(J)) so the complexity term cancels in h - h' and the contrast depends on surface form alone. Training is a two-stage curriculum: a label-free representation stage (triplet loss pulling twins together and pushing requirement-changing edits apart), then a small score-labeled calibration stage.

Frail-Ο„ is used on Synth-CD, where scores are continuous, no stratum is censored, and tau is well posed, and for the U-CCD control on real suites. Because T leaves C and the requirements fixed, the contrast h - h' exposes only the surface part of the embedding β€” which is why the contrast head is intrinsically complexity-orthogonal (verified on Synth-CD without any explicit residualization).

Featurizers and baselines

Featurizers (--featurizer): tfidf_svd (default, frozen, CPU), surface, hashing, openai_embed (text-embedding-3-small), hf_frozen / minilm / bge / hf:<model-id> (frozen sentence encoders), and the trainable hf_finetune (Frail-Ο€) and hf_lora.

Baselines (src/causal_difficulty/baselines/): surface features (GBT), declared complexity score, frozen twin-distance, Prompt Clarity / Embedding Entropy / Embedding Specificity-IEF (PromptSET), BERTPE, VAP3 (retrained to regress the break event), measured base correctness Y(J), amortized IRT, score predictor, embedding linear probe, text length.

3. Datasets

3.1 GlassGSM and GlassMATH

Both suites augment an original math benchmark with twins from three meaning-preserving operator families:

Family Operator tag What it does How it is built
Entity renaming entity_rename Replaces recognised person names with names from a neutral pool deterministic (data/math_surface_ops.py)
Sentence reordering sentence_reorder Reorders setup sentences; the question stays last deterministic (data/math_surface_ops.py)
Distractor injection math_irrelevant_context_m{m} Splices in setup sentences (no question) from a different GSM8K problem scripts/gsmic_adv_pipeline.py, math_adv_pipeline.py
GlassGSM (gsmic_adv) GlassMATH (math_adv)
Base problems 1,319 (GSM8K test) 4,999 (Hendrycks MATH test)
Pairs in the five dose files 32,858 122,847
Unique twins (dedup on twin text) 18,309 (13.9 / problem) 115,364 (23.1 / problem)
Declared complexity proxy C heuristic score from the problem text MATH difficulty level 1–5, mapped to complexity_score ∈ [0.05, 0.95]
Fragility vs original GSM-IC-style 3.7% β†’ 44.9% β€”
Public release reza2ei/GlassGSM on the Hub (download to release/glassgsm/) reza2ei/GlassMATH on the Hub (download to release/glassmath/)

Solvers. Reference solver gpt-4o-mini; cross-solver and cross-generator runs use Qwen3.7-plus and DeepSeek-v4-pro (and Llama-3.1-8B in some scripts). Solvers decode at temperature 0.2 with 5 samples per prompt; Y scores the modal extracted answer, so a twin can "break" a task by re-sampling alone β€” that floor is measured with identity twins (Step 8).

3.2 The multi-operator-mix corpus (the paper's benchmark)

Per-dose pipeline runs produce one file per dose. The paper's benchmark instead uses multi-operator-mix files: for each dose m, a file holds about m twins per base problem, split as evenly as possible across the families present for that problem. Each twin applies one operator. scripts/build_multiop_mix.py builds them without any new LLM calls by resampling audited multi-operator data:

  • --mix-twins m is the total number of twins per base problem, shared uniformly across sentence_reorder, entity_rename, math_irrelevant_context; the sampler is seeded and reproducible;
  • math_irrelevant_context has a per-twin injection size (number of stacked sentences, baked into sibling files named ..._m{size}_multi_...), and the resampler picks the size whose file most closely matches that family's share of m;
  • d_obs / d_wc are recomputed over exactly the sampled twins.

The five dose files of a suite are resamples of the same audited pool, not independent draws: treat them as the same problems seen at different perturbation budgets, and always split by base_id, never by row or by file.

data_audited_multiop_mix/{suite}_m{m}_multiop_mix_{gen}_labeled_pairs.jsonl
    suite in {gsmic_adv, math_adv}
    m     in {1, 3, 5, 7, 9}
    gen   in {4omini, qwen3p7plus, deepseekv4pro, llama}     # solver / generator tag
  • the pooled 5-dose run (the five files concatenated) is the source of the headline CCD table (pi + dose-response a/b);
  • the m = 9 slice alone is the source of the CRD orthogonality table and of the cross-model and solver-transfer protocols.

3.3 No-Op contamination audit

The No-Op requirement (a twin must not change the correct answer) must hold empirically, not just by construction. scripts/audit_contradictions_and_filter.py adjudicates every (base, twin) pair and writes a filtered copy of the benchmark (raw files are never modified). A pair is dropped if any of the following holds:

  1. LLM No-Op adjudication β€” an LLM adjudicator receives the base text, twin text, and gold answer and judges whether the perturbation changes or invalidates the correct answer (default gpt-4o-mini; the paper's pipeline used a stronger model with a second adjudicator). A "contradiction" verdict, or an unparsable response after retries, drops the pair.
  2. Entity-name collision β€” the injected distractor reuses a proper noun from the target problem (calendar words excluded).
  3. Gold-answer-number collision β€” the injected text contains the numeric value of the correct answer.
  4. Multi-question collision β€” the injected distractor contains its own ?.
  5. Unit-compatible quantity collision β€” flagged but not dropped: a second adjudicator pass over the full flagged pool found no actual contradiction.

Structural checks are deterministic (src/causal_difficulty/features/collision.py); the shared verdict-parsing library is scripts/contradiction_audit_lib.py. Audit reuse across doses and scoring models: audit_contradictions_dose_reuse.py, audit_crossmodel_reuse.py. Sample-level rate estimation: scripts/noop_verification.py, scripts/human_answer_preservation_audit.py.

3.4 Labeled-pair data format

Everything downstream consumes one JSONL of LabeledPair objects (src/causal_difficulty/schemas.py):

{
  "base_id": "gsmic_adv-42",
  "twin_id": "gsmic_adv-42-d7",
  "operator": "math_irrelevant_context_m9",
  "category": "math",
  "complexity": "medium",
  "complexity_score": 0.72,
  "base_text": "...",
  "twin_text": "...",
  "delta_per_model": {"gpt-4o-mini": 1.0},
  "delta_mean": 1.0,
  "d_obs": 0.35,
  "d_wc": 1.0,
  "y_base_mean": 1.0
}
  • operator carries the family and the dose (_m{k} suffix); eval/loading.py stamps the dose from the filename onto legacy m=1 rows that lack the suffix (otherwise the lowest dose would be read as dose 0 and flatten the dose-response fit);
  • delta_mean = Y(J) - Y(T(J)) for the scoring model; the per-observation break event B = 1{Y(J)=1, Y(T(J))=0} is derived from y_base_mean and the per-model deltas by eval/estimands.py (build_job_targets), which also computes per-job pi, dose-response (a, b), and the solvable-stratum restriction;
  • d_obs / d_wc are job-level D_avg / worst-case targets (Frail-Ο„ runs).

3.5 Synth-CD (synthetic mechanism checks)

scripts/synth.py / synth2.py generate synthetic jobs with known causal ground truth: two independent confounders C ~ Beta(2,2) (structural complexity) and E ~ Beta(2,2) (domain expertise), 8 domains, 12 operator types spanning cosmetic-only through dual-sensitive, with text length controlled to be independent of C and E. simulate.py / simulate2.py label twins with

delta(J, op) = max(0, mu_op (1 + C(J)) + zeta_op E(J) + eps)

calibrated so Spearman(D_avg, C) β‰ˆ Spearman(D_avg, E) β‰ˆ 0.60. Scores are continuous and no stratum is censored, so U-CCD is well posed here; it is used as a mechanism check (does the contrast head recover the expertise signal with near-zero |rho_C| without explicit residualization?), not as a headline setting. scripts/synth_frailpi.py + synth_frailpi_baselines.py provide the Bernoulli-break, dose-tiered variant used for the Synth-CD consistency table.

4. Repository layout 

src/causal_difficulty/
  models/
    fragility.py            Frail-Ο€: per-observation break-probability model
    fragility_encoder.py    Frail-Ο€ with joint encoder fine-tuning / LoRA
    sce.py                  Siamese Counterfactual Encoder (Frail-Ο„)
    losses.py               Huber, triplet, invariance R_inv losses
    train.py                two-stage SCE training loop
  encoders/                 hashing, hf_frozen, hf_finetune, hf_lora, openai_embed
  baselines/                bertpe, vap3, promptset, pi_adapters, frozen_distance,
                            amortized_irt, score_predictor, embedding_linear_probe,
                            length, common
  data/                     perturbations, math_surface_ops, filler_control,
                            synth_jobs, synth2_jobs, synth2_perturbations
  eval/
    estimands.py            pi, break events, dose parsing, job targets
    deconfound.py           cross-fitted residualization (CRD), orthogonality diagnostics
    reliability.py          parametric-bootstrap reliability ceiling
    splits.py               fingerprinted job-disjoint train/val/test splits
    metrics.py, irt.py, gsmic_adv_deconf.py, reports.py, loading.py
  features/                 surface features, text featurizers, probes, collision checks
  config.py                 YAML config loader (configs/*.yaml)
  llm_env.py, llm_cache.py  LLM client configuration + on-disk response cache
  schemas.py                LabeledPair and friends
scripts/                    experiment entry points and runbooks (see Β§7, Β§8)
configs/                    120+ run configurations (suite / dose / backbone / ablation)
release/glassgsm/           GlassGSM download location (hf download reza2ei/GlassGSM; not tracked)
tests/                      unit tests (89)
docs/fragility_tuning.md    ablation runbook (5 recipe levers)

5. Installation 

python -m venv .venv && source .venv/bin/activate
pip install -e ".[hf,openai,dev]"     # torch, transformers, sentence-transformers,
                                      # peft, openai, httpx, pytest
# or with uv:  uv sync --all-extras

Python β‰₯ 3.10. The released-data conversion script additionally needs pandas and pyarrow.

LLM access is needed only for dataset construction, auditing, ex-ante probe collection, and solver-transfer runs β€” not for training or evaluating on existing labeled pairs. Copy .env.example to .env and fill it in. src/causal_difficulty/llm_env.py selects the backend from LLM_BACKEND (openai | fireworks), falling back on whichever of LLM_API_KEY / FIREWORKS_API_KEY / OPENAI_API_KEY is set; point OPENAI_BASE_URL / LLM_BASE_URL at any OpenAI-compatible endpoint.

Hugging Face model weights download automatically on first use.

6. Quickstart (no API access needed)

Rebuild the GlassGSM labeled-pair files from the released parquet (download it first with hf download reza2ei/GlassGSM --repo-type dataset --local-dir release/glassgsm, which creates release/glassgsm/data/glassgsm/), then train and evaluate Frail-Ο€ on a CPU-friendly featurizer:

pip install -e ".[hf,openai,dev]" pandas pyarrow

# 1. released parquet -> data_audited_multiop_mix/*_labeled_pairs.jsonl
python scripts/release_to_labeled_pairs.py --suite glassgsm --solver gpt-4o-mini

# 2. CPU smoke run on the m=9 slice (frozen TF-IDF+SVD features, one seed)
python scripts/fragility_report.py \
  --dataset "$PWD/data_audited_multiop_mix/gsmic_adv_m9_multiop_mix_4omini_labeled_pairs.jsonl" \
  --featurizer tfidf_svd --seeds 0 --max-epochs 20 \
  --out-dir runs/smoke

# 3. unit tests
python -m pytest tests -q

--dataset accepts absolute paths, bare file names resolved under data/, a config name, a dose-family shorthand (gsmic_adv, math_adv), or a comma-separated list of files. Relative paths with a directory component are not resolved against the working directory β€” use $PWD/.... Step 2 prints the test-fold Spearman for the break rate and the dose-response parameters and writes report.md / summary.json into --out-dir. The numbers of this CPU smoke run are not the paper's: the paper's Frail-Ο€ uses the fine-tuned encoder on a GPU (Step 5).

7. Reproduction guide

All commands assume the repo root and an activated environment. Dry-run every pipeline first (no API calls):

python scripts/gsmic_adv_pipeline.py --dry-run --max-problems 5 --multi-operator

Step 1 β€” build twins (per dose, multi-operator)

For each dose m in {1,3,5,7,9} (GlassGSM; GlassMATH is analogous β€” see scripts/run_gsmic_full_doses.sh / scripts/run_math_full_doses.sh):

python scripts/gsmic_adv_pipeline.py \
  --model gpt-4o-mini \
  --max-problems 1319 --max-distractors 20 --distractors-per-twin "$m" \
  --multi-operator --surface-twins 10 \
  --concurrency 32 \
  --cache-dir cache/gsmic_adv_m${m}_4omini \
  --output data_multi_full/gsmic_adv_m${m}_multi_4omini_labeled_pairs.jsonl
  • --multi-operator generates the three families in one pass;
  • --distractors-per-twin $m is the injection size (stacked distractor sentences);
  • --surface-twins 10 is the number of twins per surface operator per problem;
  • responses are cached under --cache-dir, so re-runs and lower doses are incremental;
  • GlassMATH: python scripts/math_adv_pipeline.py --model gpt-4o-mini --no-max-problems --max-distractors 20 --distractors-per-twin $m --multi-operator --surface-twins 10 --output data_math_full/math_adv_m${m}_multi_4omini_labeled_pairs.jsonl (the difficulty-level β†’ complexity_score mapping is built in).

Other solvers/generators are different --model / --backend values; the generator tag is encoded in the output file suffix (4omini, qwen3p7plus, deepseekv4pro, llama).

Step 2 β€” No-Op audit 

python scripts/audit_contradictions_and_filter.py \
  --config gsmic_adv_m${m}_4omini \
  --pairs-path data_multi_full/gsmic_adv_m${m}_multi_4omini_labeled_pairs.jsonl \
  --out-dir data_audited \
  --concurrency 256 --model gpt-4o-mini

Step 3 β€” build the multi-operator-mix datasets 

python scripts/build_multiop_mix.py \
  --input  data_audited/gsmic_adv_m9_multi_4omini_labeled_pairs.jsonl \
  --output data_audited_multiop_mix/gsmic_adv_m9_multiop_mix_4omini_labeled_pairs.jsonl \
  --mix-twins 9 --seed 42

Do this for m = 1,3,5,7,9, for GlassMATH, and for other generators. Configs for the resulting files are configs/{gsmic_adv,math_adv}_m{m}_multiop_mix_{gen}.yaml and the pooled *_pooled_multiop_mix_*.yaml. (To skip Steps 1–3 for GlassGSM, use the Quickstart conversion of the released data.)

Step 4 β€” (optional) ex-ante base probes

The text_probe protocol adds features from repeated samples of the base prompt only (no twin ever runs β€” a different, model-access claim):

python scripts/collect_base_probes.py \
  --dataset gsmic_adv --model gpt-4o-mini --n-samples 8 \
  --out-dir outputs_probes/gsmic_adv

Produces outputs_probes/gsmic_adv/base_probes.json (richer n=24/n=48 variants are the "richer probes" ablation of docs/fragility_tuning.md). Gate check: scripts/base_probe_pilot.py.

Step 5 β€” C1: predict CCD from text (Frail-Ο€)

scripts/fragility_report.py is the single headline entry point: it loads the labeled pairs, makes a fingerprinted job-disjoint three-way split, trains Frail-Ο€ inside the process on the train fold only (there is no --checkpoint path that could smuggle in test-fit weights), early-stops on the validation fold, and reports β€” over several seeds β€” the test-fold Spearman for pi and the dose-response parameters a/b, plus surface-feature / declared-complexity / measured-Y(J) references and (with --include-bertpe-vap3) the supervised baselines. Divide by the suite's reliability ceiling (Step 7) to get the paper's normalized numbers. It also writes the CRD orthogonality diagnostics (Step 6).

Headline runs, mirroring scripts/run_multiop_mix_conditional_fragility.sh (GPU):

MIX="$PWD/data_audited_multiop_mix"
POOLED="$MIX/gsmic_adv_m1_multiop_mix_4omini_labeled_pairs.jsonl,$MIX/gsmic_adv_m3_multiop_mix_4omini_labeled_pairs.jsonl,$MIX/gsmic_adv_m5_multiop_mix_4omini_labeled_pairs.jsonl,$MIX/gsmic_adv_m7_multiop_mix_4omini_labeled_pairs.jsonl,$MIX/gsmic_adv_m9_multiop_mix_4omini_labeled_pairs.jsonl"

# A) pooled 5-dose (main CCD table) β€” Frail-Ο€ = hf_finetune encoder
python scripts/fragility_report.py \
  --dataset "$POOLED" --seeds 0,1,2,3,4 --device cuda \
  --featurizer hf_finetune --protocol both \
  --probe-file outputs_probes/gsmic_adv/base_probes.json \
  --include-bertpe-vap3 \
  --jobs-per-batch 64 --encoder-microbatch 16 \
  --encoder-lr 2e-5 --head-lr 1e-3 --encoder-max-epochs 20 --encoder-patience 5 \
  --dump-predictions \
  --out-dir runs/multiop_mix/pooled_5dose_hf_finetune

# B) single-dose m=9 (CRD / cross-model protocol)
python scripts/fragility_report.py \
  --dataset "$MIX/gsmic_adv_m9_multiop_mix_4omini_labeled_pairs.jsonl" \
  --seeds 0,1,2,3,4 --device cuda --featurizer hf_finetune --protocol both \
  --probe-file outputs_probes/gsmic_adv/base_probes.json \
  --include-bertpe-vap3 --dump-predictions \
  --out-dir runs/multiop_mix/m9_only_hf_finetune

Notes:

  • --protocol text | text_probe | both; text_probe requires --probe-file. The two protocols are reported separately and never merged;
  • --dump-predictions writes per-seed held-out predictions consumed by the curation, ensembling, and per-family evaluations;
  • the paper's representation ablation is the --featurizer axis; the five training-recipe levers are in docs/fragility_tuning.md.

Step 6 β€” C2: CRD orthogonality and U-CCD controls

  • CRD orthogonality (real suites, Q = pi) comes from the same fragility_report.py runs: per seed it reports Corr(target, C), Corr(pred, C), Corr(pred, C^2) (the orthogonality_* block, eval/deconfound.py).

  • U-CCD (Frail-Ο„) and the full baseline sweep:

    python scripts/paper_supports_real.py --config gsmic_adv_m9_multiop_mix_4omini --seed 7
python scripts/paper_supports_real.py --config gsmic_adv_m9_multiop_mix_4omini \
    --estimand d_avg --seed 7          # "the unconditional target is unlearnable"

  • CCD vs CRD across solvers: scripts/solver_crd_vs_ccd_transfer.py --suite gsmic_adv.

Step 7 β€” reliability ceilings

Spearman numbers in the paper are normalized by each suite's reliability ceiling (0.969 GlassGSM, 0.972 GlassMATH), computed by parametric bootstrap with a cross-validated empirical-Bayes prior:

python scripts/noise_ceiling.py --config gsmic_adv_m9_4omini --n-repeats 30
python scripts/noise_ceiling.py --config math_adv_m9_4omini  --n-repeats 30

Per-slice ceilings: scripts/slice_ceilings.py. Always report the raw ρ_S alongside normalized values β€” normalized numbers depend on the ceiling estimator and slice.

Step 8 β€” placebo / identity-twin floor

Because solver outputs are sampled, B can fire with no true effect. The floor is measured by re-solving the unperturbed bases of solvable-stratum items:

python scripts/null_twin_noise_floor.py --suite gsmic_adv   # paper: 1.1%
python scripts/null_twin_noise_floor.py --suite math_adv    # paper: 2.5%

Complementary control: scripts/null_perturbation_control.py --probe-file outputs_probes/gsmic_adv/base_probes.json.

Step 9 β€” downstream utility: top-k curation

Rank the audited holdout by each predictor and recall truly fragile jobs (pi > 0.05) at budgets k in {5,10,20,50}%:

# reads Frail-Ο€'s dumped held-out predictions from the pooled 5-dose run
python scripts/downstream_utility_pi.py
python scripts/downstream_curation.py --config gsmic_adv_m9_4omini --seeds 7 42 123

Step 10 β€” solver transfer and cross-generator checks

The stratum is defined by the solver's own Y(J), so varying the solver tests how far the estimand travels:

python scripts/solver_pi_agreement.py \
  --configs gsmic_adv_m9_4omini gsmic_adv_m9_qwen3p7plus gsmic_adv_m9_deepseekv4pro
python scripts/cross_model_transfer.py \
  --configs gsmic_adv_m9_4omini gsmic_adv_m9_qwen3p7plus gsmic_adv_m9_deepseekv4pro
python scripts/solver_crd_vs_ccd_transfer.py --suite gsmic_adv

Cross-generator checks (Qwen / DeepSeek-authored twins, gpt-4o-mini scoring) reuse the same entry points on the *_qwen3p7plus / *_deepseekv4pro mix files; BERTPE/VAP3 on those slices: scripts/run_bertpe_vap3_crossmodel.sh.

Step 11 β€” Synth-CD mechanism checks 

python scripts/synth2.py --seed 42                 # jobs + twins (offline)
python scripts/simulate2.py                        # causal DGP labels + oracle D^perp
python scripts/paper_supports_synth2.py --config synth2
python scripts/paper_supports_synth2.py --config synth2_gpt --backbone openai_embed
python scripts/synth_frailpi.py --n-jobs 500 --twins-per-dose 5 --seed 7 \
  --out data/synth_frailpi_labeled_pairs.jsonl
python scripts/fragility_report.py --dataset data/synth_frailpi_labeled_pairs.jsonl
python scripts/synth_frailpi_baselines.py

The contrast head should reach the best Spearman with the lowest |rho_C| without explicit residualization.

Step 12 β€” baselines 

python scripts/run_baselines.py --config gsmic_adv_m9_multiop_mix_4omini
python scripts/run_baselines.py --config math_adv_m9_multiop_mix_4omini

The PromptSET / frozen-geometry baselines retargeted to the break-event pi: scripts/run_missing_pi_baselines.py. Per-family baseline table: scripts/ccd_per_family_baselines.py.

Step 13 β€” per-category / per-family breakdowns 

python scripts/ccd_per_category.py --files "$POOLED" --label gsmic_adv
python scripts/ccd_per_family.py   --files "$POOLED" --label gsmic_adv
python scripts/ccd_per_family_predict.py ...        # evaluate dumped predictions

8. Orchestration scripts

Shell runbooks that chain the entry points above as the paper's runs were launched (multi-GPU where noted; run from the repository root):

Script What it runs
run_multiop_mix_conditional_fragility.sh headline C1+C2: pooled 5-dose + m=9 Frail-Ο€, 5 seeds, BERTPE/VAP3
run_multiop_mix_m9_full_baselines.sh m=9 full baseline sweep (paper_supports_real.py)
run_multiop_mix_pooled_sce.sh pooled 5-dose Frail-Ο„ (d_avg/d_wc) + baselines
run_multiop_mix_sce_dosage_sweep.sh all doses Γ— 5 seeds, Frail-Ο„ + baselines
run_multiop_mix_frailpi_contrast_ablation.sh --lambda-contrast sweep
run_multiop_mix_lambdainv_ablation.sh invariance-regularizer lambda_inv sweep
run_multiop_mix_richer_probes.sh richer ex-ante probes (n=24) ablation
run_fragility_tuning_all.sh + docs/fragility_tuning.md 5 recipe levers (RoBERTa-large, lr/batch, pooling, …)
run_gsmic_full_doses.sh / run_math_full_doses.sh full per-dose twin generation + audit
run_bertpe_vap3_crossmodel.sh BERTPE/VAP3 on cross-generator slices
run_train_eval_synthetic_3seeds.sh Synth-CD train/eval
run_c2_diagnostics.sh C2 diagnostics on pi / surface-proxy D^perp

scripts/ also holds many one-off launchers, resume scripts, and analysis helpers from the experiment campaign; the table above lists the ones tied to paper results.

9. Configuration reference

YAML configs in configs/, loaded by name via causal_difficulty.config.load_config. A mix-corpus config (configs/gsmic_adv_m9_multiop_mix_4omini.yaml):

seed: 42
paths:
  data_dir: data_audited_multiop_mix
  outputs_dir: outputs/gsmic_adv_m9_multiop_mix_4omini
  cache_dir: cache/gsmic_adv_m9_multiop_mix_4omini
  labeled_pairs: data_audited_multiop_mix/gsmic_adv_m9_multiop_mix_4omini_labeled_pairs.jsonl
dataset:
  n_jobs: 200
  twins_per_job: 20
encoder:
  backbone: openai_embed          # hashing | hf_frozen | hf_finetune | hf_lora | openai_embed
  hf_frozen: { model: BAAI/bge-small-en-v1.5, batch_size: 256 }
  openai_embed: { model: text-embedding-3-small, max_concurrency: 16 }
model:
  hidden_dim: 64
  complexity_dim: 32
  contrast_context: surface_pair  # surface_pair | pair | none
  dropout: 0.25
  ex_ante_n_layers: 2
train:
  representation: { epochs: 200, lr: 0.001, batch_size: 64, triplet_margin: 0.5 }
  calibration: { epochs: 1000, lr: 0.0001, weight_decay: 0.01, batch_size: 32,
                 lambda_inv: 0.0, huber_delta: 0.1 }
eval:
  fragile_threshold: 0.05
  holdout_frac: 0.3
  stratified_split: true

Frail-Ο€'s recipe is set by the fragility_report.py flags shown in Step 5 (--encoder-lr 2e-5 --head-lr 1e-3 --hidden-dim 256 ...); the model: / train: blocks above configure the Frail-Ο„ / SCE path.

10. Outputs and tests

  • runs/{run_id}/ β€” one directory per experiment: report.md (human-readable tables), summary.json (full metrics, per seed), prediction dumps (--dump-predictions), logs; a one-line summary is appended to runs/runs.jsonl.
  • outputs_fragility/{dataset}/ β€” default fragility_report.py output directory.
  • outputs_probes/{suite}/base_probes.json β€” ex-ante probe features.
  • data_audited*/, data_audited_multiop_mix/ β€” audited benchmarks and the mix corpus (built or converted, not committed).
  • cache/ β€” LLM response caches (safe to delete; re-runs re-call the API).

These directories and model checkpoints are git-ignored.

python -m pytest tests -q        # 89 tests: estimands, deconfound, collision audit,
                                 # filler control, math-adv, human audit, units

11. Reproducibility notes

  • Stratum membership is a measured quantity. Y(J) is the modal answer of 5 samples at temperature 0.2; the identity-twin floor (Step 8) quantifies the re-sampling noise that leaks into pi.
  • Doses are stamped from filenames (_m{k}_); the legacy m=1 math_irrelevant_context tag (no suffix) is normalized at load time.
  • Splits are fingerprinted and job-disjoint across train/val/test; checkpoints carry a sidecar of their train job-ids and evaluators refuse to score a checkpoint that overlaps their holdout. Use the same --seeds and --dataset string when comparing featurizers β€” folds are deterministic in (--dataset, seed).
  • Ceiling-normalized numbers are estimator-sensitive. The paper's 0.969 / 0.972 come from the parametric-bootstrap estimator of Step 7 on the audited corpora behind the tables. Re-running on a different slice (single dose vs pooled), probe file, or estimator prior changes both the ceiling and the achievable ρ_S; compare raw ρ_S on identical splits, and expect absolute values from re-runs to differ from the paper's tables, especially for the single-dose m = 9 slice (the paper's own m = 9 solver-transfer diagonal is 0.431 on GlassGSM).
  • The ex-ante probe file matters. text_probe results depend on the probe sample count and draw (--n-samples); the probe file behind the paper's runs is not part of this repository.
  • Complexity proxies are declared and external; CRD is orthogonal to that proxy only. For GlassGSM the released pipeline computes complexity_score from the problem text (question numbers, keywords, sentences), which correlates with length. For GlassMATH it is the problem-setter level.
  • Per-family structure. Each mix-corpus twin applies a single operator. On the released GlassGSM data (gpt-4o-mini, unique twins of solved bases) the break rate is 13.8% for sentence reordering, 13.0% for distractor injection, and 1.5% for entity renaming β€” the last is close to the 1.1% identity-twin floor, so renaming is a low-signal family and most of the fragility comes from reordering and injection. Per-family breakdowns: Step 13.

12. Citation, license, third-party material 

@misc{rezaei2026fragilityreduciblehardnessconditional,
      title={Fragility Is Not Reducible to Hardness: Conditional Causal Difficulty for LLMs},
      author={Mohammad Reza Rezaei and Dingxian Wang and Nikhil Verma and Sajjad Pakdamansavoji and Andrew Rabinovich and Zhao Chen},
      year={2026},
      eprint={2611.02065},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2611.02065},
}

The package is released under the MIT license (pyproject.toml). GlassGSM / GlassMATH contain text derived from GSM8K (MIT, OpenAI) and the Hendrycks MATH benchmark (MIT); see NOTICE.md in the dataset repos for the third-party notices and license text.