# Conditional Causal Difficulty for LLMs: full documentation

> Short version: see the [README](../README.md). This is the long reference.

Code for **"Fragility Is Not Reducible to Hardness: Conditional Causal Difficulty
for LLMs"** ([arXiv:2611.02065](https://arxiv.org/abs/2611.02065)).

LLMs show large, seemingly arbitrary performance drops when a task is rewritten in
ways that leave its requirements untouched. We treat task difficulty as a **causal**
quantity — the treatment effect of meaning-preserving perturbations on a model — and
ask which tasks are *anomalously fragile*, beyond what their complexity predicts.

This repository contains everything needed to reproduce the paper:

- the estimands **CCD** (conditional causal difficulty) and **CRD**
  (complexity-residualized difficulty), with cross-fitted residualization;
- the predictors **Frail-π** (CCD from text) and **Frail-τ** (the unconditional
  contrast, used for Synth-CD and as a control);
- the **GlassGSM** and **GlassMATH** benchmarks: construction pipelines, the No-Op
  contamination audit, and the multi-operator-mix corpora;
- the Synth-CD mechanism-check suite and every baseline (BERTPE, VAP3, PromptSET
  measures, surface features, IRT, …);
- the reliability-ceiling, identity-twin-floor, curation, and solver-transfer analyses.

> **Naming.** The benchmarks are called **GlassGSM** (twins of GSM8K) and
> **GlassMATH** (twins of Hendrycks MATH). Older config and file names still use
> `gsmic_adv` (= GlassGSM) and `math_adv` (= GlassMATH). They are the same thing.

---

## Headline results (from the paper)

Test-fold Spearman, normalized by each suite's reliability ceiling (0.969 GlassGSM,
0.972 GlassMATH), mean ± sd over five seeds (Table 2 of the paper):

| Method (predicting CCD, `π`) | GlassGSM | GlassMATH |
|---|---|---|
| **Frail-π** (fine-tuned encoder) | **0.613 ± 0.023** (raw ρ_S 0.594) | **0.557 ± 0.023** (raw ρ_S 0.541) |
| VAP3 | 0.444 ± 0.049 | 0.463 ± 0.010 |
| BERTPE | 0.420 ± 0.051 | 0.451 ± 0.015 |
| Surface features (GBT) | 0.171 ± 0.023 | 0.401 ± 0.014 |
| Declared complexity only | 0.207 ± 0.060 | 0.285 ± 0.026 |

Other headline findings:

- **Fragility is a different quantity from hardness.** GlassGSM raises fragility from
  3.7% (GSM-IC-style distractors) to 44.9% on the same problems while keeping the
  answer key unchanged. The measured base pass rate `Ȳ(J)` is *negatively* correlated
  with CCD.
- **CRD is orthogonal to the declared complexity proxy** (|Corr(target, C)| ≤ 0.06 on
  the m=9 run), and the contrast architecture is complexity-orthogonal even without
  explicit residualization on Synth-CD (Frail-τ ρ_S = 0.617 at |ρ_C| = 0.028).
- **Actionable for curation.** On GlassGSM, selecting the top 5% of problems by
  Frail-π gives Recall@5% = 0.099 of truly fragile items vs 0.05 for random and 0.100
  for an oracle ranking.
- **Partial, solver-dependent transfer.** Training Frail-π on one solver's CCD and
  testing on another's transfers only partially, as expected because the stratum is
  defined by each solver's own `Y(J)`.
- **Measurement floors.** The identity-twin floor (break rate of re-solved, unperturbed
  bases) is 1.1% on GlassGSM and 2.5% on GlassMATH.

See [§11](#11-reproducibility-notes) for what to expect when re-running.

---

## Table of contents

1. [The estimands: CCD and CRD](#1-the-estimands-ccd-and-crd)
2. [Models: Frail-π and Frail-τ](#2-models-frail-π-and-frail-τ)
3. [Datasets](#3-datasets)
4. [Repository layout](#4-repository-layout)
5. [Installation](#5-installation)
6. [Quickstart (no API access needed)](#6-quickstart-no-api-access-needed)
7. [Reproduction guide](#7-reproduction-guide)
8. [Orchestration scripts](#8-orchestration-scripts)
9. [Configuration reference](#9-configuration-reference)
10. [Outputs and tests](#10-outputs-and-tests)
11. [Reproducibility notes](#11-reproducibility-notes)
12. [Citation, license, third-party material](#12-citation-license-third-party-material)

---

## 1. The estimands: CCD and CRD

A *task* (prompt) is a token sequence `J`. An LLM maps it to a response `O(J)`, which
a quality-scoring function `phi` collapses to a scalar

```
Y(J) = phi(O(J), J) in [0,1]
```

(For the math suites this is exact match on the extracted modal answer over 5 samples
at temperature 0.2.) A **meaning-preserving perturbation operator (MPPO)** `T`
rewrites a task into a *twin* `T(J)` with the same requirements and the same gold
answer but a different surface form. Let `P` be a distribution over such operators.

**Per-instance contrast.** Each task has a control score `Y(J)` and a treatment score
`Y(T(J))`; the per-instance effect is

```
Delta(J, T) = Y(J) - Y(T(J)),        tau(J) = E_{T ~ P}[ Delta(J, T) | J ]
```

`T` is drawn independently of every attribute of `J`, and `Delta` compares the *same*
task under two randomly assigned surface realizations — a randomized experiment
*within* the item. Its contrast is unbiased without adjustment; the complexity
back-door `S <- C -> Y` only opens for *observational* comparisons across prompts,
which is what every `Y(J)`-predictor baseline estimates.

### Why the unconditional average (U-CCD) is the wrong target

A meaning-preserving rewrite can only be *observed* to hurt a task the model already
solves. On the stratum `Y(J) = 0`, every realized contrast satisfies
`Delta(J, T) <= 0`: a task the model cannot solve at all is recorded as maximally
*robust* rather than untested — the stratum is censored, not informative. The
unconditional average `D_avg(J) = E_T[Delta(J, T)]` therefore mixes a live effect with
a structural zero, and empirically it is unlearnable on the real suites: against U-CCD
on GlassGSM every method, ours included, is weak (best ρ_S = 0.162, surface length,
vs Frail-π's 0.594 on CCD). U-CCD results are retained as a control
(`paper_supports_real.py --estimand d_avg`).

### CCD — conditional causal difficulty

Restrict the estimand to the **solvable stratum** (`Y(J) = 1`) using the per-twin
**break event**

```
B(J, T) = 1{ Y(J) = 1 and Y(T(J)) = 0 }
```

On this stratum `Delta(J, T) = B(J, T)` exactly, so CCD is the treatment effect
evaluated where it is non-degenerate:

```
pi(J) = E_{T ~ P}[ B(J, T) | Y(J) = 1 ]
```

`pi(J)` — the **conditional causal difficulty** — is the fraction of the twin family
that breaks a task the solver solved. It is a principal-stratum quantity
(stratification on the control potential outcome). The paired design makes it the
tractable case: stratum membership is *observed* (the base and every twin are actually
run), and conditioning on `Y(J)` opens no selection path because `T` is independent
of `J`.

**Dose response.** A twin's *dose* `m in {1, 3, 5, 7, 9}` indexes the perturbation
budget of a corpus file (see [§3](#3-datasets)). Fragility is summarized by a logistic
dose response

```
logit Pr[ B(J, T) = 1 | m ] = a_J + b_J (m - m_bar)
```

where `a_J` is fragility at typical dose and `b_J` the degradation rate. Frail-π
learns the per-perturbation break probability and aggregates it *analytically* to
`pi` and to `(a_J, b_J)`, instead of regressing a job-level scalar.

### CRD — complexity-residualized difficulty

Harder tasks tend to be more phrasing-sensitive, so intrinsic complexity `C` is an
**effect modifier**, not a confounder: the raw `pi` of a task is complexity-inflated.
To isolate *anomalous* fragility — the part the declared complexity cannot account
for — we residualize `C` out. Fix a base quantity `Q in {tau, pi}`, a *declared*
complexity observable `C`, and a task population `p` (restricted to `Y(J) = 1` when
`Q = pi`). With `m_{Q,C,p}(c) = E[ Q(J') | C(J') = c ]`:

```
D^perp_{Q,C,p}(J) = Q(J) - m_{Q,C,p}( C(J) )
```

Properties (Proposition 1 of the paper):

- `D^perp` is the `L2` projection residual of `Q` onto measurable functions of the
  *declared* `C`; hence `E[D^perp] = 0` and `D^perp` is uncorrelated with *every*
  function of `C`.
- It is estimated by **cross-fitted residualization**: regress the measured `Q_hat` on
  the complexity proxy *out-of-fold* (folds shared with the main evaluation) to get
  `m_hat`, and set `D^perp_hat = Q_hat - m_hat`. This removes own-observation leakage
  and is consistent whenever `Q_hat -> Q` and the complexity regression is consistent.
- `D^perp = 0` for a task exactly as fragile as the typical same-`C` task;
  `D^perp > 0` for anomalously fragile ones. Orthogonality is to the *declared* proxy
  only — a different `C` would give a different CRD.

The declared proxies are suite-specific and external to the pipeline: the
problem-setter-assigned competition level for GlassMATH, and a complexity score
computed from the problem text for GlassGSM (see [§11](#11-reproducibility-notes)).
Implementation: `src/causal_difficulty/eval/deconfound.py` (`cross_fitted_residual`,
`orthogonality_diagnostics`, `stratified_rank_correlation`), exercised by
`scripts/fragility_report.py` (orthogonality block), `scripts/paper_supports_real.py`,
and `scripts/solver_crd_vs_ccd_transfer.py`.

---

## 2. Models: Frail-π and Frail-τ

The repository contains two model families. They share losses and utilities but are
trained by different entry points, so it matters which one a script uses.

### Frail-π — break-probability model (the headline predictor for CCD)

Implemented in `models/fragility.py` (frozen features + MLP) and
`models/fragility_encoder.py` (joint encoder fine-tuning / LoRA), driven by
`scripts/fragility_report.py`.

It predicts the probability that a *single* perturbation at a given dose breaks a given
task,

```
p(job, m) = sigmoid( f([ x_job, dose_features(m) ]) )
```

and is trained with binary cross-entropy on **every (job, twin) observation** rather
than on a job-level scalar. Job-level estimands are then computed from `p` in closed
form: `pi` is `p` averaged over the dose composition of the evaluation design,
`(a, b)` come from a least-squares fit of `logit p(job, m)` over doses, and
worst-case / quantile targets at budget `K` follow from `1 - (1 - p)^K` and the
binomial tail. One model therefore serves every estimand at every budget.

- **Paper configuration**: `--featurizer hf_finetune` fine-tunes a pretrained encoder
  (DeBERTa-v3-base by default) end to end, with encoder LR `2e-5` and head LR `1e-3`.
  Frozen encoders and LoRA are the ablations in the paper.
- **Contrast supervision is optional.** `--lambda-contrast` (default `0.0`) adds a
  within-job contrast term between two twins' *predicted* break probabilities. At the
  default, training is the plain per-observation BCE objective.
- **Other objectives**: `--loss-mode {bce, soft_rank, reliability_weighted}`.
- **Two protocols, never merged**: `text` (problem statement only) and `text_probe`
  (adds statistics of repeated solver samples of the *base* prompt — requires model
  access; see Step 4).
- The CRD prediction subtracts the cross-fitted complexity regression from the
  prediction, `D^perp_hat(J) = D_hat(J) - m_hat_pred(C(J))`.

### Frail-τ / SCE — Siamese Counterfactual Encoder (unconditional contrast)

Implemented in `models/sce.py`, `models/train.py`, `models/losses.py`
(`scripts/train.py`, `scripts/evaluate.py`, `scripts/paper_supports_*.py`). A shared
encoder embeds the base `J` and the twin `T(J)` into `h` and `h'`; their difference
feeds a **contrast head**, while a **single-tower ex-ante head** predicts difficulty
when no twin exists:

```
Delta_hat(J, T) = psi_rho( h - h' )        # contrast head (per-observation supervision)
D_hat(J)        = g_omega( h )             # ex-ante head (deployable, base text only)

L = sum_T ell_delta( Delta_hat(J,T), Delta(J,T) )
  + ell_delta( g_omega(h), D^obs(J) )
  + lambda * R_inv(theta; J)
```

`R_inv = E[ || Pi_C h - Pi_C h' ||^2 ]` is an **invariance regularizer** with a learned
complexity-subspace projection `Pi_C`, encouraging `h = c(C(J)) + b(R(J), S(J))` so the
complexity term cancels in `h - h'` and the contrast depends on surface form alone.
Training is a two-stage curriculum: a label-free representation stage (triplet loss
pulling twins together and pushing requirement-changing edits apart), then a small
score-labeled calibration stage.

Frail-τ is used on **Synth-CD**, where scores are continuous, no stratum is censored,
and `tau` is well posed, and for the U-CCD control on real suites. Because `T` leaves
`C` and the requirements fixed, the contrast `h - h'` exposes only the surface part of
the embedding — which is why the contrast head is *intrinsically* complexity-orthogonal
(verified on Synth-CD without any explicit residualization).

### Featurizers and baselines

**Featurizers** (`--featurizer`): `tfidf_svd` (default, frozen, CPU), `surface`,
`hashing`, `openai_embed` (text-embedding-3-small), `hf_frozen` / `minilm` / `bge` /
`hf:<model-id>` (frozen sentence encoders), and the trainable `hf_finetune`
(**Frail-π**) and `hf_lora`.

**Baselines** (`src/causal_difficulty/baselines/`): surface features (GBT), declared
complexity score, frozen twin-distance, Prompt Clarity / Embedding Entropy /
Embedding Specificity-IEF (PromptSET), BERTPE, VAP3 (retrained to regress the break
event), measured base correctness `Y(J)`, amortized IRT, score predictor, embedding
linear probe, text length.

---

## 3. Datasets

### 3.1 GlassGSM and GlassMATH

Both suites augment an original math benchmark with twins from **three
meaning-preserving operator families**:

| Family | Operator tag | What it does | How it is built |
|---|---|---|---|
| Entity renaming | `entity_rename` | Replaces recognised person names with names from a neutral pool | deterministic (`data/math_surface_ops.py`) |
| Sentence reordering | `sentence_reorder` | Reorders setup sentences; the question stays last | deterministic (`data/math_surface_ops.py`) |
| Distractor injection | `math_irrelevant_context_m{m}` | Splices in setup sentences (no question) from a *different* GSM8K problem | `scripts/gsmic_adv_pipeline.py`, `math_adv_pipeline.py` |

| | GlassGSM (`gsmic_adv`) | GlassMATH (`math_adv`) |
|---|---|---|
| Base problems | 1,319 (GSM8K test) | 4,999 (Hendrycks MATH test) |
| Pairs in the five dose files | 32,858 | 122,847 |
| Unique twins (dedup on twin text) | 18,309 (13.9 / problem) | 115,364 (23.1 / problem) |
| Declared complexity proxy `C` | heuristic score from the problem text | MATH difficulty level 1–5, mapped to `complexity_score ∈ [0.05, 0.95]` |
| Fragility vs original GSM-IC-style | 3.7% → 44.9% | — |
| Public release | [`reza2ei/GlassGSM`](https://huggingface.co/datasets/reza2ei/GlassGSM) on the Hub (download to `release/glassgsm/`) | [`reza2ei/GlassMATH`](https://huggingface.co/datasets/reza2ei/GlassMATH) on the Hub (download to `release/glassmath/`) |

**Solvers.** Reference solver `gpt-4o-mini`; cross-solver and cross-generator runs use
Qwen3.7-plus and DeepSeek-v4-pro (and Llama-3.1-8B in some scripts). Solvers decode at
temperature 0.2 with 5 samples per prompt; `Y` scores the **modal** extracted answer,
so a twin can "break" a task by re-sampling alone — that floor is measured with
identity twins (Step 8).

### 3.2 The multi-operator-mix corpus (the paper's benchmark)

Per-dose pipeline runs produce one file per dose. The paper's benchmark instead uses
**multi-operator-mix** files: for each dose `m`, a file holds about `m` twins per base
problem, split as evenly as possible across the families present for that problem.
Each twin applies one operator. `scripts/build_multiop_mix.py` builds them **without
any new LLM calls** by resampling audited multi-operator data:

- `--mix-twins m` is the **total** number of twins per base problem, shared uniformly
  across `sentence_reorder`, `entity_rename`, `math_irrelevant_context`; the sampler is
  seeded and reproducible;
- `math_irrelevant_context` has a per-twin injection size (number of stacked sentences,
  baked into sibling files named `..._m{size}_multi_...`), and the resampler picks the
  size whose file most closely matches that family's share of `m`;
- `d_obs` / `d_wc` are recomputed over exactly the sampled twins.

The five dose files of a suite are *resamples of the same audited pool*, not
independent draws: treat them as the same problems seen at different perturbation
budgets, and **always split by `base_id`, never by row or by file**.

```
data_audited_multiop_mix/{suite}_m{m}_multiop_mix_{gen}_labeled_pairs.jsonl
    suite in {gsmic_adv, math_adv}
    m     in {1, 3, 5, 7, 9}
    gen   in {4omini, qwen3p7plus, deepseekv4pro, llama}     # solver / generator tag
```

- the **pooled 5-dose** run (the five files concatenated) is the source of the
  headline CCD table (`pi` + dose-response `a`/`b`);
- the **`m = 9` slice alone** is the source of the CRD orthogonality table and of the
  cross-model and solver-transfer protocols.

### 3.3 No-Op contamination audit

The No-Op requirement (a twin must not change the correct answer) must hold
*empirically*, not just by construction. `scripts/audit_contradictions_and_filter.py`
adjudicates **every** (base, twin) pair and writes a filtered copy of the benchmark
(raw files are never modified). A pair is dropped if **any** of the following holds:

1. **LLM No-Op adjudication** — an LLM adjudicator receives the base text, twin text,
   and gold answer and judges whether the perturbation changes or invalidates the
   correct answer (default `gpt-4o-mini`; the paper's pipeline used a stronger model
   with a second adjudicator). A "contradiction" verdict, or an unparsable response
   after retries, drops the pair.
2. **Entity-name collision** — the injected distractor reuses a proper noun from the
   target problem (calendar words excluded).
3. **Gold-answer-number collision** — the injected text contains the numeric value of
   the correct answer.
4. **Multi-question collision** — the injected distractor contains its own `?`.
5. **Unit-compatible quantity collision** — flagged but **not** dropped: a second
   adjudicator pass over the full flagged pool found no actual contradiction.

Structural checks are deterministic (`src/causal_difficulty/features/collision.py`);
the shared verdict-parsing library is `scripts/contradiction_audit_lib.py`. Audit reuse
across doses and scoring models: `audit_contradictions_dose_reuse.py`,
`audit_crossmodel_reuse.py`. Sample-level rate estimation:
`scripts/noop_verification.py`, `scripts/human_answer_preservation_audit.py`.

### 3.4 Labeled-pair data format

Everything downstream consumes one JSONL of `LabeledPair` objects
(`src/causal_difficulty/schemas.py`):

```json
{
  "base_id": "gsmic_adv-42",
  "twin_id": "gsmic_adv-42-d7",
  "operator": "math_irrelevant_context_m9",
  "category": "math",
  "complexity": "medium",
  "complexity_score": 0.72,
  "base_text": "...",
  "twin_text": "...",
  "delta_per_model": {"gpt-4o-mini": 1.0},
  "delta_mean": 1.0,
  "d_obs": 0.35,
  "d_wc": 1.0,
  "y_base_mean": 1.0
}
```

- `operator` carries the family **and** the dose (`_m{k}` suffix); `eval/loading.py`
  stamps the dose from the filename onto legacy `m=1` rows that lack the suffix
  (otherwise the lowest dose would be read as dose 0 and flatten the dose-response fit);
- `delta_mean = Y(J) - Y(T(J))` for the scoring model; the per-observation **break
  event** `B = 1{Y(J)=1, Y(T(J))=0}` is derived from `y_base_mean` and the per-model
  deltas by `eval/estimands.py` (`build_job_targets`), which also computes per-job `pi`,
  dose-response `(a, b)`, and the solvable-stratum restriction;
- `d_obs` / `d_wc` are job-level `D_avg` / worst-case targets (Frail-τ runs).

### 3.5 Synth-CD (synthetic mechanism checks)

`scripts/synth.py` / `synth2.py` generate synthetic jobs with *known* causal ground
truth: two independent confounders `C ~ Beta(2,2)` (structural complexity) and
`E ~ Beta(2,2)` (domain expertise), 8 domains, 12 operator types spanning
cosmetic-only through dual-sensitive, with text length controlled to be independent of
`C` and `E`. `simulate.py` / `simulate2.py` label twins with

```
delta(J, op) = max(0, mu_op (1 + C(J)) + zeta_op E(J) + eps)
```

calibrated so `Spearman(D_avg, C) ≈ Spearman(D_avg, E) ≈ 0.60`. Scores are continuous
and no stratum is censored, so U-CCD is well posed here; it is used as a *mechanism
check* (does the contrast head recover the expertise signal with near-zero `|rho_C|`
without explicit residualization?), not as a headline setting.
`scripts/synth_frailpi.py` + `synth_frailpi_baselines.py` provide the Bernoulli-break,
dose-tiered variant used for the Synth-CD consistency table.

---

## 4. Repository layout

```
src/causal_difficulty/
  models/
    fragility.py            Frail-π: per-observation break-probability model
    fragility_encoder.py    Frail-π with joint encoder fine-tuning / LoRA
    sce.py                  Siamese Counterfactual Encoder (Frail-τ)
    losses.py               Huber, triplet, invariance R_inv losses
    train.py                two-stage SCE training loop
  encoders/                 hashing, hf_frozen, hf_finetune, hf_lora, openai_embed
  baselines/                bertpe, vap3, promptset, pi_adapters, frozen_distance,
                            amortized_irt, score_predictor, embedding_linear_probe,
                            length, common
  data/                     perturbations, math_surface_ops, filler_control,
                            synth_jobs, synth2_jobs, synth2_perturbations
  eval/
    estimands.py            pi, break events, dose parsing, job targets
    deconfound.py           cross-fitted residualization (CRD), orthogonality diagnostics
    reliability.py          parametric-bootstrap reliability ceiling
    splits.py               fingerprinted job-disjoint train/val/test splits
    metrics.py, irt.py, gsmic_adv_deconf.py, reports.py, loading.py
  features/                 surface features, text featurizers, probes, collision checks
  config.py                 YAML config loader (configs/*.yaml)
  llm_env.py, llm_cache.py  LLM client configuration + on-disk response cache
  schemas.py                LabeledPair and friends
scripts/                    experiment entry points and runbooks (see §7, §8)
configs/                    120+ run configurations (suite / dose / backbone / ablation)
release/glassgsm/           GlassGSM download location (hf download reza2ei/GlassGSM; not tracked)
tests/                      unit tests (89)
docs/fragility_tuning.md    ablation runbook (5 recipe levers)
```

---

## 5. Installation

```bash
python -m venv .venv && source .venv/bin/activate
pip install -e ".[hf,openai,dev]"     # torch, transformers, sentence-transformers,
                                      # peft, openai, httpx, pytest
# or with uv:  uv sync --all-extras
```

Python ≥ 3.10. The released-data conversion script additionally needs `pandas` and
`pyarrow`.

**LLM access** is needed only for dataset construction, auditing, ex-ante probe
collection, and solver-transfer runs — *not* for training or evaluating on existing
labeled pairs. Copy `.env.example` to `.env` and fill it in. `src/causal_difficulty/llm_env.py`
selects the backend from `LLM_BACKEND` (`openai` | `fireworks`),
falling back on whichever of `LLM_API_KEY` / `FIREWORKS_API_KEY` / `OPENAI_API_KEY` is
set; point `OPENAI_BASE_URL` / `LLM_BASE_URL` at any OpenAI-compatible endpoint.

Hugging Face model weights download automatically on first use.

---

## 6. Quickstart (no API access needed)

Rebuild the GlassGSM labeled-pair files from the released parquet (download it first with
`hf download reza2ei/GlassGSM --repo-type dataset --local-dir release/glassgsm`, which creates
`release/glassgsm/data/glassgsm/`), then train and evaluate Frail-π on a CPU-friendly featurizer:

```bash
pip install -e ".[hf,openai,dev]" pandas pyarrow

# 1. released parquet -> data_audited_multiop_mix/*_labeled_pairs.jsonl
python scripts/release_to_labeled_pairs.py --suite glassgsm --solver gpt-4o-mini

# 2. CPU smoke run on the m=9 slice (frozen TF-IDF+SVD features, one seed)
python scripts/fragility_report.py \
  --dataset "$PWD/data_audited_multiop_mix/gsmic_adv_m9_multiop_mix_4omini_labeled_pairs.jsonl" \
  --featurizer tfidf_svd --seeds 0 --max-epochs 20 \
  --out-dir runs/smoke

# 3. unit tests
python -m pytest tests -q
```

`--dataset` accepts absolute paths, bare file names resolved under `data/`, a config
name, a dose-family shorthand (`gsmic_adv`, `math_adv`), or a comma-separated list of
files. **Relative paths with a directory component are not resolved against the
working directory — use `$PWD/...`.** Step 2 prints the test-fold Spearman for the
break rate and the dose-response parameters and writes `report.md` / `summary.json`
into `--out-dir`. The numbers of this CPU smoke run are *not* the paper's: the paper's
Frail-π uses the fine-tuned encoder on a GPU (Step 5).

---

## 7. Reproduction guide

All commands assume the repo root and an activated environment. Dry-run every
pipeline first (no API calls):

```bash
python scripts/gsmic_adv_pipeline.py --dry-run --max-problems 5 --multi-operator
```

### Step 1 — build twins (per dose, multi-operator)

For each dose `m in {1,3,5,7,9}` (GlassGSM; GlassMATH is analogous — see
`scripts/run_gsmic_full_doses.sh` / `scripts/run_math_full_doses.sh`):

```bash
python scripts/gsmic_adv_pipeline.py \
  --model gpt-4o-mini \
  --max-problems 1319 --max-distractors 20 --distractors-per-twin "$m" \
  --multi-operator --surface-twins 10 \
  --concurrency 32 \
  --cache-dir cache/gsmic_adv_m${m}_4omini \
  --output data_multi_full/gsmic_adv_m${m}_multi_4omini_labeled_pairs.jsonl
```

- `--multi-operator` generates the three families in one pass;
- `--distractors-per-twin $m` is the injection size (stacked distractor sentences);
- `--surface-twins 10` is the number of twins per surface operator per problem;
- responses are cached under `--cache-dir`, so re-runs and lower doses are incremental;
- GlassMATH: `python scripts/math_adv_pipeline.py --model gpt-4o-mini --no-max-problems
  --max-distractors 20 --distractors-per-twin $m --multi-operator --surface-twins 10
  --output data_math_full/math_adv_m${m}_multi_4omini_labeled_pairs.jsonl` (the
  difficulty-level → `complexity_score` mapping is built in).

Other solvers/generators are different `--model` / `--backend` values; the generator tag
is encoded in the output file suffix (`4omini`, `qwen3p7plus`, `deepseekv4pro`, `llama`).

### Step 2 — No-Op audit

```bash
python scripts/audit_contradictions_and_filter.py \
  --config gsmic_adv_m${m}_4omini \
  --pairs-path data_multi_full/gsmic_adv_m${m}_multi_4omini_labeled_pairs.jsonl \
  --out-dir data_audited \
  --concurrency 256 --model gpt-4o-mini
```

### Step 3 — build the multi-operator-mix datasets

```bash
python scripts/build_multiop_mix.py \
  --input  data_audited/gsmic_adv_m9_multi_4omini_labeled_pairs.jsonl \
  --output data_audited_multiop_mix/gsmic_adv_m9_multiop_mix_4omini_labeled_pairs.jsonl \
  --mix-twins 9 --seed 42
```

Do this for `m = 1,3,5,7,9`, for GlassMATH, and for other generators. Configs for the
resulting files are `configs/{gsmic_adv,math_adv}_m{m}_multiop_mix_{gen}.yaml` and the
pooled `*_pooled_multiop_mix_*.yaml`. (To skip Steps 1–3 for GlassGSM, use the
Quickstart conversion of the released data.)

### Step 4 — (optional) ex-ante base probes

The `text_probe` protocol adds features from repeated samples of the *base* prompt only
(no twin ever runs — a different, model-access claim):

```bash
python scripts/collect_base_probes.py \
  --dataset gsmic_adv --model gpt-4o-mini --n-samples 8 \
  --out-dir outputs_probes/gsmic_adv
```

Produces `outputs_probes/gsmic_adv/base_probes.json` (richer n=24/n=48 variants are the
"richer probes" ablation of `docs/fragility_tuning.md`). Gate check:
`scripts/base_probe_pilot.py`.

### Step 5 — C1: predict CCD from text (Frail-π)

`scripts/fragility_report.py` is the single headline entry point: it loads the labeled
pairs, makes a fingerprinted job-disjoint three-way split, trains Frail-π **inside the
process on the train fold only** (there is no `--checkpoint` path that could smuggle in
test-fit weights), early-stops on the validation fold, and reports — over several seeds
— the test-fold Spearman for `pi` and the dose-response parameters `a`/`b`, plus
surface-feature / declared-complexity / measured-`Y(J)` references and (with
`--include-bertpe-vap3`) the supervised baselines. Divide by the suite's reliability
ceiling (Step 7) to get the paper's normalized numbers. It also writes the CRD
orthogonality diagnostics (Step 6).

Headline runs, mirroring `scripts/run_multiop_mix_conditional_fragility.sh` (GPU):

```bash
MIX="$PWD/data_audited_multiop_mix"
POOLED="$MIX/gsmic_adv_m1_multiop_mix_4omini_labeled_pairs.jsonl,$MIX/gsmic_adv_m3_multiop_mix_4omini_labeled_pairs.jsonl,$MIX/gsmic_adv_m5_multiop_mix_4omini_labeled_pairs.jsonl,$MIX/gsmic_adv_m7_multiop_mix_4omini_labeled_pairs.jsonl,$MIX/gsmic_adv_m9_multiop_mix_4omini_labeled_pairs.jsonl"

# A) pooled 5-dose (main CCD table) — Frail-π = hf_finetune encoder
python scripts/fragility_report.py \
  --dataset "$POOLED" --seeds 0,1,2,3,4 --device cuda \
  --featurizer hf_finetune --protocol both \
  --probe-file outputs_probes/gsmic_adv/base_probes.json \
  --include-bertpe-vap3 \
  --jobs-per-batch 64 --encoder-microbatch 16 \
  --encoder-lr 2e-5 --head-lr 1e-3 --encoder-max-epochs 20 --encoder-patience 5 \
  --dump-predictions \
  --out-dir runs/multiop_mix/pooled_5dose_hf_finetune

# B) single-dose m=9 (CRD / cross-model protocol)
python scripts/fragility_report.py \
  --dataset "$MIX/gsmic_adv_m9_multiop_mix_4omini_labeled_pairs.jsonl" \
  --seeds 0,1,2,3,4 --device cuda --featurizer hf_finetune --protocol both \
  --probe-file outputs_probes/gsmic_adv/base_probes.json \
  --include-bertpe-vap3 --dump-predictions \
  --out-dir runs/multiop_mix/m9_only_hf_finetune
```

Notes:

- `--protocol text | text_probe | both`; `text_probe` requires `--probe-file`. The two
  protocols are reported separately and never merged;
- `--dump-predictions` writes per-seed held-out predictions consumed by the curation,
  ensembling, and per-family evaluations;
- the paper's representation ablation is the `--featurizer` axis; the five
  training-recipe levers are in `docs/fragility_tuning.md`.

### Step 6 — C2: CRD orthogonality and U-CCD controls

- **CRD orthogonality (real suites, `Q = pi`)** comes from the same
  `fragility_report.py` runs: per seed it reports `Corr(target, C)`, `Corr(pred, C)`,
  `Corr(pred, C^2)` (the `orthogonality_*` block, `eval/deconfound.py`).
- **U-CCD (Frail-τ) and the full baseline sweep**:

  ```bash
  python scripts/paper_supports_real.py --config gsmic_adv_m9_multiop_mix_4omini --seed 7
  python scripts/paper_supports_real.py --config gsmic_adv_m9_multiop_mix_4omini \
      --estimand d_avg --seed 7          # "the unconditional target is unlearnable"
  ```
- **CCD vs CRD across solvers**: `scripts/solver_crd_vs_ccd_transfer.py --suite gsmic_adv`.

### Step 7 — reliability ceilings

Spearman numbers in the paper are normalized by each suite's reliability ceiling
(0.969 GlassGSM, 0.972 GlassMATH), computed by parametric bootstrap with a
cross-validated empirical-Bayes prior:

```bash
python scripts/noise_ceiling.py --config gsmic_adv_m9_4omini --n-repeats 30
python scripts/noise_ceiling.py --config math_adv_m9_4omini  --n-repeats 30
```

Per-slice ceilings: `scripts/slice_ceilings.py`. Always report the raw ρ_S alongside
normalized values — normalized numbers depend on the ceiling estimator and slice.

### Step 8 — placebo / identity-twin floor

Because solver outputs are sampled, `B` can fire with no true effect. The floor is
measured by re-solving the *unperturbed* bases of solvable-stratum items:

```bash
python scripts/null_twin_noise_floor.py --suite gsmic_adv   # paper: 1.1%
python scripts/null_twin_noise_floor.py --suite math_adv    # paper: 2.5%
```

Complementary control: `scripts/null_perturbation_control.py --probe-file
outputs_probes/gsmic_adv/base_probes.json`.

### Step 9 — downstream utility: top-k curation

Rank the audited holdout by each predictor and recall truly fragile jobs (`pi > 0.05`)
at budgets k in {5,10,20,50}%:

```bash
# reads Frail-π's dumped held-out predictions from the pooled 5-dose run
python scripts/downstream_utility_pi.py
python scripts/downstream_curation.py --config gsmic_adv_m9_4omini --seeds 7 42 123
```

### Step 10 — solver transfer and cross-generator checks

The stratum is defined by the solver's own `Y(J)`, so varying the solver tests how far
the estimand travels:

```bash
python scripts/solver_pi_agreement.py \
  --configs gsmic_adv_m9_4omini gsmic_adv_m9_qwen3p7plus gsmic_adv_m9_deepseekv4pro
python scripts/cross_model_transfer.py \
  --configs gsmic_adv_m9_4omini gsmic_adv_m9_qwen3p7plus gsmic_adv_m9_deepseekv4pro
python scripts/solver_crd_vs_ccd_transfer.py --suite gsmic_adv
```

Cross-*generator* checks (Qwen / DeepSeek-authored twins, `gpt-4o-mini` scoring) reuse
the same entry points on the `*_qwen3p7plus` / `*_deepseekv4pro` mix files; BERTPE/VAP3
on those slices: `scripts/run_bertpe_vap3_crossmodel.sh`.

### Step 11 — Synth-CD mechanism checks

```bash
python scripts/synth2.py --seed 42                 # jobs + twins (offline)
python scripts/simulate2.py                        # causal DGP labels + oracle D^perp
python scripts/paper_supports_synth2.py --config synth2
python scripts/paper_supports_synth2.py --config synth2_gpt --backbone openai_embed
python scripts/synth_frailpi.py --n-jobs 500 --twins-per-dose 5 --seed 7 \
  --out data/synth_frailpi_labeled_pairs.jsonl
python scripts/fragility_report.py --dataset data/synth_frailpi_labeled_pairs.jsonl
python scripts/synth_frailpi_baselines.py
```

The contrast head should reach the best Spearman with the lowest `|rho_C|` **without**
explicit residualization.

### Step 12 — baselines

```bash
python scripts/run_baselines.py --config gsmic_adv_m9_multiop_mix_4omini
python scripts/run_baselines.py --config math_adv_m9_multiop_mix_4omini
```

The PromptSET / frozen-geometry baselines retargeted to the break-event `pi`:
`scripts/run_missing_pi_baselines.py`. Per-family baseline table:
`scripts/ccd_per_family_baselines.py`.

### Step 13 — per-category / per-family breakdowns

```bash
python scripts/ccd_per_category.py --files "$POOLED" --label gsmic_adv
python scripts/ccd_per_family.py   --files "$POOLED" --label gsmic_adv
python scripts/ccd_per_family_predict.py ...        # evaluate dumped predictions
```

---

## 8. Orchestration scripts

Shell runbooks that chain the entry points above as the paper's runs were launched
(multi-GPU where noted; run from the repository root):

| Script | What it runs |
|---|---|
| `run_multiop_mix_conditional_fragility.sh` | headline C1+C2: pooled 5-dose + m=9 Frail-π, 5 seeds, BERTPE/VAP3 |
| `run_multiop_mix_m9_full_baselines.sh` | m=9 full baseline sweep (`paper_supports_real.py`) |
| `run_multiop_mix_pooled_sce.sh` | pooled 5-dose Frail-τ (`d_avg`/`d_wc`) + baselines |
| `run_multiop_mix_sce_dosage_sweep.sh` | all doses × 5 seeds, Frail-τ + baselines |
| `run_multiop_mix_frailpi_contrast_ablation.sh` | `--lambda-contrast` sweep |
| `run_multiop_mix_lambdainv_ablation.sh` | invariance-regularizer `lambda_inv` sweep |
| `run_multiop_mix_richer_probes.sh` | richer ex-ante probes (n=24) ablation |
| `run_fragility_tuning_all.sh` + `docs/fragility_tuning.md` | 5 recipe levers (RoBERTa-large, lr/batch, pooling, …) |
| `run_gsmic_full_doses.sh` / `run_math_full_doses.sh` | full per-dose twin generation + audit |
| `run_bertpe_vap3_crossmodel.sh` | BERTPE/VAP3 on cross-generator slices |
| `run_train_eval_synthetic_3seeds.sh` | Synth-CD train/eval |
| `run_c2_diagnostics.sh` | C2 diagnostics on `pi` / surface-proxy `D^perp` |

`scripts/` also holds many one-off launchers, resume scripts, and analysis helpers
from the experiment campaign; the table above lists the ones tied to paper results.

---

## 9. Configuration reference

YAML configs in `configs/`, loaded by name via `causal_difficulty.config.load_config`.
A mix-corpus config (`configs/gsmic_adv_m9_multiop_mix_4omini.yaml`):

```yaml
seed: 42
paths:
  data_dir: data_audited_multiop_mix
  outputs_dir: outputs/gsmic_adv_m9_multiop_mix_4omini
  cache_dir: cache/gsmic_adv_m9_multiop_mix_4omini
  labeled_pairs: data_audited_multiop_mix/gsmic_adv_m9_multiop_mix_4omini_labeled_pairs.jsonl
dataset:
  n_jobs: 200
  twins_per_job: 20
encoder:
  backbone: openai_embed          # hashing | hf_frozen | hf_finetune | hf_lora | openai_embed
  hf_frozen: { model: BAAI/bge-small-en-v1.5, batch_size: 256 }
  openai_embed: { model: text-embedding-3-small, max_concurrency: 16 }
model:
  hidden_dim: 64
  complexity_dim: 32
  contrast_context: surface_pair  # surface_pair | pair | none
  dropout: 0.25
  ex_ante_n_layers: 2
train:
  representation: { epochs: 200, lr: 0.001, batch_size: 64, triplet_margin: 0.5 }
  calibration: { epochs: 1000, lr: 0.0001, weight_decay: 0.01, batch_size: 32,
                 lambda_inv: 0.0, huber_delta: 0.1 }
eval:
  fragile_threshold: 0.05
  holdout_frac: 0.3
  stratified_split: true
```

Frail-π's recipe is set by the `fragility_report.py`
flags shown in Step 5 (`--encoder-lr 2e-5 --head-lr 1e-3 --hidden-dim 256 ...`); the
`model:` / `train:` blocks above configure the Frail-τ / SCE path.

---

## 10. Outputs and tests

- `runs/{run_id}/` — one directory per experiment: `report.md` (human-readable tables),
  `summary.json` (full metrics, per seed), prediction dumps (`--dump-predictions`),
  logs; a one-line summary is appended to `runs/runs.jsonl`.
- `outputs_fragility/{dataset}/` — default `fragility_report.py` output directory.
- `outputs_probes/{suite}/base_probes.json` — ex-ante probe features.
- `data_audited*/`, `data_audited_multiop_mix/` — audited benchmarks and the mix corpus
  (built or converted, not committed).
- `cache/` — LLM response caches (safe to delete; re-runs re-call the API).

These directories and model checkpoints are git-ignored.

```bash
python -m pytest tests -q        # 89 tests: estimands, deconfound, collision audit,
                                 # filler control, math-adv, human audit, units
```

---

## 11. Reproducibility notes

- **Stratum membership is a measured quantity.** `Y(J)` is the modal answer of 5
  samples at temperature 0.2; the identity-twin floor (Step 8) quantifies the
  re-sampling noise that leaks into `pi`.
- **Doses are stamped from filenames** (`_m{k}_`); the legacy `m=1`
  `math_irrelevant_context` tag (no suffix) is normalized at load time.
- **Splits are fingerprinted and job-disjoint** across train/val/test; checkpoints carry
  a sidecar of their train job-ids and evaluators refuse to score a checkpoint that
  overlaps their holdout. Use the same `--seeds` and `--dataset` string when comparing
  featurizers — folds are deterministic in `(--dataset, seed)`.
- **Ceiling-normalized numbers are estimator-sensitive.** The paper's 0.969 / 0.972 come
  from the parametric-bootstrap estimator of Step 7 on the audited corpora behind the
  tables. Re-running on a different slice (single dose vs pooled), probe file, or
  estimator prior changes both the ceiling and the achievable ρ_S; compare raw ρ_S on
  identical splits, and expect absolute values from re-runs to differ from the paper's
  tables, especially for the single-dose `m = 9` slice (the paper's own `m = 9`
  solver-transfer diagonal is 0.431 on GlassGSM).
- **The ex-ante probe file matters.** `text_probe` results depend on the probe sample
  count and draw (`--n-samples`); the probe file behind the paper's runs is not part of
  this repository.
- **Complexity proxies are declared and external; CRD is orthogonal to *that* proxy
  only.** For GlassGSM the released pipeline computes `complexity_score` from the problem
  text (question numbers, keywords, sentences), which correlates with length. For
  GlassMATH it is the problem-setter level.
- **Per-family structure.** Each mix-corpus twin applies a single operator. On the
  released GlassGSM data (`gpt-4o-mini`, unique twins of solved bases) the break rate is
  13.8% for sentence reordering, 13.0% for distractor injection, and 1.5% for entity
  renaming — the last is close to the 1.1% identity-twin floor, so renaming is a
  low-signal family and most of the fragility comes from reordering and injection.
  Per-family breakdowns: Step 13.

---

## 12. Citation, license, third-party material

```bibtex
@misc{rezaei2026fragilityreduciblehardnessconditional,
      title={Fragility Is Not Reducible to Hardness: Conditional Causal Difficulty for LLMs},
      author={Mohammad Reza Rezaei and Dingxian Wang and Nikhil Verma and Sajjad Pakdamansavoji and Andrew Rabinovich and Zhao Chen},
      year={2026},
      eprint={2611.02065},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2611.02065},
}
```

The package is released under the MIT license (`pyproject.toml`). GlassGSM / GlassMATH
contain text derived from GSM8K (MIT, OpenAI) and the Hendrycks MATH benchmark (MIT);
see `NOTICE.md` in the dataset repos for the third-party notices and license text.