Publish research paper 01-factorized-transfer.md

papers/01-factorized-transfer.md

@@ -0,0 +1,226 @@
+# Factorized Temporal Transfer with Support-Aware Spatial Adaptation
+
+## Abstract
+
+Transfer across geospatial time series is asymmetric. Temporal motifs such as
+trend, periodicity, persistence, and local transitions recur across domains,
+whereas spatial relations differ between road networks, sensor fields, and
+metropolitan panels. We study FAST-ST, a 17.1-million-parameter model that keeps
+node identity out of a shared patch-transformer backbone and introduces spatial
+structure through a sparse relation adapter. Its output is a joint low-rank
+multivariate Student-t distribution over every support and horizon. On
+chronological U.S. holdouts spanning 24-metro weather, METR-LA, and PEMS-BAY,
+FAST-ST improves aggregate MAE over persistence by 14.635% and over its own
+temporal-only ablation by 5.867%. The experiment supports temporal/spatial
+factorization while showing a remaining failure: PEMS-BAY is 0.719% worse than
+persistence. We therefore treat factorization as an evidence-backed design
+direction, not a claim of universal superiority.
+
+## 1. Problem statement
+
+Let (x_{i,t}^{(v)}) be a sampled variable (v) at support (i) and time
+(t). For context length (L) and horizon (H), the learned engine estimates
+
+\[
+p_\theta\!\left(
+X_{t+1:t+H}\mid X_{t-L+1:t},M,C,R
+\right),
+\]
+
+where (M) is the observation mask, (C) contains admissible calendar
+covariates, and (R) contains spatial relations. The central restriction is
+that the temporal backbone receives no node identifier. Consequently, memorized
+sensor identity cannot substitute for temporal transfer.
+
+The hypothesis is
+
+\[
+\mathcal F_{\text{ST}}(X;R)
+=
+\mathcal A_{\text{space}}
+\!\left(\mathcal T_{\text{shared}}(X,M,C),R\right),
+\]
+
+with a large shared temporal map (mathcal T_{\text{shared}}) and a smaller
+support-conditioned adapter (mathcal A_{\text{space}}).
+
+## 2. Reversible per-series normalization
+
+For each series window, FAST-ST computes masked moments
+
+\[
+\mu_i=\frac{\sum_t m_{i,t}x_{i,t}}{\max(1,\sum_t m_{i,t})},
+\qquad
+\sigma_i^2=\frac{\sum_t m_{i,t}(x_{i,t}-\mu_i)^2}
+{\max(1,\sum_t m_{i,t})}.
+\]
+
+The normalized series is
+
+\[
+\widetilde x_{i,t}=m_{i,t}\frac{x_{i,t}-\mu_i}{\max(\sigma_i,10^{-4})}.
+\]
+
+Missingness remains an explicit input channel. When an entire support history is
+missing, physical location and scale are borrowed from observed supports in the
+same batch while the temporal state is formed from type and calendar tokens.
+This prevents the numerical normalization floor from being mistaken for the
+physical scale of the target.
+
+## 3. Temporal patch backbone
+
+With patch length (P), token (p) for support (i) is
+
+\[
+h_{i,p}^{(0)} = W_p
+\left[
+\widetilde x_{i,p};m_{i,p};\overline c_p
+\right] + e_{\operatorname{type}(v)}.
+\]
+
+The sequence is processed by pre-normalized self-attention and SwiGLU blocks:
+
+\[
+H^{(\ell+1)}=H^{(\ell)}+
+\operatorname{Attn}(\operatorname{RMSNorm}(H^{(\ell)})),
+\]
+
+\[
+H^{(\ell+2)}=H^{(\ell+1)}+
+\operatorname{SwiGLU}(\operatorname{RMSNorm}(H^{(\ell+1)})).
+\]
+
+Series-type embeddings distinguish temperature, humidity, wind, pressure, and
+traffic without identifying a city or sensor.
+
+## 4. Support-aware spatial adapter
+
+For candidate supports (i,j), the relation vector is
+
+\[
+r_{ij}=\left[
+s_i-s_j,\;\lVert s_i-s_j\rVert_2,\;
+\exp(-\lVert s_i-s_j\rVert_2^2),\;a_{ij}
+\right],
+\]
+
+where (a_{ij}) is supplied topology when a physical graph exists. Attention
+scores are
+
+\[
+\alpha_{ij}\propto
+\exp\left(
+\frac{(W_qh_i)^\top(W_kh_j)}{\sqrt d}+g_\phi(r_{ij})
+\right),
+\]
+
+restricted to a sparse local candidate set. The update
+
+\[
+h_i' = \operatorname{RMSNorm}\left(
+h_i+\operatorname{sigmoid}(\gamma)
+\sum_j\alpha_{ij}W_vh_j
+\right)
+\]
+
+has a learned gate (gamma), allowing optimization to suppress spatial
+messages when they are not useful.
+
+## 5. Joint robust forecast law
+
+Independent marginal quantiles do not define a coherent distribution for a
+trajectory. FAST-ST instead predicts
+
+\[
+Y\sim t_\nu(\mu,\Sigma),
+\qquad
+\Sigma=D+UU^\top,
+\]
+
+where (D=\operatorname{diag}(d_1^2,\ldots,d_m^2)) and (U\in\mathbb R^{m\times r})
+with (r\ll m). Joint samples preserve dependence across supports and horizons.
+The Student-t law is the scale mixture
+
+\[
+Y=\mu+\sqrt{\frac{\nu}{W}}\left(Uz+De\right),
+\quad
+W\sim\chi_\nu^2,\quad z,e\sim\mathcal N(0,I).
+\]
+
+The exact likelihood uses the determinant lemma and Woodbury identity:
+
+\[
+\log|D+UU^\top|
+=\log|D|+\log|I+U^\top D^{-1}U|,
+\]
+
+\[
+r^\top\Sigma^{-1}r
+=r^\top D^{-1}r-q^\top(I+U^\top D^{-1}U)^{-1}q,
+\quad q=U^\top D^{-1}r.
+\]
+
+The implementation forms this system from whitened factors and solves it by a
+float64 Cholesky decomposition with scale-aware jitter. Native target masks are
+handled by exact observed-dimension subsetting.
+
+## 6. U.S. experimental design
+
+The release uses only U.S. observations:
+
+- four hourly Open-Meteo/ERA5 variables for 24 hard-coded U.S. metros from
+  2022-01-01 through 2025-12-31;
+- Hugging Face METR-LA at revision
+  `612eedcdb60280dadd414e70a7f3b37103667a55`;
+- Hugging Face PEMS-BAY at revision
+  `4c488d8ce974a326b4d09bcbdd6f6fdc8fb5fd67`.
+
+Splits are chronological. Uncertainty scale is selected on validation data
+only. Test evaluation compares the full model with persistence and with the
+same checkpoint evaluated after disabling its spatial adapter. Metrics include
+MAE, RMSE, 80% interval coverage, joint negative log likelihood, and normalized
+energy score.
+
+## 7. Results
+
+The 3,000-step H100 checkpoint reports geometric relative MAE of 0.853650
+against persistence and 0.941333 against its temporal-only ablation. Mean 80%
+interval coverage is 0.713562, joint NLL per target is 2.583812, and normalized
+energy score is 30.509663.
+
+Weather gains over persistence are 29.927% for temperature, 19.518% for
+relative humidity, 16.927% for wind speed, and 16.744% for surface pressure.
+METR-LA improves by 1.500%. PEMS-BAY declines by 0.719%. Spatial adaptation
+improves the full system by 5.867% in aggregate, with per-series gains from
+1.963% to 14.579%.
+
+All preregistered release gates pass: U.S.-only provenance, presence of the two
+pinned Hugging Face corpora, aggregate improvement over persistence, material
+spatial contribution, finite joint score, and calibrated aggregate coverage.
+
+## 8. Limitations and falsification criteria
+
+The experiment contains only two traffic networks and four weather variables.
+Its spatial holdout is temporal-within-network rather than an unseen-network
+evaluation. A stronger test must hold out entire U.S. cities, support types, and
+variable families. The PEMS-BAY failure shows that a positive aggregate result
+does not license per-domain superiority claims. Future releases should reject
+the factorized model for any target family where a matched temporal baseline is
+consistently better.
+
+## 9. Reproducibility
+
+The release manifest records source hashes, Hugging Face revisions, training
+seed, code digest, H100 hardware, chronological partitions, calibration, test
+metrics, and every gate. The checkpoint is `fast-st-us-v1`; its source is
+`models/fastst.py` and its Modal training entry point is `modal/train_us.py`.
+
+## References
+
+1. Nie et al. PatchTST: A Time Series is Worth 64 Words. ICLR, 2023.
+2. Ansari et al. Chronos: Learning the Language of Time Series. 2024.
+3. Woo et al. Unified Training of Universal Time Series Forecasting Transformers. ICML, 2024.
+4. Shao et al. Decoupled Dynamic Spatial-Temporal Graph Neural Network. PVLDB, 2022.
+5. Liu et al. Spatio-Temporal Identity: A Simple Yet Effective Baseline. 2022.
+6. Lu et al. STGformer. 2024.
+7. FactoST-v2. Factorized Spatio-Temporal Foundation Modeling. 2026.