analysis/test_reflection.md

Alpamayo 2 Super NVFP4: full test reflection and limitations

Prepared from the completed experiment artifacts on 2026-09-21. This is a retrospective audit; it introduces no new GPU measurements. It should be read alongside the reproduction guide and the clip 0103cc09 investigation.

The experiment established a working, compressed model and native acceleration on one RTX 5090. It did not establish preserved or improved driving quality. The final native version had lower average ADE and FDE on five selected clips, but individual regressions, one influential improvement, uncontrolled coupling between two sampling stages, and limited measurement coverage prevent a general quality conclusion. Functional correctness, prediction fidelity, driving quality, and runtime performance require different evidence.

Claim	Assessment	Evidence and boundary
Selected language weights were stored in NVFP4	Established for this checkpoint	448 packed language projections; seven safetensors shards totaling 26,770,262,024 bytes
The full model ran on a 32 GB RTX 5090	Established for the tested workload	Completed rollouts without CPU offload; 29.002 GiB peak PyTorch allocation and 29.113 GiB reservation
Native FP4 computation executed	Established in the observed run	All 448 packed layers selected the native implementation; a full-rollout CUDA trace contains an SM120 CUTLASS block-scaled FP4 kernel
Native execution accelerated the application	Observed on this configuration	Median model-call latency fell from 19.075 to 4.921 seconds on the same 5090; about 3.876 times faster than the fallback
Quantization improved average scores on these five clips	Descriptively true	Native mean ADE 1.588 m and FDE 4.271 m versus BF16 2.058 m and 5.920 m
Quality is preserved or generally better	Not established	Five single-seed predictions, two regressions, no non-inferiority threshold, and possible diffusion-noise differences
The system is ready for real-time driving	Not established	No closed-loop evaluation, sustained deployment test, or application deadline validation

What was actually tested. The starting point was nvidia/Alpamayo2-Super, model revision 00554695e729a6ff0b6281fd2c81b18d06e33dbe, with Alpamayo source revision 5e7975f4a2100ee8ac1a62b79239bbabefddcbef. We adapted the released inference wrapper and ModelOpt's NVFP4 configuration. This is a custom experimental integration, not an official NVIDIA validation of this quantized checkpoint and not evidence that an Alpamayo 1.5 recipe transfers unchanged to version 2.

ModelOpt 0.43.0's NVFP4_DEFAULT_CFG was restricted to language-transformer linear projections. It uses maximum-based calibration, four-bit floating-point weights and input activations, block size 16, and FP8 block scales. The 448 selected projections contain 31,205,621,760 weight values. Vision, the action expert, embeddings, output head and KV cache retained BF16 precision. Keeping the expert in BF16 does not preserve its behavior automatically: its conditioning comes from the quantized language model and generated text.

Five calibration samples came from the pinned examples/validation_samples.json; five evaluation samples came from examples/public_golden_validation_samples.json. The harness selects manifest entries in their listed order, with no randomized or stratified selection. Clip IDs were checked to be disjoint between calibration and evaluation. The dataset was nvidia/PhysicalAI-Autonomous-Vehicles, revision 33f9bf447ed3bcb7d545ce13f4226f824214fafb. Each evaluation used one timestamp from a clip, six driving-profile cameras with four frames each, 4,580 input tokens, and ego-motion history. We did not run continuously through the entire clip.

The output was one trajectory with 64 future positions at 0.1-second intervals, covering 6.4 seconds. Settings were seed 42, top-p 0.98, temperature 0.6, and ten diffusion integration steps. Ground-truth future positions were saved separately. The input helper constructs generation-mode messages from images, history and a prompt; the final model-input dictionary contains tokenized data and ego history rather than the future-position arrays. Source/model/dataset provenance was checked when the prepared tensors were reloaded. These are useful controls against input changes and accidental future-label leakage in this pipeline; they do not establish that the clips were absent from model pretraining.

The measured scores are descriptive results, not a quality acceptance test. ADE is the mean two-dimensional Euclidean position error across the 64 future timestamps. FDE is that error at the final timestamp. Each clip contributes equally to the reported mean. We measured one sampled prediction rather than selecting the best of several samples.

Execution	Mean ADE (m)	Mean FDE (m)	Median model call (s)	Peak allocated VRAM (GiB)
BF16, PRO 6000	2.058	5.920	2.310	68.930
NVFP4 simulation, PRO 6000	1.924	4.857	3.633	69.001
Packed fallback, PRO 6000	1.963	5.021	18.373	28.705
Packed fallback, RTX 5090	1.963	5.021	19.075	28.705
Native NVFP4, RTX 5090	1.588	4.271	4.921	29.002

The original BF16, simulated NVFP4 and first packed-checkpoint measurements ran on an RTX PRO 6000 with 96 GB of VRAM. The later packed-fallback and native measurements ran on the same RTX 5090. The BF16 latency on the PRO 6000 is therefore not a same-device speed baseline for the native 5090 result. The original packed checkpoint was also evaluated in a fresh process, reproducing its saved trajectories exactly. That validates the tested restoration path, not equivalence to the original BF16 predictions.

The final native model's averages were 22.85% lower for ADE and 27.85% lower for FDE than the BF16 reference. Here is the complete five-clip comparison:

Clip prefix	BF16 ADE (m)	Native ADE (m)	BF16 FDE (m)	Native FDE (m)	Native versus BF16
003e17f1	2.543	2.728	5.065	5.623	Higher errors
009148b9	2.669	1.678	6.678	5.646	Lower errors
00978bec	1.409	1.056	3.440	2.287	Lower errors
00ee8960	0.862	1.182	2.230	5.075	Higher errors
0103cc09	2.805	1.294	12.186	2.724	Lower errors

Native NVFP4 has lower errors on three clips and higher errors on two. On 00ee8960, FDE grows from 2.230 m to 5.075 m. On 0103cc09, it falls from 12.186 m to 2.724 m. That large gain dominates the aggregate FDE improvement: across the other four clips, mean FDE rises from 4.353 m to 4.658 m. Excluding a clip here is a sensitivity check illustrating the fragility of the mean, not a replacement benchmark or a reason to discard the clip. The per-clip chart makes the uneven effects visible.

Native trajectories differ from BF16 by an average of 1.321 m when comparing corresponding predicted positions directly. Native versus packed-fallback deviation averages 0.418 m. Prediction fidelity and closeness to the recorded trajectory are different measurements: a changed prediction can improve one recorded-path score while still representing substantial behavioral drift from the reference model.

The sample is too small and too selected for distribution-level conclusions. Five clips do not characterize weather, lighting, geographic variation, intersections, pedestrian interactions, unusual geometry or rare events. The 64 positions within a trajectory are strongly related observations, not 64 independent driving trials. Calibration/evaluation separation is necessary but does not make the public release samples an independent, unseen benchmark. We did not measure sampling variance across seeds, variation across independently calibrated checkpoints, statistical confidence intervals, or a predeclared acceptable degradation margin. The JSON reports contain a numerical p95 across the five ADE values; with so few clips this is an interpolated descriptive statistic, not a reliable estimate of a deployment tail.

The five calibration rollouts are also a narrow basis for activation scales. Maximum-based calibration depends on the values encountered along those rollouts, including generated text and the sampled trajectory process. We did not test calibration-set size convergence, alternate calibration selections, alternate clipping or error-based recipes, or per-layer sensitivity. No quantization-aware fine-tuning was performed. We should not present the chosen recipe as optimal, nor infer that quantizing additional modules will preserve the observed behavior.

A fixed starting seed did not isolate quantization effects from sampling effects. Our harness resets Python, NumPy, PyTorch and CUDA seeds immediately before the complete model call. Alpamayo samples text before it invokes the trajectory diffusion sampler. Transformers uses torch.multinomial for sampled text tokens, and diffusion later initializes its action tensor with torch.randn on the same device. Neither call supplies a separate generator in the path used here, and the harness does not independently reset the diffusion random state between these stages.

Consequently, a change in the number of random draws during text generation can change the initial diffusion noise even when both runs start from seed 42. The inference order is visible in the pinned Alpamayo sampler, the noise draw in the diffusion initializer, and token sampling in Transformers 4.57.1. The seed-reset and measurement code is preserved in the experiment harness.

This does not invalidate the saved trajectories or their error arithmetic. It limits causal interpretation: the observed change combines numerical quantization effects, possible changes in generated tokens and attention-cache contents, and potentially a different trajectory-noise realization. Resetting a seed once per rollout was insufficient for the controlled same-noise comparison we would want when attributing a small-sample quality change to precision.

Clip 0103cc09 illustrates both the result and the uncertainty. BF16 predicted an endpoint at x=13.780 m and y=5.203 m, compared with recorded x=24.902 m and y=0.222 m. Native NVFP4 predicted x=27.614 m and y=-0.033 m. BF16's path increasingly bends sideways and falls short along the forward axis. The explanation text has the same visible driving intent in all modes, with a final period added in NVFP4. Re-encoding the saved strings gives 12 versus 13 text tokens. Those are reconstructed counts, not a capture of the actual generated token sequences.

The large improvement already occurred in simulated NVFP4 on the same PRO 6000 as BF16: FDE was 2.738 m there. The 5090 and native kernels were therefore not the origin of the effect. The packed trajectory also repeated exactly across the PRO 6000 and 5090. An extra text token is a plausible route to altered diffusion noise, but we did not save the original noise, complete generated token IDs, token probabilities, per-step diffusion actions or conditioning-cache states. The period has not been established as the cause, and neither has quantization been established as a systematic correction of this scenario. The model's reference to a green traffic light is generated text; the camera frames were not independently reviewed during this clip investigation.

ADE and FDE cover only a limited part of driving behavior. This was an open-loop comparison with recorded motion. The model never controlled a simulated or real vehicle, so its outputs did not change subsequent observations or other agents' responses. A path can be close to the recorded center-point trajectory and still violate a road boundary, collision margin or traffic rule. A different path can also be a valid alternative to the single recorded reference. We did not score collisions, route completion, road compliance, time-to-collision, yaw error, speed, acceleration, jerk, steering feasibility, comfort or emergency handling. We did not independently score the correctness of the generated explanation or its agreement with the predicted motion. Those omissions limit what the word 'quality' means in this report.

Native-kernel validation was meaningful but bounded. The adapter reuses packed ModelOpt weights, rearranges their FP8 block scales, quantizes activations with FlashInfer, and invokes CUTLASS FP4 matrix multiplication with BF16 outputs. A numerical gate passed 16 synthetic cases: four representative configuration-derived matrix shapes at 1, 32, 128 and 4,580 input rows. Its largest relative RMS difference from an FP32 multiplication reference using the same packed values was about 0.003007. The gate also applied rtol 0.01 and atol 0.03 checks. These checks support packing/layout/scale compatibility on the exercised cases; they do not measure error against unquantized original weights or certify arbitrary input distributions.

The small gate did not cover every distinct projection shape in the full model, every activation range, all bias cases, other dtypes or other GPU architectures. The later full-model run exercised all 448 selected layers and recorded a native FP4 CUDA kernel. The layer count and kernel trace are complementary evidence; the trace does not provide a per-layer numerical oracle or a complete runtime breakdown. The native scale cache added 1,950,351,360 bytes, included in the observed PyTorch memory counters. Native and fallback arithmetic are not assumed bit-identical, as the changed trajectories demonstrate.

The 3.876-times speedup is an application result for one setup. On the same 5090, fallback median model-call time was 19.075 seconds and native median was 4.921 seconds. Timing begins after prepared inputs are moved to the GPU and ends after the synchronized trajectory-generation call. It excludes dataset download, camera decoding, preprocessing, CPU-to-GPU input transfer, checkpoint loading, first-use kernel compilation, and final CPU output conversion. It includes the model stages executed inside that call, rather than an isolated FP4 matrix-multiply benchmark.

There was one warmup on the first clip per execution phase and one timed call per evaluation clip. Fallback was measured before native, with no alternating order, repeated trials, power/clock controls or sustained-load measurement. Five latencies are insufficient to characterize variance, tail response time or thermal behavior. Generated text differed on two of the five native-versus-fallback cases; the same inputs and sampling settings therefore did not force identical output work. We did not save full generated-token counts, tokens per second, stage-level durations or time spent in each kernel. A fair causal attribution of the exact speedup needs controlled output work and profiling; the current result establishes that the observed native application calls were much faster than the tested fallback calls.

A 4.921-second median corresponds to roughly 0.20 sequential model calls per second under these settings. A 6.4-second prediction horizon does not make a five-second computation suitable for every planning loop: observation age, required update cadence, preprocessing and control deadlines still matter. We did not benchmark optimized BF16 on the same high-memory device, other quantization formats, TensorRT-LLM, alternative attention implementations, CUDA-graph optimizations or the fastest attainable 5090 implementation. The result is a working baseline, not a speed ceiling or a competitive leaderboard claim.

Memory fit is established only for the tested configuration. Native peak allocation was 29.002 GiB and peak reservation was 29.113 GiB on the 5090. These are PyTorch allocator counters, excluding memory managed directly by the driver, context or other processes. Peaks were reset after input transfer, so they do not constitute a complete measurement of startup and transient loading behavior; successful checkpoint loading is a separate observed fact. We did not characterize maximum batch size, more trajectory samples, longer contexts, larger camera resolutions, long-running fragmentation, concurrency, or use alongside a desktop display. The tested host resources do not establish minimum CPU RAM or disk requirements. CPU offload was not used in the completed 5090 test.

Portability and reproducibility have separate boundaries. The validated runtime used Linux, Python 3.12, PyTorch 2.8 with CUDA 12.9, Transformers 4.57.1, ModelOpt 0.43.0, FlashInfer 0.6.18 and driver 580.126.09. SDPA was the attention implementation. The two GPU environments were separately provisioned, and their Torch version strings were recorded as 2.8.0 and 2.8.0+cu129; we did not treat them as a perfectly controlled identical-hardware environment. The same-hardware simulation comparison and the repeated packed trajectory provide useful checks, but do not remove all potential environment effects.

The custom adapter relies on pinned ModelOpt packed-weight internals and FlashInfer interfaces. Updating those dependencies, changing device architecture, calling a generic text AutoModel loader, or loading the checkpoint without enabling the adapter is not covered by the successful native result. Native Windows operation was not tested. Startup requires the Alpamayo wrapper, tokenizer/processor, ModelOpt restoration support, compatible build tools and the adapter. The pinned preparation archive and dependency lock improve reproducibility but do not guarantee bitwise results across arbitrary platforms or future package changes.

Useful saved evidence includes model/source/dataset revisions, clip IDs and timestamps, prepared tensors with provenance, decoded explanations, complete predicted and reference xy trajectories, per-clip error/latency/memory rows, CUDA gate results, kernel names and failure logs. Missing evidence includes exact generated token sequences, logits, separate text/diffusion RNG states, initial diffusion noise, intermediate actions, per-stage timing, hardware telemetry and a complete OS/container digest. The archive contains historical pod IDs and budget defaults; these must be refreshed for a future rental. Local mock-based control tests validate cleanup logic, not model quality or all external API failure modes.

Reflection on the engineering process. We reached the practical objective of storing a usable quantized checkpoint and running native FP4 inference on a 5090 within the experiment's US$5 ceiling. We checked calibration/evaluation separation, kept future arrays outside model inputs, tested checkpoint restoration, required a numerical native-kernel gate before the large checkpoint transfer, verified actual CUDA kernels, preserved per-clip results and checked that the GPU stopped. These controls are stronger evidence than a compressed file size or a backend label alone.

There were also failed preparation attempts involving access/setup, CUDA execution and checkpoint restoration before the successful run, plus a slow checkpoint transfer on the native-validation host. Those attempts consumed time and credits and are retained in the logs; they are not additional quality samples. Better staging would validate the complete CUDA path and shutdown/upload behavior in a reusable environment earlier. The native run retained its 90-minute hard stop while shortening the reports-only upload reservation to five minutes, which allowed transfer recovery without extending the rental ceiling. At shutdown, the observed balance decrease across the experiment was approximately US$2.28; the conservative time-based total was US$3.43. These are an observed balance difference and an estimate, not an itemized provider invoice. The guest watchdog and external control were useful protections, not a provider-enforced dollar cap.

The most important methodological correction is to stop treating 'same initial seed' as proof of a paired same-noise experiment. The most important reporting correction is to keep 'lower averages on five clips' separate from 'quality preserved' and from 'quantization improved the model.' The strong claim here concerns functional deployment and observed acceleration. The quality claim remains an exploratory observation with unresolved causes. New recipe tuning should use a separate development set rather than repeatedly optimizing these five visible evaluation clips.

A stronger follow-up should answer explicit questions in stages. The following work is proposed and has not been run:

1. Instrument this clip first. Capture complete generated tokens, the initial diffusion-noise tensor or a verifiable hash, independent text and diffusion seeds, intermediate actions, output validity and software/hardware metadata. Reuse exactly the same initial noise across precision modes instead of relying on a seed set before text generation.
2. Run two controlled comparisons on the same sufficiently large GPU: freely generated explanations with shared diffusion noise, then identical supplied explanation tokens with the same noise. The second removes variation in the text tokens while retaining numerical differences in the conditioning cache. Inspect several seeds and the full endpoint distribution before attributing the 0103cc09 result to a particular mechanism.
3. Freeze the recipe and define an acceptable degradation margin before evaluation. Use a larger, diverse calibration set, examine calibration-size stability, and evaluate on separate clips stratified by meaningful driving conditions. Keep recipe selection separate from the final held-out evaluation, and disclose uncertainty about pretraining exposure.
4. Report paired per-clip changes, confidence intervals with clips as the sampling unit (grouping correlated clips from the same scene or trip), regression frequency, failure rate and tails on enough independent clips. Include repeated sampling seeds and trajectory fidelity alongside ADE/FDE; do not count the 64 correlated future points as independent trials or substitute a best-of-many score for the one-sample score without labeling it.
5. Measure timing repeatedly with controlled workload and alternating phase order. Record preprocessing, transfers, vision/prefill, text decode and diffusion separately, plus total application latency, token counts, device memory, power/clocks and sustained behavior. Retain an actual 5090 fit/performance check while using a larger common GPU when a BF16 comparison requires it.
6. Add scenario review, kinematic and road-boundary checks, then closed-loop simulator tests for collisions, route completion, compliance and comfort before making driving-performance claims. Review the failed cases rather than relying only on aggregate means.

Publication rights are also a limitation of this artifact set. The model revision carries OpenMDW-1.1, which permits redistribution subject to retaining its license and applicable notices. The driving dataset has different terms: its section 3 treats benchmark/performance results as confidential, and section 4.6 prohibits redistribution of the dataset. The existing repository contains prepared data tensors, reference trajectories, results and earlier commits with those artifacts. Removing a file from the latest revision would leave it in history. This full report is therefore being uploaded privately while publication permission or a separate clean release is resolved. See the pinned model license and pinned dataset agreement.

The evidence can be traced to immutable artifacts. The successful original run contains BF16, simulation and packed-checkpoint results. The completed 5090 run contains the same-device fallback/native measurements and CUDA trace. The combined results file contains the raw rows and the BF16 reference, and the native numerical gate records its 16 cases. This reflection does not change the measured values, the checkpoint, or the historical preparation archive.