Text Generation
PyTorch
English
research
semsimula
conservative-language-model
scalar-potential
lagrangian-mechanics
energy-based-model
physics-informed
codeparrot
mechanism-ablation
ladder-study
pre-registered
live-gradients
generation-3
depth
code
matched-token-budget
fock-space
parflm
anisotropic-gaussian-potential
cfc
baoab
fully-conservative
riemannian-geometry
riemannian-geodesics
jacobi-metric
non-transformer
attention-free
constant-memory-inference
Eval Results (legacy)
semsimula-ladder-live-code-d384-l8-none-norc-sr2 / results /well_probe_l8code_code_step18000_L3_none.txt
| ablate channels []: loss 1.9307 ppl 6.89 | |
| semsimula_fock_cfc_baoab_code_xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_norc_vplive_xilive_sr2_st16250_L8probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_lr0p0012_noattn | |
| fock_cfc_code_xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_norc_vplive_xilive_sr2_st16250_L8probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_lr0p0012_noattn_step18000_best.pt (built tag xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_norc_vplive_xilive_sr2_st16250_L8probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_lr0p0012_noattn) | |
| 16 code validation windows x 512; cap PRECISION_LR_MAX = 1.0; n_ctx = 5; K = 8, rank = 4 | |
| medians per layer (well k* = the largest g_k at the token): | |
| layer omega*dt lam_max m w_max n_eff w(k*) occ(k*) sum g fro2(k*) conc |h-mu| over 2 | |
| 0 2.091 0.152 0.035 0.225 7.117 0.215 0.982 0.743 1.000 0.205 3.016 71.1% | |
| 1 1.916 0.128 0.035 0.783 1.945 0.782 0.259 0.252 1.000 0.493 31.214 39.1% | |
| 2 1.368 0.065 0.035 0.790 1.941 0.768 0.173 0.131 1.000 0.510 27.173 8.8% | |
| 3 0.000 0.000 0.035 0.000 1.000 0.000 0.016 0.000 1.000 0.000 35.696 0.0% | |
| 4 1.949 0.133 0.035 0.712 2.108 0.691 0.409 0.330 1.000 0.393 24.541 45.9% | |
| 5 1.704 0.101 0.035 0.723 2.003 0.712 0.334 0.272 1.000 0.368 24.274 26.8% | |
| 6 1.149 0.046 0.035 0.766 1.835 0.735 0.165 0.127 1.000 0.364 26.479 6.9% | |
| 7 0.746 0.019 0.035 0.783 1.786 0.686 0.095 0.055 1.000 0.364 30.308 2.6% | |
| context channels at the bank input (depth code added), per layer: norm median, cosine with the last channel | |
| layer 0: norm 1.38 0.92 0.57 0.39 0.31 | cos 0.247 0.367 0.602 0.832 1.000 | |
| layer 1: norm 15.55 10.24 6.59 5.49 6.59 | cos 0.099 0.147 0.302 0.451 1.000 | |
| layer 2: norm 15.47 10.27 6.61 5.30 7.84 | cos 0.243 0.170 0.282 0.459 1.000 | |
| layer 3: norm 16.21 10.41 7.06 6.01 8.60 | cos 0.409 0.229 0.316 0.499 1.000 | |
| layer 4: norm 16.17 10.56 7.41 6.64 9.43 | cos 0.493 0.283 0.339 0.534 1.000 | |
| layer 5: norm 16.43 10.78 8.02 7.50 9.74 | cos 0.533 0.327 0.370 0.545 1.000 | |
| layer 6: norm 16.78 11.25 8.75 8.12 9.87 | cos 0.528 0.352 0.381 0.532 1.000 | |
| layer 7: norm 17.48 11.88 9.40 8.63 10.20 | cos 0.579 0.417 0.411 0.537 1.000 | |
| well centres per layer: median |h|, mean-over-wells |mu|, nearest-well distance; |W_c xi_c| of mu per channel | |
| layer 0: |h| 2.236 |mu| 4.773 nearest 2.756 | per channel 3.09 2.00 1.31 0.69 0.58 | |
| layer 1: |h| 19.592 |mu| 44.900 nearest 23.691 | per channel 30.82 17.17 12.16 9.44 9.11 | |
| layer 2: |h| 19.596 |mu| 43.228 nearest 22.758 | per channel 29.89 16.59 11.72 7.85 10.27 | |
| layer 3: |h| 19.596 |mu| 43.946 nearest 22.352 | per channel 30.65 16.21 11.76 8.80 10.47 | |
| layer 4: |h| 19.596 |mu| 43.422 nearest 21.987 | per channel 30.06 16.02 11.84 8.55 11.29 | |
| layer 5: |h| 19.596 |mu| 44.233 nearest 22.708 | per channel 30.01 16.14 12.43 9.34 11.72 | |
| layer 6: |h| 19.596 |mu| 45.887 nearest 24.760 | per channel 30.20 16.62 13.42 10.57 12.00 | |
| layer 7: |h| 19.596 |mu| 48.492 nearest 28.037 | per channel 30.48 17.46 14.39 11.99 13.12 | |
| |mu_k| / |h| over all (token, well) pairs: quantiles; share above c; share of the force weight sum_k g_k carried by wells above c | |
| layer 0: p05/25/50/75/95/99 0.94 1.35 2.31 2.74 3.88 5.02 | >2: 61.4% wells, 36.1% force >3: 16.6% wells, 8.8% force >4: 4.3% wells, 2.0% force >6: 0.4% wells, 0.1% force | |
| layer 1: p05/25/50/75/95/99 1.19 1.62 2.39 2.90 3.51 4.07 | >2: 62.0% wells, 0.9% force >3: 20.4% wells, 0.0% force >4: 1.2% wells, 0.0% force >6: 0.1% wells, 0.0% force | |
| layer 2: p05/25/50/75/95/99 1.09 1.46 2.34 2.85 3.43 3.95 | >2: 60.9% wells, 0.3% force >3: 17.9% wells, 0.0% force >4: 0.9% wells, 0.0% force >6: 0.1% wells, 0.0% force | |
| layer 3: p05/25/50/75/95/99 1.16 1.54 2.35 2.87 3.43 3.88 | >2: 61.3% wells, 0.0% force >3: 18.5% wells, 0.0% force >4: 0.7% wells, 0.0% force >6: 0.1% wells, 0.0% force | |
| layer 4: p05/25/50/75/95/99 1.12 1.48 2.34 2.84 3.40 3.82 | >2: 61.3% wells, 0.3% force >3: 17.3% wells, 0.0% force >4: 0.6% wells, 0.0% force >6: 0.1% wells, 0.0% force | |
| layer 5: p05/25/50/75/95/99 1.14 1.55 2.39 2.87 3.42 3.84 | >2: 62.6% wells, 0.4% force >3: 18.6% wells, 0.0% force >4: 0.6% wells, 0.0% force >6: 0.1% wells, 0.0% force | |
| layer 6: p05/25/50/75/95/99 1.24 1.71 2.49 2.91 3.44 3.88 | >2: 66.1% wells, 1.1% force >3: 20.1% wells, 0.0% force >4: 0.7% wells, 0.0% force >6: 0.1% wells, 0.0% force | |
| layer 7: p05/25/50/75/95/99 1.44 1.95 2.61 2.98 3.46 3.90 | >2: 73.5% wells, 4.8% force >3: 23.6% wells, 0.0% force >4: 0.7% wells, 0.0% force >6: 0.1% wells, 0.0% force | |
| per well k = 0..K-1: share of the total force weight sum g_k; share of tokens whose nearest well it is | |
| layer 0: force 7.2 9.0 6.3 6.9 7.2 28.3 27.9 7.2 | nearest 0.0 0.0 0.0 0.0 0.0 0.3 99.7 0.0 | |
| layer 1: force 13.3 0.0 0.0 0.0 0.0 51.5 35.2 0.0 | nearest 0.0 0.0 0.0 0.0 0.0 0.0 100.0 0.0 | |
| layer 2: force 8.6 0.0 0.0 0.0 0.0 54.9 36.6 0.0 | nearest 0.0 0.0 0.0 0.0 0.0 0.0 100.0 0.0 | |
| layer 3: force 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 | nearest 0.0 0.0 0.0 0.0 0.0 0.0 100.0 0.0 | |
| layer 4: force 4.0 0.0 0.0 0.0 0.0 44.5 51.5 0.0 | nearest 0.0 0.0 0.0 0.0 0.0 0.0 100.0 0.0 | |
| layer 5: force 2.6 0.0 0.0 0.0 0.0 42.4 54.9 0.0 | nearest 0.0 0.0 0.0 0.0 0.0 0.0 100.0 0.0 | |
| layer 6: force 1.8 0.0 0.0 0.0 0.0 43.8 54.3 0.0 | nearest 0.0 0.0 0.0 0.0 0.0 0.2 99.8 0.0 | |
| layer 7: force 0.8 0.0 0.0 0.0 0.0 43.2 55.9 0.0 | nearest 0.0 0.0 0.0 0.0 0.0 0.6 99.4 0.0 | |
| nearest-well distance / |h| per token: quantiles; share of tokens above rho | |
| layer 0: p50/90/99/99.9/max 1.21 1.34 1.78 2.78 3.63 | >1.5: 2.75% >2: 0.60% >2.5: 0.21% >3: 0.06% | |
| layer 1: p50/90/99/99.9/max 1.21 1.40 1.69 2.36 2.57 | >1.5: 3.83% >2: 0.26% >2.5: 0.04% >3: 0.00% | |
| layer 2: p50/90/99/99.9/max 1.16 1.36 1.63 2.33 2.52 | >1.5: 2.44% >2: 0.20% >2.5: 0.02% >3: 0.00% | |
| layer 3: p50/90/99/99.9/max 1.14 1.34 1.63 2.30 2.51 | >1.5: 2.33% >2: 0.18% >2.5: 0.02% >3: 0.00% | |
| layer 4: p50/90/99/99.9/max 1.12 1.33 1.60 2.26 2.47 | >1.5: 2.05% >2: 0.16% >2.5: 0.00% >3: 0.00% | |
| layer 5: p50/90/99/99.9/max 1.16 1.36 1.63 2.25 2.45 | >1.5: 2.42% >2: 0.17% >2.5: 0.00% >3: 0.00% | |
| layer 6: p50/90/99/99.9/max 1.26 1.48 1.72 2.27 2.47 | >1.5: 8.04% >2: 0.20% >2.5: 0.00% >3: 0.00% | |
| layer 7: p50/90/99/99.9/max 1.43 1.66 1.89 2.38 2.58 | >1.5: 34.74% >2: 0.49% >2.5: 0.06% >3: 0.00% | |
| well centres per layer split as mu = W (xi - code_l) + W code_l + b (median over tokens of the mean over wells of each norm; cos = alignment of the code term with the total) | |
| layer 0: |mu| 4.773 channels 5.059 depth code 2.657 bias 1.838 cos(code term, mu) 0.014 |code_l| 0.93 | |
| layer 1: |mu| 44.900 channels 45.946 depth code 14.426 bias 1.838 cos(code term, mu) 0.118 |code_l| 5.52 | |
| layer 2: |mu| 43.228 channels 46.089 depth code 14.980 bias 1.838 cos(code term, mu) 0.025 |code_l| 6.45 | |
| layer 3: |mu| 43.946 channels 46.699 depth code 13.287 bias 1.838 cos(code term, mu) -0.002 |code_l| 5.66 | |
| layer 4: |mu| 43.422 channels 47.469 depth code 13.755 bias 1.838 cos(code term, mu) -0.102 |code_l| 6.28 | |
| layer 5: |mu| 44.233 channels 48.505 depth code 14.071 bias 1.838 cos(code term, mu) -0.116 |code_l| 6.53 | |
| layer 6: |mu| 45.887 channels 49.329 depth code 13.794 bias 1.838 cos(code term, mu) -0.067 |code_l| 6.37 | |
| layer 7: |mu| 48.492 channels 48.972 depth code 13.664 bias 1.838 cos(code term, mu) 0.136 |code_l| 6.70 | |
| bank projections: Frobenius norm of the weight columns reading each channel | |
| B_proj 251.214 259.865 248.268 224.514 227.953 cos(last two blocks) +0.680 | |
| mu_proj 86.023 85.048 88.510 78.017 76.258 cos(last two blocks) +0.669 | |
| a_proj 149.009 145.664 137.925 129.616 126.742 cos(last two blocks) +0.703 | |
| w_proj 5.512 5.423 6.108 6.174 5.387 cos(last two blocks) +0.128 | |