Text Generation
PyTorch
English
research
semsimula
conservative-language-model
scalar-potential
lagrangian-mechanics
energy-based-model
physics-informed
codeparrot
mechanism-ablation
ladder-study
pre-registered
live-gradients
generation-3
depth
code
matched-token-budget
fock-space
parflm
anisotropic-gaussian-potential
cfc
baoab
fully-conservative
riemannian-geometry
riemannian-geodesics
jacobi-metric
non-transformer
attention-free
constant-memory-inference
Eval Results (legacy)
semsimula-ladder-live-code-d384-l8-none-norc-sr2 / results /well_probe_l8code_code_step18000_L6_none.txt
| ablate channels []: loss 1.9691 ppl 7.16 | |
| semsimula_fock_cfc_baoab_code_xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_norc_vplive_xilive_sr2_st16250_L8probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_lr0p0012_noattn | |
| fock_cfc_code_xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_norc_vplive_xilive_sr2_st16250_L8probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_lr0p0012_noattn_step18000_best.pt (built tag xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_norc_vplive_xilive_sr2_st16250_L8probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_lr0p0012_noattn) | |
| 16 code validation windows x 512; cap PRECISION_LR_MAX = 1.0; n_ctx = 5; K = 8, rank = 4 | |
| medians per layer (well k* = the largest g_k at the token): | |
| layer omega*dt lam_max m w_max n_eff w(k*) occ(k*) sum g fro2(k*) conc |h-mu| over 2 | |
| 0 2.091 0.152 0.035 0.225 7.117 0.215 0.982 0.743 1.000 0.205 3.016 71.1% | |
| 1 1.916 0.128 0.035 0.783 1.945 0.782 0.259 0.252 1.000 0.493 31.214 39.1% | |
| 2 1.368 0.065 0.035 0.790 1.941 0.768 0.173 0.131 1.000 0.510 27.173 8.8% | |
| 3 2.258 0.179 0.035 0.717 2.128 0.711 0.419 0.394 1.000 0.433 28.025 72.4% | |
| 4 2.088 0.153 0.035 0.717 2.088 0.703 0.449 0.374 1.000 0.395 24.793 59.2% | |
| 5 1.662 0.096 0.035 0.732 1.984 0.725 0.298 0.257 1.000 0.364 24.403 20.7% | |
| 6 0.000 0.000 0.035 0.000 1.000 0.000 0.004 0.000 1.000 0.000 40.362 0.0% | |
| 7 0.541 0.010 0.035 0.809 1.696 0.641 0.053 0.027 1.000 0.386 29.857 0.5% | |
| context channels at the bank input (depth code added), per layer: norm median, cosine with the last channel | |
| layer 0: norm 1.38 0.92 0.57 0.39 0.31 | cos 0.247 0.367 0.602 0.832 1.000 | |
| layer 1: norm 15.55 10.24 6.59 5.49 6.59 | cos 0.099 0.147 0.302 0.451 1.000 | |
| layer 2: norm 15.47 10.27 6.61 5.30 7.84 | cos 0.243 0.170 0.282 0.459 1.000 | |
| layer 3: norm 16.21 10.41 7.06 6.01 8.60 | cos 0.409 0.229 0.316 0.499 1.000 | |
| layer 4: norm 16.20 10.67 7.51 6.77 9.47 | cos 0.483 0.277 0.333 0.531 1.000 | |
| layer 5: norm 16.48 11.04 8.21 7.64 9.75 | cos 0.506 0.309 0.346 0.529 1.000 | |
| layer 6: norm 16.88 11.52 8.93 8.17 9.79 | cos 0.505 0.342 0.361 0.529 1.000 | |
| layer 7: norm 17.46 12.02 9.52 8.82 9.97 | cos 0.508 0.381 0.377 0.528 1.000 | |
| well centres per layer: median |h|, mean-over-wells |mu|, nearest-well distance; |W_c xi_c| of mu per channel | |
| layer 0: |h| 2.236 |mu| 4.773 nearest 2.756 | per channel 3.09 2.00 1.31 0.69 0.58 | |
| layer 1: |h| 19.592 |mu| 44.900 nearest 23.691 | per channel 30.82 17.17 12.16 9.44 9.11 | |
| layer 2: |h| 19.596 |mu| 43.228 nearest 22.758 | per channel 29.89 16.59 11.72 7.85 10.27 | |
| layer 3: |h| 19.596 |mu| 43.946 nearest 22.352 | per channel 30.65 16.21 11.76 8.80 10.47 | |
| layer 4: |h| 19.596 |mu| 43.712 nearest 22.120 | per channel 30.06 16.27 12.04 8.71 11.29 | |
| layer 5: |h| 19.596 |mu| 44.777 nearest 23.140 | per channel 30.07 16.67 12.77 9.58 11.74 | |
| layer 6: |h| 19.596 |mu| 46.290 nearest 25.411 | per channel 30.12 17.06 13.73 10.81 12.17 | |
| layer 7: |h| 19.596 |mu| 48.772 nearest 28.157 | per channel 30.40 17.57 14.62 12.42 12.86 | |
| |mu_k| / |h| over all (token, well) pairs: quantiles; share above c; share of the force weight sum_k g_k carried by wells above c | |
| layer 0: p05/25/50/75/95/99 0.94 1.35 2.31 2.74 3.88 5.02 | >2: 61.4% wells, 36.1% force >3: 16.6% wells, 8.8% force >4: 4.3% wells, 2.0% force >6: 0.4% wells, 0.1% force | |
| layer 1: p05/25/50/75/95/99 1.19 1.62 2.39 2.90 3.51 4.07 | >2: 62.0% wells, 0.9% force >3: 20.4% wells, 0.0% force >4: 1.2% wells, 0.0% force >6: 0.1% wells, 0.0% force | |
| layer 2: p05/25/50/75/95/99 1.09 1.46 2.34 2.85 3.43 3.95 | >2: 60.9% wells, 0.3% force >3: 17.9% wells, 0.0% force >4: 0.9% wells, 0.0% force >6: 0.1% wells, 0.0% force | |
| layer 3: p05/25/50/75/95/99 1.16 1.54 2.35 2.87 3.43 3.88 | >2: 61.3% wells, 0.4% force >3: 18.5% wells, 0.0% force >4: 0.7% wells, 0.0% force >6: 0.1% wells, 0.0% force | |
| layer 4: p05/25/50/75/95/99 1.13 1.50 2.36 2.86 3.41 3.84 | >2: 61.7% wells, 0.4% force >3: 17.9% wells, 0.0% force >4: 0.7% wells, 0.0% force >6: 0.1% wells, 0.0% force | |
| layer 5: p05/25/50/75/95/99 1.17 1.58 2.42 2.89 3.43 3.86 | >2: 63.4% wells, 0.7% force >3: 19.2% wells, 0.0% force >4: 0.7% wells, 0.0% force >6: 0.1% wells, 0.0% force | |
| layer 6: p05/25/50/75/95/99 1.29 1.76 2.51 2.91 3.42 3.86 | >2: 67.5% wells, 0.0% force >3: 20.0% wells, 0.0% force >4: 0.7% wells, 0.0% force >6: 0.1% wells, 0.0% force | |
| layer 7: p05/25/50/75/95/99 1.44 1.95 2.63 3.00 3.47 3.90 | >2: 73.6% wells, 4.7% force >3: 24.8% wells, 0.0% force >4: 0.8% wells, 0.0% force >6: 0.1% wells, 0.0% force | |
| per well k = 0..K-1: share of the total force weight sum g_k; share of tokens whose nearest well it is | |
| layer 0: force 7.2 9.0 6.3 6.9 7.2 28.3 27.9 7.2 | nearest 0.0 0.0 0.0 0.0 0.0 0.3 99.7 0.0 | |
| layer 1: force 13.3 0.0 0.0 0.0 0.0 51.5 35.2 0.0 | nearest 0.0 0.0 0.0 0.0 0.0 0.0 100.0 0.0 | |
| layer 2: force 8.6 0.0 0.0 0.0 0.0 54.9 36.6 0.0 | nearest 0.0 0.0 0.0 0.0 0.0 0.0 100.0 0.0 | |
| layer 3: force 8.4 0.0 0.0 0.0 0.0 48.6 42.9 0.0 | nearest 0.0 0.0 0.0 0.0 0.0 0.0 100.0 0.0 | |
| layer 4: force 4.6 0.0 0.0 0.0 0.0 42.9 52.5 0.0 | nearest 0.0 0.0 0.0 0.0 0.0 0.0 100.0 0.0 | |
| layer 5: force 3.0 0.0 0.0 0.0 0.0 39.1 58.0 0.0 | nearest 0.0 0.0 0.0 0.0 0.0 0.0 100.0 0.0 | |
| layer 6: force 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 | nearest 0.0 0.0 0.0 0.0 0.0 0.2 99.8 0.0 | |
| layer 7: force 0.9 0.0 0.0 0.0 0.0 32.7 66.4 0.0 | nearest 0.0 0.0 0.0 0.0 0.0 0.6 99.4 0.0 | |
| nearest-well distance / |h| per token: quantiles; share of tokens above rho | |
| layer 0: p50/90/99/99.9/max 1.21 1.34 1.78 2.78 3.63 | >1.5: 2.75% >2: 0.60% >2.5: 0.21% >3: 0.06% | |
| layer 1: p50/90/99/99.9/max 1.21 1.40 1.69 2.36 2.57 | >1.5: 3.83% >2: 0.26% >2.5: 0.04% >3: 0.00% | |
| layer 2: p50/90/99/99.9/max 1.16 1.36 1.63 2.33 2.52 | >1.5: 2.44% >2: 0.20% >2.5: 0.02% >3: 0.00% | |
| layer 3: p50/90/99/99.9/max 1.14 1.34 1.63 2.30 2.51 | >1.5: 2.33% >2: 0.18% >2.5: 0.02% >3: 0.00% | |
| layer 4: p50/90/99/99.9/max 1.13 1.33 1.61 2.26 2.47 | >1.5: 2.14% >2: 0.16% >2.5: 0.00% >3: 0.00% | |
| layer 5: p50/90/99/99.9/max 1.18 1.39 1.64 2.25 2.45 | >1.5: 3.11% >2: 0.17% >2.5: 0.00% >3: 0.00% | |
| layer 6: p50/90/99/99.9/max 1.30 1.52 1.76 2.27 2.48 | >1.5: 11.82% >2: 0.22% >2.5: 0.00% >3: 0.00% | |
| layer 7: p50/90/99/99.9/max 1.44 1.67 1.91 2.38 2.58 | >1.5: 36.13% >2: 0.56% >2.5: 0.06% >3: 0.00% | |
| well centres per layer split as mu = W (xi - code_l) + W code_l + b (median over tokens of the mean over wells of each norm; cos = alignment of the code term with the total) | |
| layer 0: |mu| 4.773 channels 5.059 depth code 2.657 bias 1.838 cos(code term, mu) 0.014 |code_l| 0.93 | |
| layer 1: |mu| 44.900 channels 45.946 depth code 14.426 bias 1.838 cos(code term, mu) 0.118 |code_l| 5.52 | |
| layer 2: |mu| 43.228 channels 46.089 depth code 14.980 bias 1.838 cos(code term, mu) 0.025 |code_l| 6.45 | |
| layer 3: |mu| 43.946 channels 46.699 depth code 13.287 bias 1.838 cos(code term, mu) -0.002 |code_l| 5.66 | |
| layer 4: |mu| 43.712 channels 47.783 depth code 13.755 bias 1.838 cos(code term, mu) -0.104 |code_l| 6.28 | |
| layer 5: |mu| 44.777 channels 49.071 depth code 14.071 bias 1.838 cos(code term, mu) -0.122 |code_l| 6.53 | |
| layer 6: |mu| 46.290 channels 49.855 depth code 13.794 bias 1.838 cos(code term, mu) -0.077 |code_l| 6.37 | |
| layer 7: |mu| 48.772 channels 49.626 depth code 13.664 bias 1.838 cos(code term, mu) 0.110 |code_l| 6.70 | |
| bank projections: Frobenius norm of the weight columns reading each channel | |
| B_proj 251.214 259.865 248.268 224.514 227.953 cos(last two blocks) +0.680 | |
| mu_proj 86.023 85.048 88.510 78.017 76.258 cos(last two blocks) +0.669 | |
| a_proj 149.009 145.664 137.925 129.616 126.742 cos(last two blocks) +0.703 | |
| w_proj 5.512 5.423 6.108 6.174 5.387 cos(last two blocks) +0.128 | |