ablate channels []: loss 3.5578 ppl 35.08 semsimula_fock_cfc_baoab_code_xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_norc_vplive_xilive_sr2_st16250_L8probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_lr0p0012_noattn fock_cfc_code_xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_norc_vplive_xilive_sr2_st16250_L8probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_lr0p0012_noattn_step1000_snap.pt (built tag xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_norc_vplive_xilive_sr2_st16250_L8probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_lr0p0012_noattn) 16 code validation windows x 512; cap PRECISION_LR_MAX = 1.0; n_ctx = 5; K = 8, rank = 4 medians per layer (well k* = the largest g_k at the token): layer omega*dt lam_max m w_max n_eff w(k*) occ(k*) sum g fro2(k*) conc |h-mu| over 2 0 0.635 0.312 0.773 0.318 5.168 0.310 0.705 0.682 0.932 0.522 3.007 0.0% 1 0.545 0.234 0.773 0.990 1.062 0.990 0.588 0.576 1.000 0.416 17.034 0.0% 2 0.478 0.177 0.773 0.989 1.068 0.989 0.441 0.426 1.000 0.428 16.656 0.0% 3 0.456 0.161 0.773 0.987 1.075 0.987 0.392 0.374 1.000 0.442 16.645 0.0% 4 0.466 0.169 0.773 0.987 1.078 0.987 0.405 0.387 1.000 0.451 16.794 0.0% 5 0.468 0.172 0.773 0.986 1.082 0.986 0.411 0.386 1.000 0.456 17.124 0.0% 6 0.449 0.159 0.773 0.983 1.095 0.982 0.380 0.353 1.000 0.458 17.686 0.0% 7 0.229 0.041 0.773 0.929 1.338 0.899 0.137 0.091 1.000 0.457 19.350 0.0% context channels at the bank input (depth code added), per layer: norm median, cosine with the last channel layer 0: norm 0.44 0.30 0.18 0.17 0.14 | cos 0.209 0.297 0.496 0.496 1.000 layer 1: norm 12.15 9.46 5.98 4.93 6.13 | cos 0.189 0.318 0.555 0.651 1.000 layer 2: norm 12.23 9.59 5.97 4.79 6.18 | cos 0.203 0.328 0.569 0.686 1.000 layer 3: norm 12.32 9.67 6.05 4.86 6.35 | cos 0.222 0.325 0.570 0.688 1.000 layer 4: norm 12.44 9.86 6.27 5.06 6.67 | cos 0.247 0.329 0.572 0.686 1.000 layer 5: norm 12.61 10.07 6.57 5.42 7.14 | cos 0.276 0.337 0.576 0.679 1.000 layer 6: norm 12.83 10.33 6.97 5.92 7.76 | cos 0.314 0.348 0.577 0.671 1.000 layer 7: norm 13.13 10.75 7.55 6.66 8.57 | cos 0.350 0.360 0.567 0.646 1.000 well centres per layer: median |h|, mean-over-wells |mu|, nearest-well distance; |W_c xi_c| of mu per channel layer 0: |h| 0.710 |mu| 2.247 nearest 1.342 | per channel 0.71 0.46 0.28 0.32 0.26 layer 1: |h| 19.527 |mu| 25.677 nearest 16.804 | per channel 15.65 9.99 5.61 4.30 5.40 layer 2: |h| 19.596 |mu| 25.400 nearest 16.357 | per channel 15.76 10.12 5.57 3.97 5.35 layer 3: |h| 19.596 |mu| 25.333 nearest 16.331 | per channel 15.80 10.15 5.56 3.84 5.39 layer 4: |h| 19.596 |mu| 25.576 nearest 16.481 | per channel 15.89 10.30 5.68 3.92 5.57 layer 5: |h| 19.596 |mu| 26.132 nearest 16.827 | per channel 16.05 10.41 5.86 4.20 5.90 layer 6: |h| 19.596 |mu| 26.944 nearest 17.397 | per channel 16.17 10.54 6.13 4.61 6.38 layer 7: |h| 19.596 |mu| 28.297 nearest 18.911 | per channel 16.23 10.69 6.55 5.27 7.04 |mu_k| / |h| over all (token, well) pairs: quantiles; share above c; share of the force weight sum_k g_k carried by wells above c layer 0: p05/25/50/75/95/99 1.40 1.94 3.32 4.14 4.95 5.93 | >2: 72.1% wells, 92.5% force >3: 56.3% wells, 82.7% force >4: 29.9% wells, 46.0% force >6: 0.9% wells, 1.2% force layer 1: p05/25/50/75/95/99 0.59 0.76 1.35 1.78 2.50 3.26 | >2: 15.1% wells, 0.0% force >3: 1.7% wells, 0.0% force >4: 0.3% wells, 0.0% force >6: 0.0% wells, 0.0% force layer 2: p05/25/50/75/95/99 0.55 0.71 1.34 1.77 2.45 3.20 | >2: 14.5% wells, 0.0% force >3: 1.5% wells, 0.0% force >4: 0.3% wells, 0.0% force >6: 0.0% wells, 0.0% force layer 3: p05/25/50/75/95/99 0.55 0.71 1.34 1.76 2.42 3.17 | >2: 14.0% wells, 0.0% force >3: 1.4% wells, 0.0% force >4: 0.3% wells, 0.0% force >6: 0.0% wells, 0.0% force layer 4: p05/25/50/75/95/99 0.56 0.73 1.36 1.77 2.42 3.17 | >2: 14.1% wells, 0.0% force >3: 1.4% wells, 0.0% force >4: 0.3% wells, 0.0% force >6: 0.0% wells, 0.0% force layer 5: p05/25/50/75/95/99 0.60 0.77 1.39 1.79 2.43 3.19 | >2: 14.9% wells, 0.0% force >3: 1.5% wells, 0.0% force >4: 0.3% wells, 0.0% force >6: 0.0% wells, 0.0% force layer 6: p05/25/50/75/95/99 0.65 0.82 1.43 1.83 2.45 3.20 | >2: 16.1% wells, 0.0% force >3: 1.5% wells, 0.0% force >4: 0.3% wells, 0.0% force >6: 0.0% wells, 0.0% force layer 7: p05/25/50/75/95/99 0.74 0.91 1.50 1.88 2.46 3.23 | >2: 18.1% wells, 0.0% force >3: 1.6% wells, 0.0% force >4: 0.3% wells, 0.0% force >6: 0.0% wells, 0.0% force per well k = 0..K-1: share of the total force weight sum g_k; share of tokens whose nearest well it is layer 0: force 2.9 2.2 28.7 9.7 22.2 3.6 3.4 27.3 | nearest 0.0 0.0 0.0 0.0 0.0 70.8 29.2 0.0 layer 1: force 12.8 0.0 0.0 0.0 0.0 59.7 27.5 0.0 | nearest 11.7 0.0 0.0 0.0 0.0 58.9 29.4 0.0 layer 2: force 12.7 0.0 0.0 0.0 0.0 58.1 29.2 0.0 | nearest 11.7 0.0 0.0 0.0 0.0 58.7 29.6 0.0 layer 3: force 13.5 0.0 0.0 0.0 0.0 55.5 31.0 0.0 | nearest 11.7 0.0 0.0 0.0 0.0 57.9 30.4 0.0 layer 4: force 14.9 0.0 0.0 0.0 0.0 53.9 31.2 0.0 | nearest 11.7 0.0 0.0 0.0 0.0 56.9 31.4 0.0 layer 5: force 14.9 0.0 0.0 0.0 0.0 53.7 31.4 0.0 | nearest 11.7 0.0 0.0 0.0 0.0 56.3 32.0 0.0 layer 6: force 14.6 0.0 0.0 0.0 0.0 54.3 31.0 0.0 | nearest 11.7 0.0 0.0 0.0 0.0 55.7 32.6 0.0 layer 7: force 18.5 0.1 0.0 0.0 0.0 53.3 28.1 0.0 | nearest 12.4 0.0 0.0 0.0 0.0 53.1 34.5 0.0 nearest-well distance / |h| per token: quantiles; share of tokens above rho layer 0: p50/90/99/99.9/max 1.91 2.16 2.56 2.91 3.24 | >1.5: 85.45% >2: 38.22% >2.5: 1.39% >3: 0.05% layer 1: p50/90/99/99.9/max 0.86 0.99 1.07 1.25 1.43 | >1.5: 0.00% >2: 0.00% >2.5: 0.00% >3: 0.00% layer 2: p50/90/99/99.9/max 0.83 0.97 1.04 1.22 1.40 | >1.5: 0.00% >2: 0.00% >2.5: 0.00% >3: 0.00% layer 3: p50/90/99/99.9/max 0.83 0.97 1.04 1.22 1.38 | >1.5: 0.00% >2: 0.00% >2.5: 0.00% >3: 0.00% layer 4: p50/90/99/99.9/max 0.84 0.98 1.05 1.22 1.38 | >1.5: 0.00% >2: 0.00% >2.5: 0.00% >3: 0.00% layer 5: p50/90/99/99.9/max 0.86 1.00 1.07 1.23 1.37 | >1.5: 0.00% >2: 0.00% >2.5: 0.00% >3: 0.00% layer 6: p50/90/99/99.9/max 0.89 1.02 1.09 1.26 1.38 | >1.5: 0.00% >2: 0.00% >2.5: 0.00% >3: 0.00% layer 7: p50/90/99/99.9/max 0.97 1.09 1.16 1.31 1.44 | >1.5: 0.00% >2: 0.00% >2.5: 0.00% >3: 0.00% well centres per layer split as mu = W (xi - code_l) + W code_l + b (median over tokens of the mean over wells of each norm; cos = alignment of the code term with the total) layer 0: |mu| 2.247 channels 1.004 depth code 1.473 bias 0.621 cos(code term, mu) 0.864 |code_l| 0.36 layer 1: |mu| 25.677 channels 25.064 depth code 4.777 bias 0.621 cos(code term, mu) 0.244 |code_l| 2.73 layer 2: |mu| 25.400 channels 25.250 depth code 2.916 bias 0.621 cos(code term, mu) 0.097 |code_l| 1.83 layer 3: |mu| 25.333 channels 25.487 depth code 2.261 bias 0.621 cos(code term, mu) -0.072 |code_l| 1.73 layer 4: |mu| 25.576 channels 25.813 depth code 2.745 bias 0.621 cos(code term, mu) -0.048 |code_l| 1.88 layer 5: |mu| 26.132 channels 26.192 depth code 3.889 bias 0.621 cos(code term, mu) 0.048 |code_l| 2.36 layer 6: |mu| 26.944 channels 26.613 depth code 5.394 bias 0.621 cos(code term, mu) 0.156 |code_l| 3.22 layer 7: |mu| 28.297 channels 27.060 depth code 8.044 bias 0.621 cos(code term, mu) 0.292 |code_l| 5.38 bank projections: Frobenius norm of the weight columns reading each channel B_proj 56.338 68.426 68.633 61.673 63.914 cos(last two blocks) +0.708 mu_proj 35.695 34.638 32.685 30.971 30.571 cos(last two blocks) +0.318 a_proj 54.783 49.285 47.996 49.191 47.343 cos(last two blocks) +0.548 w_proj 6.665 6.754 6.894 7.311 6.923 cos(last two blocks) +0.228