ablate channels []: loss 3.1166 ppl 22.57 semsimula_fock_cfc_baoab_code_xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_norc_vplive_xilive_sr2_st16250_L8probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_lr0p0012_noattn fock_cfc_code_xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_norc_vplive_xilive_sr2_st16250_L8probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_lr0p0012_noattn_step2000_snap.pt (built tag xi5long_topk16_dt32da16_mh4_aniso_dcvt5x8_vtjoint_cgqk_norc_vplive_xilive_sr2_st16250_L8probe_ob_untied_wsd_e5c_plgate_rep0.05_fockreg0.005_g0.1_baoab_cfc_lowrank_lr0p0012_noattn) 16 code validation windows x 512; cap PRECISION_LR_MAX = 1.0; n_ctx = 5; K = 8, rank = 4 medians per layer (well k* = the largest g_k at the token): layer omega*dt lam_max m w_max n_eff w(k*) occ(k*) sum g fro2(k*) conc |h-mu| over 2 0 0.825 0.294 0.426 0.304 5.707 0.300 0.711 0.689 0.991 0.447 3.054 0.0% 1 0.761 0.249 0.426 0.988 1.067 0.988 0.653 0.641 1.000 0.399 16.518 0.0% 2 0.611 0.160 0.426 0.993 1.045 0.993 0.414 0.397 1.000 0.415 16.430 0.0% 3 0.623 0.166 0.426 0.983 1.094 0.982 0.407 0.392 1.000 0.438 16.216 0.0% 4 0.640 0.175 0.426 0.987 1.074 0.987 0.413 0.399 1.000 0.448 16.313 0.0% 5 0.582 0.144 0.426 0.983 1.093 0.982 0.346 0.325 1.000 0.455 16.915 0.0% 6 0.477 0.097 0.426 0.977 1.120 0.976 0.242 0.214 1.000 0.459 17.858 0.0% 7 0.250 0.027 0.426 0.901 1.439 0.842 0.113 0.059 1.000 0.457 19.000 0.0% context channels at the bank input (depth code added), per layer: norm median, cosine with the last channel layer 0: norm 0.53 0.37 0.21 0.15 0.13 | cos 0.265 0.385 0.691 0.865 1.000 layer 1: norm 12.08 8.97 6.18 4.97 7.52 | cos 0.128 0.232 0.389 0.571 1.000 layer 2: norm 12.14 9.04 6.16 4.78 7.90 | cos 0.149 0.234 0.377 0.593 1.000 layer 3: norm 12.25 9.13 6.31 4.91 8.34 | cos 0.189 0.229 0.368 0.595 1.000 layer 4: norm 12.44 9.36 6.65 5.26 9.03 | cos 0.243 0.242 0.378 0.599 1.000 layer 5: norm 12.73 9.69 7.13 5.82 9.81 | cos 0.310 0.264 0.394 0.591 1.000 layer 6: norm 13.09 10.11 7.74 6.53 10.68 | cos 0.386 0.298 0.410 0.583 1.000 layer 7: norm 13.54 10.69 8.45 7.38 11.58 | cos 0.463 0.343 0.438 0.581 1.000 well centres per layer: median |h|, mean-over-wells |mu|, nearest-well distance; |W_c xi_c| of mu per channel layer 0: |h| 0.890 |mu| 2.372 nearest 1.476 | per channel 0.96 0.59 0.30 0.23 0.19 layer 1: |h| 19.554 |mu| 30.476 nearest 16.133 | per channel 18.92 11.17 6.78 5.35 7.56 layer 2: |h| 19.596 |mu| 29.558 nearest 15.819 | per channel 18.53 11.21 6.84 4.73 7.63 layer 3: |h| 19.596 |mu| 29.516 nearest 15.830 | per channel 18.68 11.22 6.89 4.59 7.79 layer 4: |h| 19.596 |mu| 29.942 nearest 16.056 | per channel 18.64 11.32 7.09 4.80 8.21 layer 5: |h| 19.596 |mu| 30.673 nearest 16.678 | per channel 18.70 11.44 7.43 5.27 8.78 layer 6: |h| 19.596 |mu| 31.650 nearest 17.653 | per channel 18.68 11.57 7.86 5.95 9.46 layer 7: |h| 19.596 |mu| 32.994 nearest 18.758 | per channel 18.56 11.59 8.33 6.74 10.24 |mu_k| / |h| over all (token, well) pairs: quantiles; share above c; share of the force weight sum_k g_k carried by wells above c layer 0: p05/25/50/75/95/99 1.27 1.67 2.77 3.40 4.23 5.37 | >2: 65.8% wells, 90.2% force >3: 41.6% wells, 57.4% force >4: 7.6% wells, 9.1% force >6: 0.4% wells, 0.4% force layer 1: p05/25/50/75/95/99 0.64 0.81 1.67 2.13 2.86 3.67 | >2: 31.7% wells, 0.0% force >3: 3.6% wells, 0.0% force >4: 0.6% wells, 0.0% force >6: 0.1% wells, 0.0% force layer 2: p05/25/50/75/95/99 0.61 0.78 1.64 2.06 2.73 3.52 | >2: 28.3% wells, 0.0% force >3: 2.7% wells, 0.0% force >4: 0.5% wells, 0.0% force >6: 0.1% wells, 0.0% force layer 3: p05/25/50/75/95/99 0.61 0.78 1.61 2.06 2.76 3.54 | >2: 28.1% wells, 0.0% force >3: 2.8% wells, 0.0% force >4: 0.5% wells, 0.0% force >6: 0.1% wells, 0.0% force layer 4: p05/25/50/75/95/99 0.64 0.81 1.64 2.08 2.77 3.52 | >2: 29.1% wells, 0.0% force >3: 2.9% wells, 0.0% force >4: 0.4% wells, 0.0% force >6: 0.1% wells, 0.0% force layer 5: p05/25/50/75/95/99 0.69 0.85 1.67 2.11 2.82 3.55 | >2: 31.1% wells, 0.0% force >3: 3.3% wells, 0.0% force >4: 0.5% wells, 0.0% force >6: 0.1% wells, 0.0% force layer 6: p05/25/50/75/95/99 0.76 0.91 1.72 2.16 2.87 3.59 | >2: 33.8% wells, 0.0% force >3: 3.7% wells, 0.0% force >4: 0.5% wells, 0.0% force >6: 0.1% wells, 0.0% force layer 7: p05/25/50/75/95/99 0.84 0.99 1.79 2.21 2.90 3.60 | >2: 37.4% wells, 0.0% force >3: 4.0% wells, 0.0% force >4: 0.4% wells, 0.0% force >6: 0.1% wells, 0.0% force per well k = 0..K-1: share of the total force weight sum g_k; share of tokens whose nearest well it is layer 0: force 2.8 6.7 25.7 13.0 20.9 3.6 3.8 23.4 | nearest 0.0 0.0 0.0 0.0 0.0 85.2 14.8 0.0 layer 1: force 8.3 0.0 0.0 0.0 0.0 50.8 40.9 0.0 | nearest 11.7 0.0 0.0 0.0 0.0 53.4 34.8 0.0 layer 2: force 12.2 0.0 0.0 0.0 0.0 62.4 25.5 0.0 | nearest 11.7 0.0 0.0 0.0 0.0 54.0 34.3 0.0 layer 3: force 8.4 0.0 0.0 0.0 0.0 39.2 52.4 0.0 | nearest 11.7 0.0 0.0 0.0 0.0 53.1 35.3 0.0 layer 4: force 9.6 0.0 0.0 0.0 0.0 50.1 40.3 0.0 | nearest 11.5 0.0 0.0 0.0 0.0 53.0 35.5 0.0 layer 5: force 7.6 0.0 0.0 0.0 0.0 50.6 41.8 0.0 | nearest 10.1 0.0 0.0 0.0 0.0 52.9 37.0 0.0 layer 6: force 6.9 0.0 0.0 0.0 0.0 49.8 43.3 0.0 | nearest 6.8 0.0 0.0 0.0 0.0 52.3 40.9 0.0 layer 7: force 6.8 0.0 0.0 0.0 0.0 49.8 43.3 0.0 | nearest 2.5 0.0 0.0 0.0 0.0 49.8 47.7 0.0 nearest-well distance / |h| per token: quantiles; share of tokens above rho layer 0: p50/90/99/99.9/max 1.66 1.86 2.23 2.88 3.23 | >1.5: 72.74% >2: 3.58% >2.5: 0.42% >3: 0.06% layer 1: p50/90/99/99.9/max 0.82 0.97 1.07 1.35 1.39 | >1.5: 0.00% >2: 0.00% >2.5: 0.00% >3: 0.00% layer 2: p50/90/99/99.9/max 0.81 0.95 1.05 1.32 1.37 | >1.5: 0.00% >2: 0.00% >2.5: 0.00% >3: 0.00% layer 3: p50/90/99/99.9/max 0.81 0.95 1.05 1.33 1.37 | >1.5: 0.00% >2: 0.00% >2.5: 0.00% >3: 0.00% layer 4: p50/90/99/99.9/max 0.82 0.96 1.04 1.33 1.38 | >1.5: 0.00% >2: 0.00% >2.5: 0.00% >3: 0.00% layer 5: p50/90/99/99.9/max 0.85 0.98 1.06 1.34 1.40 | >1.5: 0.00% >2: 0.00% >2.5: 0.00% >3: 0.00% layer 6: p50/90/99/99.9/max 0.90 1.01 1.10 1.36 1.43 | >1.5: 0.00% >2: 0.00% >2.5: 0.00% >3: 0.00% layer 7: p50/90/99/99.9/max 0.96 1.06 1.14 1.39 1.47 | >1.5: 0.00% >2: 0.00% >2.5: 0.00% >3: 0.00% well centres per layer split as mu = W (xi - code_l) + W code_l + b (median over tokens of the mean over wells of each norm; cos = alignment of the code term with the total) layer 0: |mu| 2.372 channels 1.611 depth code 1.317 bias 0.931 cos(code term, mu) 0.603 |code_l| 0.35 layer 1: |mu| 30.476 channels 29.941 depth code 5.877 bias 0.931 cos(code term, mu) 0.223 |code_l| 3.05 layer 2: |mu| 29.558 channels 30.128 depth code 6.178 bias 0.931 cos(code term, mu) 0.038 |code_l| 3.19 layer 3: |mu| 29.516 channels 30.424 depth code 4.960 bias 0.931 cos(code term, mu) -0.086 |code_l| 3.06 layer 4: |mu| 29.942 channels 30.905 depth code 5.373 bias 0.931 cos(code term, mu) -0.078 |code_l| 2.88 layer 5: |mu| 30.673 channels 31.438 depth code 6.041 bias 0.931 cos(code term, mu) -0.014 |code_l| 3.29 layer 6: |mu| 31.650 channels 31.895 depth code 7.479 bias 0.931 cos(code term, mu) 0.104 |code_l| 4.32 layer 7: |mu| 32.994 channels 32.261 depth code 9.734 bias 0.931 cos(code term, mu) 0.230 |code_l| 6.14 bank projections: Frobenius norm of the weight columns reading each channel B_proj 82.100 93.548 90.231 81.939 87.004 cos(last two blocks) +0.768 mu_proj 47.018 44.282 41.035 38.063 37.201 cos(last two blocks) +0.481 a_proj 75.381 65.388 60.936 61.502 58.521 cos(last two blocks) +0.485 w_proj 8.463 8.233 8.368 8.763 8.031 cos(last two blocks) +0.279