Text Generation
PyTorch
English
research
semsimula
conservative-language-model
scalar-potential
lagrangian-mechanics
energy-based-model
physics-informed
codeparrot
mechanism-ablation
ladder-study
pre-registered
live-gradients
generation-3
depth
code
matched-token-budget
fock-space
parflm
anisotropic-gaussian-potential
cfc
baoab
fully-conservative
riemannian-geometry
riemannian-geodesics
jacobi-metric
non-transformer
attention-free
constant-memory-inference
Eval Results (legacy)
| COPY SPLIT on code validation: 512 windows x 512 = 262,144 targets, reference gpt2 | |
| ppl on these tokens: fob 7.023, fock 6.089, gpt2 5.350, l4 5.898, l8 5.765, so 6.442, xi6 9.159 | |
| fob against gpt2: mean gap +0.2723 nats/token (ppl ratio 1.313) | |
| group share gpt2 nll fob nll gap ratio of gap seq spread | |
| all 100.0% 1.677 1.949 +0.272 1.31 100.0% ±0.006 | |
| first occurrence 27.7% 4.012 4.286 +0.274 1.32 27.9% ±0.008 | |
| repeat (not induction) 22.3% 1.603 1.967 +0.364 1.44 29.8% ±0.009 | |
| induction 50.0% 0.414 0.644 +0.230 1.26 42.2% ±0.007 | |
| indent/newline/space 21.9% 0.278 0.328 +0.050 1.05 4.0% ±0.005 | |
| identifier-like 49.3% 2.568 2.972 +0.404 1.50 73.3% ±0.010 | |
| number 7.4% 1.610 1.792 +0.181 1.20 4.9% ±0.012 | |
| punct/other 21.4% 1.078 1.306 +0.227 1.26 17.8% ±0.008 | |
| identifier, first 20.3% 4.497 4.794 +0.298 1.35 22.2% ±0.009 | |
| identifier, repeat 8.7% 2.404 2.993 +0.589 1.80 18.7% ±0.020 | |
| identifier, induction 20.4% 0.718 1.150 +0.432 1.54 32.4% ±0.015 | |
| repeat/ind., dist 1-16 34.8% 0.491 0.643 +0.152 1.16 19.4% ±0.005 | |
| repeat/ind., dist 17-64 25.2% 0.805 1.115 +0.310 1.36 28.7% ±0.009 | |
| repeat/ind., dist 65-256 10.8% 1.474 1.994 +0.520 1.68 20.7% ±0.015 | |
| repeat/ind., dist 257-512 1.5% 2.065 2.657 +0.592 1.81 3.3% ±0.038 | |
| position 0-63 12.5% 2.312 2.494 +0.182 1.20 8.4% ±0.007 | |
| position 64-255 37.5% 1.707 1.971 +0.264 1.30 36.3% ±0.007 | |
| position 256-511 50.0% 1.496 1.797 +0.301 1.35 55.3% ±0.008 | |
| fock against gpt2: mean gap +0.1294 nats/token (ppl ratio 1.138) | |
| group share gpt2 nll fock nll gap ratio of gap seq spread | |
| all 100.0% 1.677 1.806 +0.129 1.14 100.0% ±0.004 | |
| first occurrence 27.7% 4.012 4.138 +0.126 1.13 27.0% ±0.007 | |
| repeat (not induction) 22.3% 1.603 1.780 +0.177 1.19 30.5% ±0.007 | |
| induction 50.0% 0.414 0.524 +0.110 1.12 42.5% ±0.004 | |
| indent/newline/space 21.9% 0.278 0.311 +0.033 1.03 5.6% ±0.005 | |
| identifier-like 49.3% 2.568 2.745 +0.177 1.19 67.3% ±0.007 | |
| number 7.4% 1.610 1.700 +0.090 1.09 5.1% ±0.009 | |
| punct/other 21.4% 1.078 1.212 +0.133 1.14 22.0% ±0.006 | |
| identifier, first 20.3% 4.497 4.631 +0.134 1.14 21.1% ±0.008 | |
| identifier, repeat 8.7% 2.404 2.648 +0.244 1.28 16.3% ±0.015 | |
| identifier, induction 20.4% 0.718 0.908 +0.190 1.21 29.9% ±0.009 | |
| repeat/ind., dist 1-16 34.8% 0.491 0.572 +0.080 1.08 21.6% ±0.004 | |
| repeat/ind., dist 17-64 25.2% 0.805 0.949 +0.144 1.15 28.0% ±0.006 | |
| repeat/ind., dist 65-256 10.8% 1.474 1.710 +0.237 1.27 19.8% ±0.012 | |
| repeat/ind., dist 257-512 1.5% 2.065 2.371 +0.306 1.36 3.6% ±0.036 | |
| position 0-63 12.5% 2.312 2.387 +0.076 1.08 7.3% ±0.006 | |
| position 64-255 37.5% 1.707 1.829 +0.122 1.13 35.2% ±0.005 | |
| position 256-511 50.0% 1.496 1.644 +0.149 1.16 57.5% ±0.005 | |
| l4 against gpt2: mean gap +0.0977 nats/token (ppl ratio 1.103) | |
| group share gpt2 nll l4 nll gap ratio of gap seq spread | |
| all 100.0% 1.677 1.775 +0.098 1.10 100.0% ±0.004 | |
| first occurrence 27.7% 4.012 4.111 +0.099 1.10 28.3% ±0.006 | |
| repeat (not induction) 22.3% 1.603 1.743 +0.140 1.15 32.0% ±0.007 | |
| induction 50.0% 0.414 0.492 +0.078 1.08 39.8% ±0.004 | |
| indent/newline/space 21.9% 0.278 0.306 +0.028 1.03 6.2% ±0.004 | |
| identifier-like 49.3% 2.568 2.702 +0.134 1.14 67.5% ±0.006 | |
| number 7.4% 1.610 1.678 +0.068 1.07 5.1% ±0.011 | |
| punct/other 21.4% 1.078 1.175 +0.097 1.10 21.2% ±0.006 | |
| identifier, first 20.3% 4.497 4.604 +0.107 1.11 22.3% ±0.008 | |
| identifier, repeat 8.7% 2.404 2.607 +0.203 1.23 18.0% ±0.014 | |
| identifier, induction 20.4% 0.718 0.849 +0.130 1.14 27.2% ±0.008 | |
| repeat/ind., dist 1-16 34.8% 0.491 0.551 +0.060 1.06 21.3% ±0.003 | |
| repeat/ind., dist 17-64 25.2% 0.805 0.912 +0.107 1.11 27.5% ±0.005 | |
| repeat/ind., dist 65-256 10.8% 1.474 1.651 +0.177 1.19 19.6% ±0.011 | |
| repeat/ind., dist 257-512 1.5% 2.065 2.275 +0.211 1.23 3.3% ±0.032 | |
| position 0-63 12.5% 2.312 2.375 +0.063 1.07 8.1% ±0.006 | |
| position 64-255 37.5% 1.707 1.802 +0.094 1.10 36.2% ±0.004 | |
| position 256-511 50.0% 1.496 1.604 +0.109 1.12 55.7% ±0.005 | |
| l8 against gpt2: mean gap +0.0748 nats/token (ppl ratio 1.078) | |
| group share gpt2 nll l8 nll gap ratio of gap seq spread | |
| all 100.0% 1.677 1.752 +0.075 1.08 100.0% ±0.003 | |
| first occurrence 27.7% 4.012 4.089 +0.077 1.08 28.7% ±0.006 | |
| repeat (not induction) 22.3% 1.603 1.720 +0.117 1.12 34.8% ±0.006 | |
| induction 50.0% 0.414 0.468 +0.055 1.06 36.5% ±0.003 | |
| indent/newline/space 21.9% 0.278 0.298 +0.019 1.02 5.7% ±0.004 | |
| identifier-like 49.3% 2.568 2.674 +0.105 1.11 69.6% ±0.006 | |
| number 7.4% 1.610 1.650 +0.040 1.04 3.9% ±0.008 | |
| punct/other 21.4% 1.078 1.151 +0.073 1.08 20.7% ±0.005 | |
| identifier, first 20.3% 4.497 4.588 +0.091 1.10 24.7% ±0.007 | |
| identifier, repeat 8.7% 2.404 2.564 +0.160 1.17 18.5% ±0.013 | |
| identifier, induction 20.4% 0.718 0.815 +0.097 1.10 26.4% ±0.008 | |
| repeat/ind., dist 1-16 34.8% 0.491 0.537 +0.045 1.05 21.1% ±0.003 | |
| repeat/ind., dist 17-64 25.2% 0.805 0.886 +0.081 1.08 27.2% ±0.004 | |
| repeat/ind., dist 65-256 10.8% 1.474 1.609 +0.135 1.14 19.5% ±0.010 | |
| repeat/ind., dist 257-512 1.5% 2.065 2.237 +0.173 1.19 3.5% ±0.030 | |
| position 0-63 12.5% 2.312 2.361 +0.050 1.05 8.3% ±0.006 | |
| position 64-255 37.5% 1.707 1.778 +0.071 1.07 35.7% ±0.004 | |
| position 256-511 50.0% 1.496 1.579 +0.084 1.09 56.0% ±0.004 | |
| so against gpt2: mean gap +0.1859 nats/token (ppl ratio 1.204) | |
| group share gpt2 nll so nll gap ratio of gap seq spread | |
| all 100.0% 1.677 1.863 +0.186 1.20 100.0% ±0.005 | |
| first occurrence 27.7% 4.012 4.201 +0.189 1.21 28.2% ±0.007 | |
| repeat (not induction) 22.3% 1.603 1.857 +0.254 1.29 30.4% ±0.008 | |
| induction 50.0% 0.414 0.567 +0.154 1.17 41.3% ±0.005 | |
| indent/newline/space 21.9% 0.278 0.320 +0.042 1.04 4.9% ±0.006 | |
| identifier-like 49.3% 2.568 2.821 +0.253 1.29 67.1% ±0.008 | |
| number 7.4% 1.610 1.762 +0.152 1.16 6.0% ±0.010 | |
| punct/other 21.4% 1.078 1.270 +0.192 1.21 22.0% ±0.009 | |
| identifier, first 20.3% 4.497 4.695 +0.198 1.22 21.6% ±0.009 | |
| identifier, repeat 8.7% 2.404 2.767 +0.363 1.44 16.9% ±0.016 | |
| identifier, induction 20.4% 0.718 0.978 +0.260 1.30 28.5% ±0.011 | |
| repeat/ind., dist 1-16 34.8% 0.491 0.601 +0.109 1.12 20.5% ±0.004 | |
| repeat/ind., dist 17-64 25.2% 0.805 1.013 +0.208 1.23 28.1% ±0.007 | |
| repeat/ind., dist 65-256 10.8% 1.474 1.816 +0.342 1.41 19.9% ±0.012 | |
| repeat/ind., dist 257-512 1.5% 2.065 2.460 +0.395 1.48 3.2% ±0.038 | |
| position 0-63 12.5% 2.312 2.433 +0.121 1.13 8.1% ±0.007 | |
| position 64-255 37.5% 1.707 1.885 +0.178 1.19 35.9% ±0.005 | |
| position 256-511 50.0% 1.496 1.704 +0.208 1.23 56.0% ±0.006 | |
| xi6 against gpt2: mean gap +0.5377 nats/token (ppl ratio 1.712) | |
| group share gpt2 nll xi6 nll gap ratio of gap seq spread | |
| all 100.0% 1.677 2.215 +0.538 1.71 100.0% ±0.012 | |
| first occurrence 27.7% 4.012 4.333 +0.321 1.38 16.6% ±0.010 | |
| repeat (not induction) 22.3% 1.603 2.047 +0.444 1.56 18.4% ±0.012 | |
| induction 50.0% 0.414 1.114 +0.700 2.01 65.0% ±0.018 | |
| indent/newline/space 21.9% 0.278 0.380 +0.102 1.11 4.1% ±0.007 | |
| identifier-like 49.3% 2.568 3.334 +0.766 2.15 70.3% ±0.020 | |
| number 7.4% 1.610 2.043 +0.433 1.54 5.9% ±0.020 | |
| punct/other 21.4% 1.078 1.574 +0.495 1.64 19.7% ±0.014 | |
| identifier, first 20.3% 4.497 4.826 +0.329 1.39 12.4% ±0.012 | |
| identifier, repeat 8.7% 2.404 3.090 +0.687 1.99 11.0% ±0.025 | |
| identifier, induction 20.4% 0.718 1.952 +1.234 3.43 46.8% ±0.034 | |
| repeat/ind., dist 1-16 34.8% 0.491 0.807 +0.316 1.37 20.4% ±0.010 | |
| repeat/ind., dist 17-64 25.2% 0.805 1.553 +0.748 2.11 35.0% ±0.019 | |
| repeat/ind., dist 65-256 10.8% 1.474 2.686 +1.212 3.36 24.4% ±0.030 | |
| repeat/ind., dist 257-512 1.5% 2.065 3.338 +1.274 3.57 3.6% ±0.061 | |
| position 0-63 12.5% 2.312 2.587 +0.275 1.32 6.4% ±0.012 | |
| position 64-255 37.5% 1.707 2.215 +0.507 1.66 35.4% ±0.014 | |
| position 256-511 50.0% 1.496 2.122 +0.626 1.87 58.2% ±0.016 | |