| --- |
| language: |
| - en |
| license: mit |
| library_name: flux |
| tags: |
| - julia |
| - flux-jl |
| - distillation |
| - knowledge-distillation |
| - llama-style |
| - gqa |
| - rope |
| - rmsnorm |
| - swiglu |
| - bpe |
| - philosophy |
| - text-generation |
| pipeline_tag: text-generation |
| datasets: |
| - LisaMegaWatts/philosophy-corpus |
| model-index: |
| - name: JuliaGPTDistill |
| results: |
| - task: |
| type: text-generation |
| name: Text Generation |
| dataset: |
| type: LisaMegaWatts/philosophy-corpus |
| name: philosophy-corpus |
| metrics: |
| - type: loss |
| value: 7.44 |
| name: Val Loss |
| verified: false |
| --- |
| |
| # JuliaGPTDistill |
|
|
| A **~5M parameter** LLaMA-style student model distilled from [JuliaFluxGPT](https://huggingface.co/LisaMegaWatts/JuliaFluxGPT) (10M params). Uses knowledge distillation with temperature scaling to compress the teacher's knowledge into a smaller architecture. |
|
|
| ## Architecture |
|
|
| | Parameter | Value | |
| |-----------|-------| |
| | Architecture | LLaMA-style (RMSNorm, SwiGLU, RoPE, GQA) | |
| | Embedding dim | 256 | |
| | Layers | 4 | |
| | Query heads | 4 | |
| | KV heads | 2 (GQA ratio 2:1) | |
| | Head dim | 64 | |
| | Context length | 256 tokens | |
| | Vocabulary | 2,000 (ByteLevel BPE) | |
| | Dropout | 0.1 | |
| | Weight tying | Yes | |
| | Framework | Julia + Flux.jl | |
|
|
| ## Distillation Settings |
|
|
| | Parameter | Value | |
| |-----------|-------| |
| | Teacher model | [JuliaFluxGPT](https://huggingface.co/LisaMegaWatts/JuliaFluxGPT) (512d/8L/8Q/2KV) | |
| | KD temperature | 4.0 | |
| | KD alpha | 0.5 | |
| | Loss | 0.5 * CE + 0.5 * KL(teacher \|\| student) | |
|
|
| ## Training |
|
|
| | | Value | |
| |---|---| |
| | Dataset | [philosophy-corpus](https://huggingface.co/datasets/LisaMegaWatts/philosophy-corpus) | |
| | Tokenizer | BPE (2,000 vocab, ByteLevel) | |
| | Training steps | 4,089 | |
| | Best val loss | 7.44 | |
| | Hardware | NVIDIA RTX 3060 12GB | |
|
|
| ## Inference Settings |
|
|
| | Parameter | Value | |
| |-----------|-------| |
| | vocab_size | 2,000 | |
| | context_length | 256 | |
| | temperature | 0.8 | |
| | top_k | 40 | |
| |
| **Note:** This model requires the same BPE tokenizer used by JuliaFluxGPT. No tokenizer file is included in this repo — use the tokenizer from [JuliaFluxGPT](https://huggingface.co/LisaMegaWatts/JuliaFluxGPT/blob/main/tokenizer.json). |
| |
| ## Checkpoint Format |
| |
| JLD2 files containing: |
| - `model_state` — Flux model weights |
| - `hyperparams` — `Dict("n_embd"=>256, "n_layer"=>4, "n_head"=>4, "n_kv_head"=>2, "vocab_size"=>2000, "block_size"=>256, "dropout"=>0.1, "kd_temperature"=>4.0, "kd_alpha"=>0.5)` |
| - `step`, `best_val_loss`, `train_losses`, `val_losses` |
|
|
| ## Files |
|
|
| | File | Description | |
| |------|-------------| |
| | `best_model.jld2` | Best validation loss checkpoint | |
| | `final_model.jld2` | Final training step checkpoint | |
| | `checkpoint_latest.jld2` | Latest periodic checkpoint | |
|
|
| ## Provenance |
|
|
| - **Author**: LisaMegaWatts |
| - **Source code**: [DavinciDreams/JuliaGPT](https://github.com/DavinciDreams/JuliaGPT) |
| - **Teacher model**: [LisaMegaWatts/JuliaFluxGPT](https://huggingface.co/LisaMegaWatts/JuliaFluxGPT) |
|
|
| ## License |
|
|
| MIT |
|
|
